Coding Horror

programming and human factors

Standard Flavored Markdown

In 2009 I lamented the state of Markdown:

Right now we have the worst of both worlds. Lack of leadership from the top, and a bunch of fragmented, poorly coordinated community efforts to advance Markdown, none of which are officially canon. This isn't merely incovenient for anyone trying to find accurate information about Markdown; it's actually harming the project's future.

In late 2012, David Greenspan from Meteor approached me and proposed we move forward, and a project crystallized:

I propose that Stack Exchange, GitHub, Meteor, Reddit, and any other company with lots of traffic and a strategic investment in Markdown, all work together to come up with an official Markdown specification, and standard test suites to validate Markdown implementations. We've all been working at cross purposes for too long, accidentally fragmenting Markdown while popularizing it.

We formed a small private working group with key representatives from GitHub, from Reddit, from Stack Exchange, from the open source community. We spent months hashing out the details and agreeing on the necessary changes to turn Markdown into a language you can parse without feeling like you just walked through a sewer – while preserving the simple, clear, ASCII email inspired spirit of Markdown.

We really struggled with this at Discourse, which is also based on Markdown, but an even more complex dialect than the one we built at Stack Overflow. In Discourse, you can mix three forms of markup interchangeably:

  • Markdown
  • HTML (safe subset)
  • BBCode (subset)

Discourse is primarily a JavaScript app, so naturally we needed a nice, compliant implementation of Markdown in JavaScript. Surely such a thing exists, yes? Nope. Even in 2012, we found zero JavaScript implementations of Markdown that could pass the only Markdown test suite I know of, MDTest. It isn't authoritative, it's a community created initiative that embodies its own decisions about rendering ambiguities in Markdown, but it's all we've got. We contributed many upstream fixes to markdown.js to make it pass MDTest – but it still only passes in our locally extended version.

As an open source project ourselves, we're perfectly happy contributing upstream code to improve it for everyone. But it's an indictment of the state of the Markdown ecosystem that any remotely popular implementation wasn't already testing itself against a formal spec and test suite. But who can blame them, because it didn't exist!

Well, now it does.

It took a while, but I'm pleased to announce that Standard Markdown is now finally ready for public review.

standardmarkdown.com

It's a spec, including embedded examples, and implementations in portable C and JavaScript. We strived mightily to stay true to the spirit of Markdown in writing it. The primary author, John MacFarlane, explains in the introduction to the spec:

Because Gruber’s syntax description leaves many aspects of the syntax undetermined, writing a precise spec requires making a large number of decisions, many of them somewhat arbitrary. In making them, I have appealed to existing conventions and considerations of simplicity, readability, expressive power, and consistency. I have tried to ensure that “normal” documents in the many incompatible existing implementations of markdown will render, as far as possible, as their authors intended. And I have tried to make the rules for different elements work together harmoniously. In places where different decisions could have been made (for example, the rules governing list indentation), I have explained the rationale for my choices. In a few cases, I have departed slightly from the canonical syntax description, in ways that I think further the goals of markdown as stated in that description.

Part of my contribution to the project is to host the discussion / mailing list for Standard Markdown in a Discourse instance.

talk.standardmarkdown.com

Fortunately, Discourse itself just reached version 1.0. If the only thing Standard Markdown does is help save a few users from the continuing horror that is mailing list web UI, we all win.

What I'm most excited about is that we got a massive contribution from the one person who, in my mind, was the most perfect person in the world to work on this project: John MacFarlane. He took our feedback and wrote the entire Standard Markdown spec and both implementations.

A lot of people know of John through his Pandoc project, which is amazing in its own right, but I found out about him because he built Babelmark. I learned to refer to Babelmark extensively while working on Stack Overflow and MarkdownSharp, a C# implementation of Markdown.

Here's how crazy Markdown is: to decide what the "correct" behavior is, you provide sample Markdown input to 20+ different Markdown parsers … and then pray that some consensus emerges in all their output. That's what Babelmark does.

Consider this simple Markdown example:

# Hello there

This is a paragraph.

- one
- two
- three
- four

1. pirate
2. ninja
3. zombie

Just for that, I count fifteen different rendered outputs from 22 different Markdown parsers.

In Markdown, we literally built a Tower of Babel.

Have I mentioned that it's a good idea for a language to have a formal specification and test suites? Maybe now you can see why that is.

Oh, and in his spare time, John is also the chair of the department of philosophy at the University of California, Berkeley. No big deal. While I don't mean to minimize the contributions of anyone to the Standard Markdown project, we all owe a special thanks to John.

Markdown is indeed everywhere. And that's a good thing. But it needs to be sane, parseable, and standard. That's the goal of Standard Markdown — but we need your help to get there. If you use Markdown on a website, ask what it would take for that site to become compatible with Standard Markdown; when you see the word "Markdown" you have the right to expect consistent rendering across all the websites you visit. If you implement Markdown, take a look at the spec, try to make your parser compatible with Standard Markdown, and discuss improvements or refinements to the spec.

Update: The project was renamed CommonMark. See my subsequent blog post.

[advertisement] How are you showing off your awesome? Create a Stack Overflow Careers profile and show off all of your hard work from Stack Overflow, Github, and virtually every other coding site. Who knows, you might even get recruited for a great new position!
Discussion

The "Just In Time" Theory of User Behavior

I've long believed that the design of your software has a profound impact on how users behave within your software. But there are two sides to this story:

  • Encouraging the "right" things by making those things intentionally easy to do.

  • Discouraging the "wrong" things by making those things intentionally difficult, complex, and awkward to do.

Whether the software is doing this intentionally, or completely accidentally, it's a fact of life: the path of least resistance is everyone's best friend. Learn to master this path, or others will master it for you.

For proof, consider Dan Ariely's new and amazing book, The (Honest) Truth About Dishonesty: How We Lie to Everyone – Especially Ourselves.

Indeed, let's be honest: we all lie, all the time. Not because we're bad people, mind you, but because we have to regularly lie to ourselves as a survival mechanism. You think we should be completely honest all the time? Yeah. Good luck with that.

But these healthy little white lies we learn to tell ourselves have a darker side. Have you ever heard this old adage?

One day, Peter locked himself out of his house. After a spell, the locksmith pulled up in his truck and picked the lock in about a minute.

“I was amazed at how quickly and easily this guy was able to open the door,” Peter said. The locksmith told him that locks are on doors only to keep honest people honest. One percent of people will always be honest and never steal. Another 1% will always be dishonest and always try to pick your lock and steal your television; locks won’t do much to protect you from the hardened thieves, who can get into your house if they really want to.

The purpose of locks, the locksmith said, is to protect you from the 98% of mostly honest people who might be tempted to try your door if it had no lock.

I had heard this expressed less optimistically before as

10% of people will never steal, 10% of people will always steal, and for everyone else … it depends.

The "it depends" part is crucial to understanding human nature, and that's what Ariely spends most of the book examining in various tests. If for most people, honesty depends, what exactly does it depend on? The experiments Ariely conducts prove again and again that most people will consistently and reliably cheat "just a little", to the extent that they can still consider themselves honest people. The gating factor isn't laws, penalties, or ethics. Surprisingly, that stuff has virtually no effect on behavior. What does, though, is whether they can personally still feel like they are honest people.

This is because they don't even consider it cheating – they're just taking a little extra, giving themselves a tiny break, enjoying a minor boost, because well, haven't they been working extra specially hard lately and earned it? Don't they of all people deserve something nice once in a while, and who would even miss this tiny amount? There's so much!

These little white lies are the path of least resistance. They are everywhere. If laws don't work, if ethics classes don't work, if severe penalties don't work, how do you encourage people to behave in a way that "feels" honest that is actually, you know, honest? Feelings are some pretty squishy stuff.

It's easier than you think.

My colleagues and I ran an experiment at the University of California, Los Angeles. We took a group of 450 participants, split them into two groups and set them loose on our usual matrix task. We asked half of them to recall the Ten Commandments and the other half to recall 10 books that they had read in high school.

Among the group who recalled the 10 books, we saw the typical widespread but moderate cheating. But in the group that was asked to recall the Ten Commandments, we observed no cheating whatsoever. We reran the experiment, reminding students of their schools' honor codes instead of the Ten Commandments, and we got the same result. We even reran the experiment on a group of self-declared atheists, asking them to swear on a Bible, and got the same no-cheating results yet again.

That's the good news: a simple reminder at the time of the temptation is usually all it takes for people to suddenly "remember" their honesty.

The bad news is Clippy was right.

In my experience, nobody reads manuals, nobody reads FAQs, and nobody reads tutorials. I am exaggerating a little here for effect, of course. Some A+ students will go out of their way to read these things. That's how they became A+ students, by naturally going the extra mile, and generally being the kind of users who teach themselves perfectly well without needing special resources to get there. When I say "nobody" I mean the vast overwhelming massive majority of people you would really, really want to read things like that. People who don't have the time or inclination to expend any effort at all other than the absolute minimum required, people who are most definitely not going to go the extra mile.

In other words, the whole world.

So how do you help people who, like us, just never seem to have the time to figure this stuff out becase they're, like, suuuuper busy and stuff?

You do it by showing them …

  • the minumum helpful reminder
  • at exactly the right time

This is what I've called the "Just In Time" theory of user behavior for years. Sure, FAQs and tutorials and help centers are great and all, but who has the time for that? We're all perpetual intermediates here, at best.

The closer you can get your software to practical, useful "Just In Time" reminders, the better you can help the users who are most in need. Not the A+ students who already read the FAQ, and studied the help center intently, but those users who never read anything. And now, thanks to Dan Ariely, I have the science to back this up. Even something as simple as putting your name on the top of a form to report auto insurance milage, rather than the bottom, resulted in a mysterious 10% increase in average miles reported. Having that little reminder right at the start that hey, your name is here on this form, inspired additional honesty. It works.

Did we use this technique on Stack Overflow and Stack Exchange? Indeed we did. Do I use this technique on Discourse? You bet, in even more places, because this is social discussion, not technical Q&A. We are rather big on civility, so we like to remind people when they post on Discourse they aren't talking to a computer or a robot, but a real person, a lot like you.

When's the natural time to remind someone of this? Not when they sign up, not when they're reading, but at the very moment they begin typing their first words in their first post. This is the moment of temptation when you might be super mega convinced that someone is Wrong on the Internet. So we put up a gentle little reminder Just In Time, right above where they are typing:

Then hopefully, as Dan Ariely showed us with honesty, this little reminder will tap into people's natural reserves of friendliness and civility, so cooler heads will prevail – and a few people are inspired to get along a little better than they did yesterday. Just because you're on the Internet doesn't mean you need to be yelling at folks 24/7.

We use this same technique a bunch of other places: if you are posting a lot but haven't set an avatar, if you are adding a new post to a particularly old conversation, if you are replying a bunch of times in the same topic, and so forth. Wherever we feel a gentle nudge might help, at the exact time the behavior is occurring.

It's important to understand that we use these reminders in Discourse not because we believe people are dumb; quite the contrary, we use them because we believe people are smart, civil, and interesting. Turns out everyone just needs to be reminded of that once in a while for it to continue to be true.

[advertisement] Stack Overflow Careers matches the best developers (you!) with the best employers. You can search our job listings or create a profile and even let employers find you.
Discussion

The Infinite Space Between Words

Computer performance is a bit of a shell game. You're always waiting for one of four things:

  • Disk
  • CPU
  • Memory
  • Network

But which one? How long will you wait? And what will you do while you're waiting?

Did you see the movie "Her"? If not, you should. It's great. One of my favorite scenes is the AI describing just how difficult it becomes to communicate with humans:

It's like I'm reading a book… and it's a book I deeply love. But I'm reading it slowly now. So the words are really far apart and the spaces between the words are almost infinite. I can still feel you… and the words of our story… but it's in this endless space between the words that I'm finding myself now. It's a place that's not of the physical world. It's where everything else is that I didn't even know existed. I love you so much. But this is where I am now. And this who I am now. And I need you to let me go. As much as I want to, I can't live your book any more.

I have some serious reservations about the work environment pictured in Her where everyone's spending all day creepily whispering to their computers, but there is deep fundamental truth in that one pivotal scene. That infinite space "between" what we humans feel as time is where computers spend all their time. It's an entirely different timescale.

The book Systems Performance: Enterprise and the Cloud has a great table that illustrates just how enormous these time differentials are. Just translate computer time into arbitrary seconds:

1 CPU cycle0.3 ns1 s
Level 1 cache access0.9 ns3 s
Level 2 cache access2.8 ns9 s
Level 3 cache access12.9 ns43 s
Main memory access120 ns6 min
Solid-state disk I/O50-150 μs2-6 days
Rotational disk I/O1-10 ms1-12 months
Internet: SF to NYC40 ms4 years
Internet: SF to UK81 ms8 years
Internet: SF to Australia183 ms19 years
OS virtualization reboot4 s423 years
SCSI command time-out30 s3000 years
Hardware virtualization reboot40 s4000 years
Physical system reboot5 m32 millenia

The above Internet times are kind of optimistic. If you look at the AT&T real time US internet latency chart, the time from SF to NYC is more like 70ms. So I'd double the Internet numbers in that chart.

Latency is one thing, but it's also worth considering the cost of that bandwidth.

Speaking of the late, great Jim Gray, he also had an interesting way of explaining this. If the CPU registers are how long it takes you to fetch data from your brain, then going to disk is the equivalent of fetching data from Pluto.

He was probably referring to traditional spinning rust hard drives, so let's adjust that extreme endpoint for today:

  • Distance to Pluto: 4.67 billion miles.
  • Latest fastest spinning HDD performance (49.7) versus latest fastest PCI Express SSD (506.8). That's an improvement of 10x.
  • New distance: 467 million miles.
  • Distance to Jupiter: 500 million miles.

So instead of travelling to Pluto to get our data from disk in 1999, today we only need to travel to … Jupiter.

That's disk performance over the last decade. How much faster did CPUs, memory, and networks get in the same time frame? Would a 10x or 100x improvement really make a dent in these vast infinite spaces in time that computers deal with?

To computers, we humans work on a completely different time scale, practically geologic time. Which is completely mind-bending. The faster computers get, the bigger this time disparity grows.

[advertisement] Stack Overflow Careers matches the best developers (you!) with the best employers. You can search our job listings or create a profile and even let employers find you.
Discussion

What Can Men Do?

(The title references Shanley Kane's post by the same name. This post represents my views on what men can do.)

It's no secret that programming is an incredibly male dominated field.

  • Figures vary, but somewhere from 20% to 29% of currently working programmers are female.

  • Less than 12% of Computer Science bachelor's degrees were awarded to women at US PhD-granting institutions in 2010.

So, on average, only about 1 out of every 5 working programmers you'll encounter will be female. You could say technology has a man problem.

In an earlier post I noted that many software developers I've known have traits of Aspergers. Aspergers is a spectrum disorder; the more severe the symptoms, the closer it is to autism. And did you know that autism skews heavily towards males at a 4:1 ratio?

Interesting. I might even go so far as to say some of those traits are what makes one good at programming.

That's the way it currently is. But is that the way it should be? I remember noticing that the workforce of the maternity ward at the hospital where our children were born was incredibly female dominated. Is there something inherently wrong with professions that naturally skew heavily male or female?

Consider this list of the most male and female dominated occupations in the Netherlands from 2004. It notes that:

In higher and academic level positions, men and women are more often represented equally. This pattern of employment has hardly changed over the last years.

Is programming a higher and academic level occupation? I'm not so sure, given that I've compared programmers to auto mechanics and plumbers in the past. And you'll notice squarely where those occupations are on the above graphs. There's nothing wrong with being an auto mechanic or a plumber (or a programmer, for that matter), but is there anything about those particular professions that demands, in the name of social justice, that there must be 50% male plumbers and 50% female plumbers?

For a counterpoint, here's a blog post from Sara J. Chipps. When I've e-mailed her in the past with my stupid questions on topics like this, she tries her best to educate me with empathy and compassion. That's why I love her.

This is an excerpt from a blog post she wrote in 2012 which answered my question:

Many people I meet ask me a variant of the question “I understand we want more women in technology, but why?” It’s a great question, and not at all something we should be offended by. Often men are afraid to ask questions like this for fear there will be backlash, and I think that fear can lead to stifling an important conversation.

Frankly, the Internet is thriving without women building it, why should that change? Three reasons:

1) Diversity leads to better products and results

As illustrated in this Cornell study along with many others, diversity improves performance, morale, and end product. More women engineers means building a better internet, and improving software that can service society as a whole. Building a better Internet is why I started doing software development in the first place. I think we can all agree this is of utmost importance.

2) The Internet is the largest recording of human history ever built

Right now the architecture for that platform is being built disproportionally by white and asian males. You’ve heard the phrase “he who writes history makes history”? We don’t yet know how this will affect future generations.

How can architecture be decidedly male? I like to refer to the anecdotal story of the Apple Store glass stairs. While visually appealing, there was one unforeseen consequence to their design: the large groups of strange men that spend hours each day standing under them looking up. As a woman, the first time I saw them I thought “thank god I’m not wearing a skirt today.” Such considerations were not taken in designing these stairs. I think it’s probable, if not easily predictable, that in a few years we will see such holes in the design of the web.

3) Women in 10 years need to be able to provide for themselves, and their families

Now, this reason is purely selfish on the part of women, but we all have mothers, and sisters, so I hope we can relate.

This year there are 6 million information technology jobs in the US, up from 628,600 in 1987 and 1.34 million in 1997. Right now jobs in technology have half the unemployment rate of the rest of the workforce. There is no sign this will change anytime soon. If growth continues at the current rate, it will not be long until women will not be able to sustain themselves if not involved in a technical field.

We have to start educating young girls about this now, or they may ultimately become the poorest demographic among us.

These are good reasons. I'm particularly fond of #1. Diversity in social perspectives is hugely valuable when building social software intended for, y'know, human beings of all genders, like Discourse and Stack Exchange. Also, I get really, really tired of all the aggressive mansplaining in software development. Yes, even my own. Sometimes it would be good to get some ladysplaining all mixed up in there for variety.

I suppose any effort to encourage more women to become software engineers should ideally start in childhood.

boy toys vs girl toys

Dolls? Pshaw. In our household, every child, male or female, is issued a regulation iPad at birth. You know, the best, most complex toy there is: a computer. And, shocker, I'm kind of weird about it – I religiously refer to it as a computer, never as an iPad. Never. Not once. Not gonna happen in my house. Branding is for marketing weasels. So the twin girls will run around, frantically calling out for their so-called "'puter". It puts a grin on my face every time. And when anything isn't here, Maisie has gotten in the habit of saying "dada chargin'". Where's the milk, Maisie? "dada chargin'".

But not everyone has the luxury of spawning their own processes and starting from boot. (You really should, though. It will kick your ass.)

What can you do?

If you're reading this, there's about an 80% chance that you're a man. So after you give me the secret man club handshake, let's talk about what we men can do, right now, today, to make programming a more welcoming profession for women.

  1. Abide by the Hacker School Rules

    Let's start with the freaking brilliant Hacker School rules. This cuts directly to the unfortunate but oh-so-common Aspergers tendencies in programmers I mentioned earlier:

    • No feigning surprise. "I can't believe you don't know what the stack is!"
    • No well-actuallys. "Well, actually, you can do that without a regular expression."
    • No back seat driving. Don't intermittently lob advice across the room.
    • No subtle sexism via public debate.

    Does any of this sound familiar? Because it should. Oh God does this sound familar. Just read the whole set of Hacker School guidelines and recognize your natural tendencies, and try to rein them in. That's all I'm proposing.

    Well, actually, I'll be proposing a few more things.

  2. Really listen. What? I SAID LISTEN.

    Remember this scene in Fight Club?

    This is why I loved the support groups so much, if people thought you were dying, they gave you their full attention. If this might be the last time they saw you, they really saw you. Everything else about their checkbook balance and radio songs and messy hair went out the window. You had their full attention. People listened instead of just waiting for their turn to speak. And when they spoke, they weren't just telling you a story. When the two of you talked, you were building something, and afterward you were both different than before.

    Guilty as charged.

    My wife is a scientist, and she complains about this happening a lot at her work. I don't even think this one is about sexism, it's about basic respect. What does respect mean? Well, a bunch of things, but let's start with openly listening to people and giving them our full attention when they talk to us – rather than just waiting for our turn to speak.

    Let's shut up and listen quietly with the same thoughtfulness that we wish others would listen to us. We'll get our turn. We always do, don't we?

  3. If you see bad behavior from other men, speak up.

    It's not other people's job to make sure that everyone enjoys a safe, respectful, civil environment at work and online.

    It's my job. It's your job. It is our job.

    There is no mythical men's club where it is OK to be a jerk to women. If you see any behavior that gives you pause, behavior that makes you wonder "is that OK?", behavior that you'd be uncomfortable with directed toward your sister, your wife, your daughter – speak up. Honestly, as one man to another. And if that doesn't work for whatever reason, escalate.

  4. Don't attempt romantic relationships at work.

    Do you run a company? Institute a no-dating rule as policy. Yeah, I know, you can't truly enforce it, but it should still be the official company policy. And whether the place where you work has this policy or not, you should have it on a personal level.

    I'm sorry I have to be that guy who dumps on true love, but let's be honest: the odds of any random office romance working out are pretty slim. And when it doesn't, how will you handle showing up to work every day and seeing this person? Will there be Capulet vs Montague drama? The women usually get the rough end of this deal, too, because men aren't good at handling the inevitable rejection.

    Just don't do it. Have all the romantic relationships you want outside work, but do not bring it to work.

  5. No drinking at work events.

    I think it is very, very unwise for companies to have a culture associated with drinking and the lowered inhibitions that come with drinking. I've heard some terrifyingly awful stories that I don't even want to link to here. Men, plus women, plus alcohol is a great recipe for college. That's about all I remember from college, in fact. But as a safe work environment for women? Not so much.

    If you want to drink, be my guest. Drink. You're a grown up. I'm not the boss of you. But don't drink in a situation or event that is officially connected with work in any way. That should absolutely be your personal and company policy – no exceptions.

There you have it. Five relatively simple things you, I, and all other working male programmers can do to help encourage a better environment for men and women in software plumbing. I mean engineering.

So let's get to it.

(I haven't listed anything here about mentoring. That's because I am an awful mentor. But please do feel free to mention good resources, like Girl Develop It, that encourage mentoring of female software engineers by people that are actually good at it, in the comments.)

[advertisement] How are you showing off your awesome? Create a Stack Overflow Careers profile and show off all of your hard work from Stack Overflow, Github, and virtually every other coding site. Who knows, you might even get recruited for a great new position!
Discussion

Three Things

I've expressed my disillusionment with to-do lists before.

But let's try something simpler, a little experiment. What do you use to keep track of what you need to do? Hold it up, so I can see it. Humor me.

Seriously! No no no, hold it closer, near the screen here. Let me look at it. Let me get a good, long look at it.

Now imagine me slapping this thing out of your hand.

don't go there

I just want to make a point, not break your fancy whatchamacallit. So pretend I slapped it into a soft fluffy pillow on the ground, not the hard concrete of the sidewalk. Though I probably should have.

Whatever that thing is, it's a crutch. You don't need it. It's hurting you more than it is helping. Get rid of it.

Instead, ask yourself this:

What three things do you need to do today?

You should be able to instantly answer this simple question, each day, every day, for the rest of your life. Without any tools other than the brain you were born with.

If you don't have this skill, develop it. Practice, starting today. Right now.

What are you doing right now? Is it going to somehow result in one of those three things getting done today? Will this you get you to where you need to be by the end of the day?

I'm not asking you to admonish yourself or to make any changes to your routine. Just keep it simple, focus on the important things, and add a little layer of awareness.

So. Two items left. I'm doing pretty good today.

[advertisement] Hiring developers? Post your open positions with Stack Overflow Careers and reach over 20MM awesome devs already on Stack Overflow. Create your satisfaction-guaranteed job listing today!
Discussion