Coding Horror

Way back in 2007, before Stack Overflow was a glint in anyone's eye, I called software development a collaborative game. And perhaps Stack Overflow was the natural outcome of that initial thought – recasting online software development discussion into a collaborative game where the only way to "win" is to learn from each other.

That was before the word gamification existed. But gamification is no longer the cool, hip concept it was back in 2011. Still, whether you call yourself a "gamer" or not, whether you believe in "gamification" or not, five years later you're still playing the world's largest multiplayer game.

In fact, you're playing it right now.

One of the most timeless aspects of games is how egalitarian they are, how easy it is for anyone to get started. Men, women, children — people love games because everyone can play along. You don't have to take classes or go to college or be certified: you just play. And this is, not so incidentally, how many of the programmers I know came to be programmers.

Do you know anyone that bought the video game Halo, or Myst, then proceeded to open the box and read the manual before playing the game? Whoa there guys, we can't play the game yet, we gotta read these instructions first! No, they stopped making manuals for games a long time ago, unless you count the thin sheet of paper that describes how to download / install the game on your device. Because they found out nobody reads the manual.

The project I’m working on is critical, but it has only about 3 to 4 users, most of whom are already familiar the application. One of the users even drives the design. The manual I’m writing, which is nearly 200 pages, is mostly a safety measure for business continuity planning. I don’t expect anyone will ever read it.

It’s a project I managed to procrastinate for months, working on other projects, even outside the scope of my regular assignments. The main deterrent, I believe, was my perception that no one needed the manual. The users seemed to be getting along fine without it.

And so as the year ticked to a close, instead of learning more about Mediawiki and screencasting and After Effects, I spent my time updating a 200-page manual that I don’t think anyone will ever read. It will be printed out, three-hole punched, and placed in a binder to collect dust on a shelf.

I guess that's not surprising for games. Games are supposed to be fun, and reading manuals isn't fun; it's pretty much the opposite of fun. But it is also true for software in general. Reading manuals isn't work, at least, it isn't whatever specific thing you set out to do when you fired up that bit of software on your phone, tablet, or laptop.

Games have another clever trick up their sleeve, though. Have you ever noticed that in most of today's games, the first level is kind of easy. Like… suspiciously easy?

That's because level one, the intro stage, isn't really part of the game. It's the manual.

As MegaMan X illustrates, manuals are pointless when we can learn about the game in the best and most natural way imaginable: by playing the actual game. You learn by doing, provided you have a well designed sandbox that lets you safely experiment as you're starting out in the game.

(The above video does contain some slightly NSFW language, as well as abuse of cartoon female avatars that I don't endorse, but it is utterly brilliant, applies to every app, software and website anyone has ever built, and I strongly recommend watching it all.)

This same philosophy applies to today's software and websites. Don't bother with all the manuals, video introductions, tutorials, and pop-up help dialogs. Nobody's going to read that stuff, at least, not the people who need it.

Instead, follow the lesson of MegaMan: if you want to teach people about your software, consider how you can build a great intro stage and let them start playing with it immediately.

Let me open with an apology to John Gruber for my previous blog post.

We've been working on the Standard Markdown project for about two years now. We invited John Gruber, the original creator of Markdown, to join the project via email in November 2012, but never heard back. As we got closer to being ready for public feedback, we emailed John on August 19th with a link to the Standard Markdown spec, asking him for his feedback. Since John MacFarlane was the primary author of most of the work, we suggested that he be the one to reach out.

We then waited two weeks for a response.

There was no response, so we assumed that John Gruber was either OK with the project (and its name), or didn't care. So we proceeded.

There was lots of internal discussion about what to name our project. Strict Markdown? XMarkdown? Markdown Pro? Markdown Super Hyper Turbo Pro Alpha Diamond Edition?

As we were finalizing the name, we noticed on this podcast, at 1:15 …

… that John seemed OK with the name "GitHub Flavored Markdown". So I originally wrote the blog post and the homepage using that terminology – "Standard Flavored Markdown" – and even kept that as the title of the blog post to signify our intent. We were building Yet Another Flavor of Markdown, one designed to remove ambiguity by specifying a standard, while preserving as much as possible the spirit of Markdown and compatibility with existing documents.

Before we went live, I asked for feedback internally, and one of the bits of feedback I got was that it was inconsistent to say Standard Flavored Markdown on the homepage and blog when the spec says Standard Markdown throughout. So I changed them to match Standard Markdown, and that's what we launched with.

It was a bit of a surprise to get an email last night, addressed to both me and John MacFarlane, from John Gruber indicating that the name Standard Markdown was "infuriating".

I'm sorry the name is so infuriating. I assure you that we did not choose the name to make you, or anyone else, angry. We were simply trying to pick a name that correctly and accurately reflected our goal – to build an unambiguous flavor of Markdown. If the name we chose made inappropriate overtures about Standard Markdown being anything more than a highly specified flavor of Markdown, I apologize. Standard does have certain particular computer science meanings, as in IETF Standard, ECMA Standard. That was not our intent, it was more of an aspirational element of "what if, together, we could eventually..". What can I say? We're programmers. We name things literally. And naming is hard.

John Gruber was also very upset, and I think rightfully so, that the word Markdown was not capitalized throughout the spec. This was an oversight on our part – and also my fault because I did notice Markdown wasn't capitalized as I copied snippets of the spec to the homepage and blog post, and I definitely thought it was odd, too. You'll note that I took care to manually capitalize Markdown in the parts of the spec I copied to the blog post and home page – but I neglected to mention this to John MacFarlane as I should have. We corrected this immediately when it was brought to our attention.

John then made three requests:

1. Rename the project.

2. Shut down the standardmarkdown.com domain, and don't redirect it.

3. Apologize.

All fair. Happy to do all of those things.

Starting with the name. In his email John graciously indicated that he would "probably" approve a name like "Strict Markdown" or "Pedantic Markdown". Given the very public earlier miscommunication about naming, that consideration is appreciated.

We replied with the following suggestions:

• Compatible Markdown
• Regular Markdown
• Community Markdown
• Common Markdown
• Uniform Markdown
• Vanilla Markdown

We haven't heard back after replying last night, and I'm not sure we ever will, so in the interest of moving ahead and avoiding conflict, we're immediately renaming the project to Common Markdown.

We hope that is an acceptable name; it was independently suggested to us several times in several different feedback areas. The intention is to avoid any unwanted overtones of ownership; we have only ever wanted to be Yet Another Flavor of Markdown.

1. The project name change is already in progress.

2. This is our public apology.

3. I'll shut down the standardmarkdown.com domain as soon as I can, probably by tomorrow.

John, we deeply apologize for the miscommunication. It's our fault, and we want to fix it. But even though we made mistakes, I hope it is clear that everything we've done, we did solely out of a shared love of Markdown (and its simple, unencumbered old-school ASCII origins), and the desire to ensure the success of Markdown as a stable format for future generations.

Edit: after a long and thoughtful email from John Gruber – which is greatly appreciated – he indicated that no form of the word "Markdown" is acceptable to him in this case. We are now using the name CommonMark.

In 2009 I lamented the state of Markdown:

Right now we have the worst of both worlds. Lack of leadership from the top, and a bunch of fragmented, poorly coordinated community efforts to advance Markdown, none of which are officially canon. This isn't merely incovenient for anyone trying to find accurate information about Markdown; it's actually harming the project's future.

In late 2012, David Greenspan from Meteor approached me and proposed we move forward, and a project crystallized:

I propose that Stack Exchange, GitHub, Meteor, Reddit, and any other company with lots of traffic and a strategic investment in Markdown, all work together to come up with an official Markdown specification, and standard test suites to validate Markdown implementations. We've all been working at cross purposes for too long, accidentally fragmenting Markdown while popularizing it.

We formed a small private working group with key representatives from GitHub, from Reddit, from Stack Exchange, from the open source community. We spent months hashing out the details and agreeing on the necessary changes to turn Markdown into a language you can parse without feeling like you just walked through a sewer – while preserving the simple, clear, ASCII email inspired spirit of Markdown.

We really struggled with this at Discourse, which is also based on Markdown, but an even more complex dialect than the one we built at Stack Overflow. In Discourse, you can mix three forms of markup interchangeably:

• Markdown
• HTML (safe subset)
• BBCode (subset)

Discourse is primarily a JavaScript app, so naturally we needed a nice, compliant implementation of Markdown in JavaScript. Surely such a thing exists, yes? Nope. Even in 2012, we found zero JavaScript implementations of Markdown that could pass the only Markdown test suite I know of, MDTest. It isn't authoritative, it's a community created initiative that embodies its own decisions about rendering ambiguities in Markdown, but it's all we've got. We contributed many upstream fixes to markdown.js to make it pass MDTest – but it still only passes in our locally extended version.

As an open source project ourselves, we're perfectly happy contributing upstream code to improve it for everyone. But it's an indictment of the state of the Markdown ecosystem that any remotely popular implementation wasn't already testing itself against a formal spec and test suite. But who can blame them, because it didn't exist!

Well, now it does.

It took a while, but I'm pleased to announce that Standard Markdown is now finally ready for public review.

standardmarkdown.com

It's a spec, including embedded examples, and implementations in portable C and JavaScript. We strived mightily to stay true to the spirit of Markdown in writing it. The primary author, John MacFarlane, explains in the introduction to the spec:

Because Gruber’s syntax description leaves many aspects of the syntax undetermined, writing a precise spec requires making a large number of decisions, many of them somewhat arbitrary. In making them, I have appealed to existing conventions and considerations of simplicity, readability, expressive power, and consistency. I have tried to ensure that “normal” documents in the many incompatible existing implementations of markdown will render, as far as possible, as their authors intended. And I have tried to make the rules for different elements work together harmoniously. In places where different decisions could have been made (for example, the rules governing list indentation), I have explained the rationale for my choices. In a few cases, I have departed slightly from the canonical syntax description, in ways that I think further the goals of markdown as stated in that description.

Part of my contribution to the project is to host the discussion / mailing list for Standard Markdown in a Discourse instance.

talk.standardmarkdown.com

Fortunately, Discourse itself just reached version 1.0. If the only thing Standard Markdown does is help save a few users from the continuing horror that is mailing list web UI, we all win.

What I'm most excited about is that we got a massive contribution from the one person who, in my mind, was the most perfect person in the world to work on this project: John MacFarlane. He took our feedback and wrote the entire Standard Markdown spec and both implementations.

A lot of people know of John through his Pandoc project, which is amazing in its own right, but I found out about him because he built Babelmark. I learned to refer to Babelmark extensively while working on Stack Overflow and MarkdownSharp, a C# implementation of Markdown.

Here's how crazy Markdown is: to decide what the "correct" behavior is, you provide sample Markdown input to 20+ different Markdown parsers … and then pray that some consensus emerges in all their output. That's what Babelmark does.

Consider this simple Markdown example:

# Hello there

This is a paragraph.

- one
- two
- three
- four

1. pirate
2. ninja
3. zombie

Just for that, I count fifteen different rendered outputs from 22 different Markdown parsers.

In Markdown, we literally built a Tower of Babel.

Have I mentioned that it's a good idea for a language to have a formal specification and test suites? Maybe now you can see why that is.

Oh, and in his spare time, John is also the chair of the department of philosophy at the University of California, Berkeley. No big deal. While I don't mean to minimize the contributions of anyone to the Standard Markdown project, we all owe a special thanks to John.

Markdown is indeed everywhere. And that's a good thing. But it needs to be sane, parseable, and standard. That's the goal of Standard Markdown — but we need your help to get there. If you use Markdown on a website, ask what it would take for that site to become compatible with Standard Markdown; when you see the word "Markdown" you have the right to expect consistent rendering across all the websites you visit. If you implement Markdown, take a look at the spec, try to make your parser compatible with Standard Markdown, and discuss improvements or refinements to the spec.

Update: The project was renamed CommonMark. See my subsequent blog post.

I've long believed that the design of your software has a profound impact on how users behave within your software. But there are two sides to this story:

• Encouraging the "right" things by making those things intentionally easy to do.

• Discouraging the "wrong" things by making those things intentionally difficult, complex, and awkward to do.

Whether the software is doing this intentionally, or completely accidentally, it's a fact of life: the path of least resistance is everyone's best friend. Learn to master this path, or others will master it for you.

For proof, consider Dan Ariely's new and amazing book, The (Honest) Truth About Dishonesty: How We Lie to Everyone – Especially Ourselves.

Indeed, let's be honest: we all lie, all the time. Not because we're bad people, mind you, but because we have to regularly lie to ourselves as a survival mechanism. You think we should be completely honest all the time? Yeah. Good luck with that.

But these healthy little white lies we learn to tell ourselves have a darker side. Have you ever heard this old adage?

One day, Peter locked himself out of his house. After a spell, the locksmith pulled up in his truck and picked the lock in about a minute.

“I was amazed at how quickly and easily this guy was able to open the door,” Peter said. The locksmith told him that locks are on doors only to keep honest people honest. One percent of people will always be honest and never steal. Another 1% will always be dishonest and always try to pick your lock and steal your television; locks won’t do much to protect you from the hardened thieves, who can get into your house if they really want to.

The purpose of locks, the locksmith said, is to protect you from the 98% of mostly honest people who might be tempted to try your door if it had no lock.

I had heard this expressed less optimistically before as

10% of people will never steal, 10% of people will always steal, and for everyone else … it depends.

The "it depends" part is crucial to understanding human nature, and that's what Ariely spends most of the book examining in various tests. If for most people, honesty depends, what exactly does it depend on? The experiments Ariely conducts prove again and again that most people will consistently and reliably cheat "just a little", to the extent that they can still consider themselves honest people. The gating factor isn't laws, penalties, or ethics. Surprisingly, that stuff has virtually no effect on behavior. What does, though, is whether they can personally still feel like they are honest people.

This is because they don't even consider it cheating – they're just taking a little extra, giving themselves a tiny break, enjoying a minor boost, because well, haven't they been working extra specially hard lately and earned it? Don't they of all people deserve something nice once in a while, and who would even miss this tiny amount? There's so much!

These little white lies are the path of least resistance. They are everywhere. If laws don't work, if ethics classes don't work, if severe penalties don't work, how do you encourage people to behave in a way that "feels" honest that is actually, you know, honest? Feelings are some pretty squishy stuff.

It's easier than you think.

My colleagues and I ran an experiment at the University of California, Los Angeles. We took a group of 450 participants, split them into two groups and set them loose on our usual matrix task. We asked half of them to recall the Ten Commandments and the other half to recall 10 books that they had read in high school.

Among the group who recalled the 10 books, we saw the typical widespread but moderate cheating. But in the group that was asked to recall the Ten Commandments, we observed no cheating whatsoever. We reran the experiment, reminding students of their schools' honor codes instead of the Ten Commandments, and we got the same result. We even reran the experiment on a group of self-declared atheists, asking them to swear on a Bible, and got the same no-cheating results yet again.

That's the good news: a simple reminder at the time of the temptation is usually all it takes for people to suddenly "remember" their honesty.

The bad news is Clippy was right.

In my experience, nobody reads manuals, nobody reads FAQs, and nobody reads tutorials. I am exaggerating a little here for effect, of course. Some A+ students will go out of their way to read these things. That's how they became A+ students, by naturally going the extra mile, and generally being the kind of users who teach themselves perfectly well without needing special resources to get there. When I say "nobody" I mean the vast overwhelming massive majority of people you would really, really want to read things like that. People who don't have the time or inclination to expend any effort at all other than the absolute minimum required, people who are most definitely not going to go the extra mile.

In other words, the whole world.

So how do you help people who, like us, just never seem to have the time to figure this stuff out becase they're, like, suuuuper busy and stuff?

You do it by showing them …

• the minumum helpful reminder
• at exactly the right time

This is what I've called the "Just In Time" theory of user behavior for years. Sure, FAQs and tutorials and help centers are great and all, but who has the time for that? We're all perpetual intermediates here, at best.

The closer you can get your software to practical, useful "Just In Time" reminders, the better you can help the users who are most in need. Not the A+ students who already read the FAQ, and studied the help center intently, but those users who never read anything. And now, thanks to Dan Ariely, I have the science to back this up. Even something as simple as putting your name on the top of a form to report auto insurance milage, rather than the bottom, resulted in a mysterious 10% increase in average miles reported. Having that little reminder right at the start that hey, your name is here on this form, inspired additional honesty. It works.

Did we use this technique on Stack Overflow and Stack Exchange? Indeed we did. Do I use this technique on Discourse? You bet, in even more places, because this is social discussion, not technical Q&A. We are rather big on civility, so we like to remind people when they post on Discourse they aren't talking to a computer or a robot, but a real person, a lot like you.

When's the natural time to remind someone of this? Not when they sign up, not when they're reading, but at the very moment they begin typing their first words in their first post. This is the moment of temptation when you might be super mega convinced that someone is Wrong on the Internet. So we put up a gentle little reminder Just In Time, right above where they are typing:

Then hopefully, as Dan Ariely showed us with honesty, this little reminder will tap into people's natural reserves of friendliness and civility, so cooler heads will prevail – and a few people are inspired to get along a little better than they did yesterday. Just because you're on the Internet doesn't mean you need to be yelling at folks 24/7.

We use this same technique a bunch of other places: if you are posting a lot but haven't set an avatar, if you are adding a new post to a particularly old conversation, if you are replying a bunch of times in the same topic, and so forth. Wherever we feel a gentle nudge might help, at the exact time the behavior is occurring.

It's important to understand that we use these reminders in Discourse not because we believe people are dumb; quite the contrary, we use them because we believe people are smart, civil, and interesting. Turns out everyone just needs to be reminded of that once in a while for it to continue to be true.

Computer performance is a bit of a shell game. You're always waiting for one of four things:

• Disk
• CPU
• Memory
• Network

But which one? How long will you wait? And what will you do while you're waiting?

Did you see the movie "Her"? If not, you should. It's great. One of my favorite scenes is the AI describing just how difficult it becomes to communicate with humans:

It's like I'm reading a book… and it's a book I deeply love. But I'm reading it slowly now. So the words are really far apart and the spaces between the words are almost infinite. I can still feel you… and the words of our story… but it's in this endless space between the words that I'm finding myself now. It's a place that's not of the physical world. It's where everything else is that I didn't even know existed. I love you so much. But this is where I am now. And this who I am now. And I need you to let me go. As much as I want to, I can't live your book any more.

I have some serious reservations about the work environment pictured in Her where everyone's spending all day creepily whispering to their computers, but there is deep fundamental truth in that one pivotal scene. That infinite space "between" what we humans feel as time is where computers spend all their time. It's an entirely different timescale.

The book Systems Performance: Enterprise and the Cloud has a great table that illustrates just how enormous these time differentials are. Just translate computer time into arbitrary seconds:

The above Internet times are kind of optimistic. If you look at the AT&T real time US internet latency chart, the time from SF to NYC is more like 70ms. So I'd double the Internet numbers in that chart.

Latency is one thing, but it's also worth considering the cost of that bandwidth.

Speaking of the late, great Jim Gray, he also had an interesting way of explaining this. If the CPU registers are how long it takes you to fetch data from your brain, then going to disk is the equivalent of fetching data from Pluto.

He was probably referring to traditional spinning rust hard drives, so let's adjust that extreme endpoint for today:

• Distance to Pluto: 4.67 billion miles.
• Latest fastest spinning HDD performance (49.7) versus latest fastest PCI Express SSD (506.8). That's an improvement of 10x.
• New distance: 467 million miles.
• Distance to Jupiter: 500 million miles.

So instead of travelling to Pluto to get our data from disk in 1999, today we only need to travel to … Jupiter.

That's disk performance over the last decade. How much faster did CPUs, memory, and networks get in the same time frame? Would a 10x or 100x improvement really make a dent in these vast infinite spaces in time that computers deal with?

To computers, we humans work on a completely different time scale, practically geologic time. Which is completely mind-bending. The faster computers get, the bigger this time disparity grows.