Coding Horror

Occasionally, startups will ask me for advice. That's a shame, because I am a terrible person to ask for advice. The conversation usually goes something like this:

We'd love to get your expert advice on our thing.

I probably don't use your thing. Even if I tried your thing out and I gave you my so-called Expert advice, how would it matter? Anyway, why are you asking me? Why don't you ask your community what they think of your thing?

And if you don't have a community of users and customers around your thing, well, there's your problem right there. Go fix that.

Like I said, I don't get asked for advice too often. But for what it's worth, it is serious advice. And the next question they ask always strikes fear into my heart.

You're so right! We need a place for online community around our thing. What software should we use?

This is the part where I start playing sad trombone in my head. Because all your software options for online community are, quite frankly, terrible. Stack Exchange? We only do strict, focused Q&A there and you'd have to marshal your proposal through Area 51. Get Satisfaction, UserVoice, Desk, etcetera? Sorry, customer support isn't the same as community. Mailing lists? Just awful.

Forum software? Maybe. Let's see, it's 2013, has forum software advanced at all in the last ten years?

I'm thinking no.

Forums are the dark matter of the web, the B-movies of the Internet. But they matter. To this day I regularly get excellent search results on forum pages for stuff I'm interested in. Rarely a day goes by that I don't end up on some forum, somewhere, looking for some obscure bit of information. And more often than not, I find it there.

There's an amazing depth of information on forums.

• A 12 year old girl who finds a forum community of rabid enthusiasts willing to help her rebuild a Fiero from scratch? Check.
• The most obsessive breakdown of Lego collectible minifig kits you'll find anywhere on the Internet? Check.
• Some of the most practical information on stunt kiting in the world? Check.
• The only place I could find with scarily powerful squirt gun instructions and advice? Check.
• The underlying research for a New Yorker article outing a potential serial marathon cheater? Check.

I could go on and on. As much as existing forum software is inexplicably and terrifyingly awful after all these years, it is still the ongoing basis for a huge chunk of deeply interesting information on the Internet. These communities are incredibly passionate about incredibly obscure things. They aren't afraid to let their freak flag fly, and the world is a better place for it.

At Stack Exchange, one of the tricky things we learned about Q&A is that if your goal is to have an excellent signal to noise ratio, you must suppress discussion. Stack Exchange only supports the absolute minimum amount of discussion necessary to produce great questions and great answers. That's why answers get constantly re-ordered by votes, that's why comments have limited formatting and length and only a few display, and so forth. Almost every design decision we made was informed by our desire to push discussion down, to inhibit it in every way we could. Spare us the long-winded diatribe, just answer the damn question already.

After spending four solid years thinking of discussion as the established corrupt empire, and Stack Exchange as the scrappy rebel alliance, I began to wonder – what would it feel like to change sides? What if I became a champion of random, arbitrary discussion, of the very kind that I'd spent four years designing against and constantly lecturing users on the evil of?

I already built an X-Wing; could I build a better Tie Fighter?

If you're wondering what all those sly references to Tie Fighters were about in my previous blog posts and tweets, now you know. All hail the Emperor, and by the way, what's your favorite programming food?

Today we announce the launch of Discourse, a next-generation, 100% open source discussion platform built for the next decade of the Internet.

The goal of the company we formed, Civilized Discourse Construction Kit, Inc., is exactly that – to raise the standard of civilized discourse on the Internet through seeding it with better discussion software:

• 100% open source and free to the world, now and forever.
• Feels great to use. It's fun.
• Designed for hi-resolution tablets and advanced web browsers.
• Built in moderation and governance systems that let discussion communities protect themselves from trolls, spammers, and bad actors – even without official moderators.

Our amazingly talented team has been working on Discourse for almost a year now, and although like any open source software it's never entirely done, we believe it is already a generation ahead of any other forum software we've used.

I greatly admire what WordPress did for the web; to say that we want to be the WordPress of forums is not a stretch at all. We're also serious about this eventually being a viable open-source business, in the mold of WordPress. And we're not the only people who believe in the mission: I'm proud to announce that we have initial venture capital funding from First Round, Greylock, and SV Angel. We're embarking on a five year mission to improve the fabric of the Internet, and we're just getting started. Let a million discussions bloom!

So now, when someone says to me …

You're so right! We need a place for community around our thing. What software should we use?

I can reply without hesitation.

And hopefully, so can you.

When Joel Spolsky, my business partner on Stack Overflow and Stack Exchange, asked me what I wanted to do after I left Stack Exchange, I distinctly remember mentioning Aaron Swartz. That's what Aaron was to us hackers: an exemplar of the noble, selfless behavior and positive action that all hackers aspire to – but very few actually achieve.

And now, tragically, Aaron is gone at the tender age of 26. He won't be achieving anything any more.

I never knew Aaron, but I knew Aaron.

Most of all, I am disappointed.

I'm deeply disappointed in myself, for not understanding just how bitterly unfair the government charges were against Aaron. Perhaps the full, grotesque details couldn't be revealed for a pending legal case. But we should have been outraged. I am gutted that I did not contribute to his defense in any way, either financially or by writing about it here. I blindly assumed he would prevail, as powerful activists on the side of fairness, openness, and freedom are fortunate enough to often do in our country. I was wrong.

I'm disappointed in our government, for going to such lengths to make an example of someone who was so obviously a positive force. Someone who actively worked to change the world for the better in everything he did, starting from the age of 12. There was no evil in this man. And yet the absurd government case against him was cited by his family as directly contributing to his death.

I'm frustrated by the idea that martyrdom works. The death of Aaron Swartz is now turning into an effective tool for change, a rallying cry, proving the perverse lesson that nobody takes an issue seriously until a great person dies for the cause. The idea that Aaron killing himself was a viable strategy, more than going on to prevail in this matter and so many more in his lifetime, makes me incredibly angry.

But also, I must admit that I am a little disappointed in Aaron. I understand that depression is a serious disease that can fell any person, however strong. But he chose the path of the activist long ago. And the path of the activist is to fight, for as long and as hard as it takes, to effect change. Aaron had powerful friends, a powerful support network, and a keen sense of moral cause that put him in the right. That's how he got that support network of powerful friends and fellow activists in the first place.

It is appropriate to write about Aaron on Martin Luther King day, because he too was a tireless activist for moral causes.

I hope you are able to see the distinction I am trying to point out. In no sense do I advocate evading or defying the law, as would the rabid segregationist. That would lead to anarchy. One who breaks an unjust law must do so openly, lovingly, and with a willingness to accept the penalty. I submit that an individual who breaks a law that conscience tells him is unjust, and who willingly accepts the penalty of imprisonment in order to arouse the conscience of the community over its injustice, is in reality expressing the highest respect for law.

Let's be clear that the penalty in Aaron's case was grossly unfair, bordering on corrupt. I've been a part of exactly one trial, but I can't even imagine having the full resources of the US Government brought to bear against me, with extreme prejudice, for a year or more. His defense was estimated to cost millions. The idea that such an engaged citizen would be forever branded a felon – serving at least some jail time and stripped of the most fundamental citizenship right, the ability to vote – must have weighed heavily on Aaron. And Aaron was no stranger to depresson, having written about it on his blog many times, even penning a public will of sorts on his blog all the way back in 2002.

I think about ragequitting a lot.

Rage Quit, also seen as RageQuit in one word, is Internet slang commonly used to describe the act of suddenly quitting a game or chatroom after either an argument, extreme frustration, or loss of the game.

At least one user ragequits Stack Exchange every six months, because our rules are strict. Some people don't like rules, and can respond poorly when confronted by the rules of the game they choose to play. It came up often enough that we had to create even more rules to deal with it. I was forced to think about ragequitting.

I say this not as a person who wishes to judge Aaron Swartz. I say it as a fellow gamer who has also considered playing the same move quite recently. To the point that I – like Aaron himself, I am sure – was actively researching it. But the more I researched, the more I thought about it, the more it felt like what it really was: giving up. And the toll on friends and family would be unimaginably, unbearably heavy.

What happened to Aaron was not fair. Not even a little. But this is the path of the activist. The greater the injustice, the greater wrong undone when you ultimately prevail. And I am convinced, absolutely and utterly convinced, that Aaron would have prevailed. He would have gone on to do so many other great things. It is our great failing that we did not provide Aaron the support network he needed to see this. All we can do now is continue the mission he started and lobby for change to our corrupt government practices of forcing plea bargains.

It gets dark sometimes. I know it does. I'm right there with you. But do not, under any circumstances, give anyone the satisfaction of seeing you ragequit. They don't deserve it. Play other, better moves – and consider your long game.

It's been six years since I wrote Discussions: Flat or Threaded? and, despite a bunch of evolution on the web since then, my opinion on this has not fundamentally changed.

If anything, my opinion has strengthened based on the observed data: precious few threaded discussion models survive on the web. Putting aside Usenet as a relic and artifact of the past, it is rare to find threaded discussions of any kind on the web today; for web discussion communities that are more than ten years old, the vast majority are flat as a pancake.

I'm game for trying anything new, I mean, I even tried Google Wave. But the more I've used threaded discussions of any variety, the less I like them. I find precious few redeeming qualities, while threading tends to break crucial parts of discussion like reading and replying in deep, fundamental, unfixable ways. I have yet to discover a threaded discussion design that I can tolerate long term.

A part of me says this is software Darwinism in action: threaded discussion is ultimately too complex to survive on the public Internet.

Before threaded discussion fans bring out their pitchforks and torches, I fully acknowledge that aspects of threading can be useful in certain specific situations. I will get to that. I know I'm probably wasting my time even attempting to say this, but please: keep reading before commenting. Ideally, read the whole article before commenting. Like Parappa, I gotta believe!

Before I defend threaded discussion, let's enumerate the many problems it brings to the table:

1. It's a tree.
   

   Poems about trees are indeed lovely, as Joyce Kilmer promised us, but data of any kind represented as a tree … isn't. Rigid hierarchy is generally not how the human mind works, and the strict parent-child relationship it enforces is particularly terrible for fluid human group discussion. Browsing a tree is complicated, because you have to constantly think about what level you're at, what's expanded, what's collapsed … there's always this looming existential crisis of where the heck am I? Discussion trees force me to spend too much time mentally managing that two-dimensional tree more than the underlying discussion.

2. Where did that reply go?
   

   In a threaded discussion, replies can arrive any place in the tree at any time. How do you know if there are new replies? Where do you find them? Only if you happen to be browsing the tree at the right place at the right time. It's annoying to follow discussions over time when new posts keep popping up anywhere in the middle of the big reply tree. And God help you if you accidentally reply at the wrong level of the tree; then you're suddenly talking to the wrong person, or maybe nobody at all. It absolutely kills me that there might be amazing, insightful responses buried somewhere in the middle of a reply chain that I will never be able to find.

3. It pushes discussion off your screen.
   

   So the first reply is indented under the post. Fair enough; how else would you know that one post is a reply to another post? But this indentation game doesn't ever end. Reply long and hard enough and you've either made the content column impossibly narrow, or you've pushed the content to exit, stage right. That's how endless pedantic responses-to-responses ruin the discussion for everyone. When we play the "indent everything to the right" game, everyone loses. It is natural to scroll down on the web, but it is utterly unnatural to scroll right. Indentation takes the discussion in the wrong direction.

4. You're talking to everyone.
   

   You think because you clicked "reply" and your post is indented under the person you're replying to, that your post is talking only to that person? That's so romantic. Maybe the two of you should get a room. A special, private room at the far, far, far, far, far right of that threaded discussion. This illusion that you are talking to one other person ends up harming the discussion for everyone by polluting the tree with these massive narrow branches that are constantly in the way.

   At an absolute minimum you're addressing everyone else in that discussion, but in reality, you're talking to anyone who will listen, for all time. Composing your reply as if it is a reply to just one person is a quaint artifact of a world that doesn't exist any more. Every public post you make on the Internet, reply or not, is actually talking to everyone who will ever read it. It'd be helpful if the systems we used for discussion made that clear, rather than maintaining this harmful pretense of private conversations in a public space.

5. I just want to scroll down.

   Reddit (and to a lesser extent, Hacker News) are probably the best known examples of threaded comments applied to a large audience. While I find Reddit so much more tolerable than the bad old days of Digg, I can still barely force myself to wade through the discussions there, because it's so much darn work. As a lazy reader, I feel I've already done my part by deciding to enter the thread; after that all I should need to do is scroll or swipe down.

   Take what's on the top of reddit right now. It's a cool picture; who wouldn't want to meet Steve Martin and Morgan Freeman? But what's the context? Who is this kid? How did he get so lucky? To find out, I need to collapse and suppress dozens of random meaningless tangents, and the replies-to-tangents, by clicking the little minus symbol next to each one. So that's what I'm doing: reading a little, deciding that tangent is not useful or interesting, and clicking it to get rid of it. Then I arrive at the end and find out that information wasn't even in the topic, or at least I couldn't find it. I'm OK with scrolling down to find information and/or entertainment, to a point. What I object to is the menial labor of collapsing and expanding threaded portions of the topic as I read. Despite what the people posting them might think, those tangents aren't so terribly important that they're worth making me, and every other reader, act on them.

Full bore, no-holds-barred threading is an unmitigated usability disaster for discussion, everywhere I've encountered it. But what if we didn't commit to this idea of threaded discussion quite so wholeheartedly?

The most important guidance for non-destructive use of threading is to put a hard cap on the level of replies that you allow. Although Stack Exchange is not a discussion system – it's actually the opposite of a discussion system, which we have to explain to people all the time – we did allow, in essence, one level of threading. There are questions and answers, yes, but underneath each of those, in smaller type, are the comments.

Now there's a bunch of hard-core discussion sociology here that I don't want to get into, like different rules for comments, special limitations for comments, only showing the top n of comments by default, and so forth. What matters is that we allow one level of replies and that's it. Want to reply to a comment? You can, but it'll be at the same level. You can go no deeper. This is by design, but remember: Stack Exchange is not a discussion system. It's a question and answer system. If you build your Q&A system like a discussion system, it will devolve into Yahoo Answers, or even worse, Quora. Just kidding Quora. You're great.

Would Hacker News be a better place for discussion if they capped reply level? Would Reddit? From my perspective as a poor, harried reader and very occasional participant, absolutely. There are many chronic problems with threaded discussion, but capping reply depth is the easiest way to take a giant step in the right direction.

Another idea is to let posts bring their context with them. This is one of the things that Twitter, the company that always does everything wrong and succeeds anyway, gets … shockingly right out of the gate. When I view one of my tweets, it can stand alone, as it should. But it can also bring some context along with it on demand:

Here you can see how my tweet can be expanded with a direct link or click to show the necessary context for the conversation. But it'll only show three levels: the post, my reply to the post, and replies to my post. This idea that tweets – and thus, conversations – should be mostly standalone is not well understood, but it illustrates how Twitter got the original concept so fundamentally right. I guess that's why they can get away with the terrible execution.

I believe selective and judicious use of threading is the only way it can work for discussion. You should be wary of threading as a general purpose solution for human discussions. Always favor simple, flat discussions instead.

In the go-go world of software development, we're so consumed with learning new things, so fascinated with the procession of shiny new objects that I think we sometimes lose sight of our history. I don't mean the big era-defining successes. Everyone knows those stories. I'm talking about the things we've tried before that … didn't quite work out. The failures. The also-rans. The noble experiments. The crazy plans.

I'm all for reinventing the wheel, because it's one of the best ways to learn. But you shouldn't even think about reinventing a damn thing until you've exhaustively researched every single last wheel, old or new, working or broken, that you can lay your hands on. Do your homework.

That's why I love unearthing stories like The Lessons of Lucasfilm's Habitat. It's basically World of Warcraft … in 1985.

Habitat is "a multi-participant online virtual environment," a cyberspace.

Each participant ("player") uses a home computer (Commodore 64) as an intelligent, interactive client, communicating via modem and telephone over a commercial packet-switching network to a centralized, mainframe host system. The client software provides the user interface, generating a real-time animated display of what is going on and translating input from the player into messages to the host. The host maintains the system's world model enforcing the rules and keeping each player's client informed about the constantly changing state of the universe.

This was the dark ages of home computing. In 1985, that 64k of memory in a Commodore 64 was a lot. The entirety of Turbo Pascal 3.02 for DOS, released a year later in 1986, was barely 40k.

The technical information in The Lessons of Lucasfilm's Habitat is incredibly dated, as you'd expect, and barely useful even as trivia. But the sociological lessons of Habitat cut to the bone. They're as fresh today as they were in 1985. Computers have radically changed in the intervening 27 years, whereas people's behavior hasn't. At all. This particular passage hit home:

Again and again we found that activities based on often unconscious assumptions about player behavior had completely unexpected outcomes (when they were not simply outright failures). It was clear that we were not in control. The more people we involved in something, the less in control we were. We could influence things, we could set up interesting situations, we could provide opportunities for things to happen, but we could not predict nor dictate the outcome. Social engineering is, at best, an inexact science, even in proto-cyberspaces. Or, as some wag once said, "in the most carefully constructed experiment under the most carefully controlled conditions, the organism will do whatever it damn well pleases."

Even more specifically:

Propelled by these experiences, we shifted into a style of operations in which we let the players themselves drive the direction of the design. This proved far more effective. Instead of trying to push the community in the direction we thought it should go, an exercise rather like herding mice, we tried to observe what people were doing and aid them in it. We became facilitators as much as designers and implementors. This often meant adding new features and new regions to the system at a frantic pace, but almost all of what we added was used and appreciated, since it was well matched to people's needs and desires. As the experts on how the system worked, we could often suggest new activities for people to try or ways of doing things that people might not have thought of. In this way we were able to have considerable influence on the system's development in spite of the fact that we didn't really hold the steering wheel -- more influence, in fact, than we had had when we were operating under the delusion that we controlled everything.

That's exactly what I was trying to say in Listen to Your Community, But Don't Let Them Tell You What to Do. Unfortunately, because I hadn't read this essay until a few months ago, I figured this important lesson out 25 years later than Randy Farmer and Chip Morningstar. So many Stack Overflow features were the direct result of observing what the community was doing, then attempting to aid them in it:

• We noticed early in the Stack Overflow beta that users desperately wanted to reply to each other, and were cluttering up the system with "answers" that were, well, not answers to the question. Rather than chastize them for doing it wrong – stupid users! – we added the commenting system to give them a method of annotating answers and questions for clarifications, updates, and improvements.
  
  
• I didn't think it was necessary to have a place to discuss Stack Overflow. And I was … kind of a jerk about it. The community was on the verge of creating a phpBB forum instance to discuss Stack Overflow. Faced with a nuclear ultimatum, I relented, and you know what? They were right. And I was wrong.
  
  
• The community came up with an interesting convention for handling duplicate questions, by manually editing a blockquote into the top of the question with a link to the authoritative question that it was a duplicate of. This little user editing convention eventually became the template for the official implementation.

I could go on and on, but I won't bore you. I'd say for every 3 features we introduced on Stack Overflow, at least two of them came more or less directly from observing the community, then trying to run alongside them, building tools that helped them do what they wanted to do with less fuss and effort. That was my job for the last four years. And I loved it, until I had to stop loving it.

Randy Farmer, one of the primary designers of Habitat at Lucasfilm, went on to work on a bunch of things that you may recognize: with Douglas Crockford on JSON, The Sims Online, Second Life, Yahoo 360°, Yahoo Answers, Answers.com, and so forth. He eventually condensed some of his experience into a book, Building Web Reputation Systems, which I bought in April 2011 as a Kindle edition. I didn't know who Mr. Farmer was at this time. I just saw a new O'Reilly book on an area of interest, and I thought I'd check it out.

As the co-founder of Stack Overflow, I know a thing or two about web reputation systems! Out of curiosity, I looked up the author on my own site. And I found him, with a tiny reputation. So I sent this friendly jibe on Twitter:

But the last laugh was on Randy, as it should be, because I didn't realize he had over 6,000 reputation on rpg.stackexchange.com. Turns out, Randy Farmer was already an avid Stack Exchange user. And, as you might guess given his background, a rather expert Stack Exchange user at that. The Stack Exchange ruleset is complex, strict, and requires discipline to understand. Kind of like.. maybe a certain role playing game, if you will.

Randy is the sort of dad who had his first edition Dungeons & Dragons books bound into a single leather tome and handed it down to his son as a family heirloom. How awesome is that?

If we've learned anything in the last 25 years since Habitat, it is that people are the source of, and solution to, all the problems you'll run into when building social software. Are you looking to (dungeon) master the art of guiding and nudging your online community through their collective adventure, without violating the continuity of your own little universe? If so, you could do a whole heck of lot worse than reading Building Web Reputation Systems and following @FRandallFarmer on Twitter.

For the last year or so, I've been getting these two page energy assessment reports in the mail from Pacific Gas & Electric, our California utility company, comparing our household's energy use to those of the houses around us.

Here's the relevant excerpts from the latest report; click through for a full-page view of each page.

These poor results are particularly galling because I go far out of my way to Energy Star all the things, I use LED light bulbs just about everywhere, we set our thermostat appropriately, and we're still getting crushed. I have no particular reason to care about this stupid energy assessment report showing our household using 33% more energy than similar homes in our neighborhood. And yet… I must win this contest. I can't let it go.

• Installed a Nest 2.0 learning thermostat.
• I made sure every last bulb in our house that gets any significant use is LED. Fortunately there are some very good $10 LED bulbs on Amazon now offering excellent 60 watt equivalents at 9 watt, without any of those pesky early adopter LED quirks (color, dimming, size, weight, etc).
• I even put appliance LED bulbs in our refrigerator and freezer.
• Switched to a low-flow shower head.
• Upgraded to a high efficiency tankless water heater, the Noritz NCC1991-SV.
• Nearly killed myself trying to source LED candelabra bulbs for the fixture in our dining room which has 18 of the damn things, and is used quite a bit now with the twins in the house. Turns out, 18 times any number … is still kind of a large number. In cash.

(Most of this has not helped much on the report. The jury is still out on the Nest thermostat and the candelabra LED bulbs, as I haven't had them long enough to judge. I'm gonna defeat this thing, man!)

I'm ashamed to admit that it's only recently I realized that this technique – showing a set of metrics alongside your peers – is exactly the same thing we built at Stack Overflow and Stack Exchange. Notice any resemblance on the user profile page here?

You've tricked me into becoming obsessed with understanding and reducing my household energy consumption. Something that not only benefits me, but also benefits the greater community and, more broadly, benefits the entire world. You've beaten me at my own game. Well played, Pacific Gas & Electric. Well played.

This peer motivation stuff, call it gamification if you must, really works. That's why we do it. But these systems are like firearms: so powerful they're kind of dangerous if you don't know what you're doing. If you don't think deeply about what you're incentivizing, why you're incentivizing it, and the full ramifications of all emergent behaviors in your system, you may end up with … something darker. A lot darker.

The key lesson for me is that our members became very thoroughly obsessed with those numbers. Even though points on Consumating were redeemable for absolutely nothing, not even a gold star, our members had an unquenchable desire for them. What we saw as our membership scrabbled over valueless points was that there didn't actually need to be any sort of material reward other than the points themselves. We didn't need to allow them to trade the points in for benefits, virtual or otherwise. It was enough of a reward for most people just to see their points wobble upwards. If only we had been able to channel that obsession towards something with actual value!

Since I left Stack Exchange, I've had a difficult time explaining what exactly it is I do, if anything, to people. I finally settled on this: what I do, what I'm best at, what I love to do more than anything else in the world, is design massively multiplayer games for people who like to type paragraphs to each other. I channel their obsessions – and mine – into something positive, something that they can learn from, something that creates wonderful reusable artifacts for the whole world. And that's what I still hope to do, because I have an endless well of obsession left.

Just ask PG&E.