Coding Horror

I've been a Microsoft developer for decades now. I weaned myself on various flavors of home computer Microsoft Basic, and I got my first paid programming gigs in Microsoft FoxPro, Microsoft Access, and Microsoft Visual Basic. I have seen the future of programming, my friends, and it is terrible CRUD apps running on Wintel boxes!

Of course, we went on to build Stack Overflow in Microsoft .NET. That's a big reason it's still as fast as it is. So one of the most frequently asked questions after we announced Discourse was:

Why didn't you build Discourse in .NET, too?

Let me be clear about something: I love .NET. One of the greatest thrills of my professional career was getting the opportunity to place a Coding Horror sticker in the hand of Anders Hejlsberg. Pardon my inner fanboy for a moment, but oh man I still get chills. There are maybe fifty world class computer language designers on the planet. Anders is the only one of them who built Turbo Pascal and Delphi. It is thanks to Anders' expert guidance that .NET started out such a remarkably well designed language – literally what Java should have been on every conceivable level – and has continued to evolve in remarkably practical ways over the last 10 years, leveraging the strengths of other influential dynamically typed languages.

All that said, it's true that I intentionally chose not to use .NET for my next project. So you might expect to find an angry, righteous screed here about how much happier I am leaving the oppressive shackles of my Microsoft masters behind. Free at last, free at least, thank God almighty I'm free at last!

Sorry. I already wrote that post five years ago.

Like any pragmatic programmer, I pick the appropriate tool for the job at hand. And as much as I may love .NET, it would be an extraordinarily poor choice for an 100% open source project like Discourse. Why? Three reasons, mainly:

1. The licensing. My God, the licensing. It's not so much the money, as the infernal, mind-bending tax code level complexity involved in making sure all your software is properly licensed: determining what 'level' and 'edition' you are licensed at, who is licensed to use what, which servers are licensed... wait, what? Sorry, I passed out there for a minute when I was attacked by rabid licensing weasels.

   I'm not inclined to make grand pronouncements about the future of software, but if anything kills off commercial software, let me tell you, it won't be open source software. They needn't bother. Commercial software will gleefully strangle itself to death on its own licensing terms.

2. The friction. If you want to build truly viable open source software, you need people to contribute to your project, so that it is a living, breathing, growing thing. And unless you can download all the software you need to hack on your project freely from all over the Internet, no strings attached, there's just … too much friction.

   If Stack Overflow taught me anything, it is that we now live in a world where the next brilliant software engineer can come from anywhere on the planet. I'm talking places this ugly American programmer has never heard of, where they speak crazy nonsense moon languages I can't understand. But get this. Stand back while I blow your mind, people: these brilliant programmers still code in the same keywords we do! I know, crazy, right?

   Getting up and running with a Microsoft stack is just plain too hard for a developer in, say, Argentina, or Nepal, or Bulgaria. Open source operating systems, languages, and tool chains are the great equalizer, the basis for the next great generation of programmers all over the world who are going to help us change the world.

3. The ecosystem. When I was at Stack Exchange we strove mightily to make as much of our infrastructure open source as we could. It was something that we made explicit in the compensation guidelines, this idea that we would all be (partially) judged by how much we could do in public, and try to leave behind as many useful, public artifacts of our work as we could. Because wasn't all of Stack Exchange itself, from the very first day, built on your Creative Commons contributions that we all share ownership of?

   You can certainly build open source software in .NET. And many do. But it never feels natural. It never feels right. Nobody accepts your patch to a core .NET class library no matter how hard you try. It always feels like you're swimming upstream, in a world of small and large businesses using .NET that really aren't interested in sharing their code with the world – probably because they know it would suck if they did, anyway. It is just not a native part of the Microsoft .NET culture to make things open source, especially not the things that suck. If you are afraid the things you share will suck, that fear will render you incapable of truly and deeply giving back. The most, uh, delightful… bit of open source communities is how they aren't afraid to let it "all hang out", so to speak.

   So as a result, for any given task in .NET you might have – if you're lucky – a choice of maybe two decent-ish libraries. Whereas in any popular open source language, you'll easily have a dozen choices for the same task. Yeah, maybe six of them will be broken, obsolete, useless, or downright crazy. But hey, even factoring in some natural open source spoilage, you're still ahead by a factor of three! A winner is you!

As I wrote five years ago:

I'm a pragmatist. For now, I choose to live in the Microsoft universe. But that doesn't mean I'm ignorant of how the other half lives. There's always more than one way to do it, and just because I chose one particular way doesn't make it the right way – or even a particularly good way. Choosing to be provincial and insular is a sure-fire path to ignorance. Learn how the other half lives. Get to know some developers who don't live in the exact same world you do. Find out what tools they're using, and why. If, after getting your feet wet on both sides of the fence, you decide the other half is living better and you want to join them, then I bid you a fond farewell.

However, I'd also be lying if I didn't mention that I truly believe the sort of project we are building in Discourse does represent most future software. If you squint your eyes a little, I think you can see a future not too far in the distance where .NET is a specialized niche outside the mainstream.

But why Ruby? Well, the short and not very glamorous answer is that I had narrowed it down to either Python or Ruby, and my original co-founder Robin Ward has been building major Rails apps since 2006. So that clinched it.

I've always been a little intrigued by Ruby, mostly because of the absolutely gushing praise Steve Yegge had for the language way back in 2006. I've never forgotten this.

For the most part, Ruby took Perl's string processing and Unix integration as-is, meaning the syntax is identical, and so right there, before anything else happens, you already have the Best of Perl. And that's a great start, especially if you don't take the Rest of Perl.

But then Matz took the best of list processing from Lisp, and the best of OO from Smalltalk and other languages, and the best of iterators from CLU, and pretty much the best of everything from everyone.

And he somehow made it all work together so well that you don't even notice that it has all that stuff. I learned Ruby faster than any other language, out of maybe 30 or 40 total; it took me about 3 days before I was more comfortable using Ruby than I was in Perl, after eight years of Perl hacking. It's so consistent that you start being able to guess how things will work, and you're right most of the time. It's beautiful. And fun. And practical.

Steve is one of those polyglot programmers I respect so much that I basically just take whatever his opinion is, provided it's not about something wacky like gun control or feminism or T'Pau, and accept it as fact.

I apologize, Steve. I'm sorry it took me 7 years to get around to Ruby. But maybe I was better off waiting a while anyway:

• Ruby is a decent performer, but you really need to throw fast hardware at it for good performance. Yeah, I know, interpreted languages are what they are, and caching, database, network, blah blah blah. Still, we obtained the absolute fastest CPUs you could buy for the Discourse servers, 4.0 Ghz Ivy Bridge Xeons, and performance is just … good on today's fastest hardware. Not great. Good.

  Yes, I'll admit that I am utterly spoiled by the JIT compiled performance of .NET. That's what I am used to. I do sometimes pine away for the bad old days of .NET when we could build pages that serve in well under 50 milliseconds without thinking about it too hard. Interpreted languages aren't going to be able to reach those performance levels. But I can only imagine how rough Ruby performance had to be back in the dark ages of 2006 when CPUs and servers were five times slower than they are today! I'm so very glad that I am hitting Ruby now, with the strong wind of many solid years of Moore's law at our backs.

• Ruby is maturing up nicely in the 2.0 language release, which happened not more than a month after Discourse was announced. So, yes, the downside is that Ruby is slow. But the upside is there is a lot of low hanging performance fruit in Ruby-land. Like.. a lot a lot. On Discourse we got an across the board 20% performance improvement just upgrading to Ruby 2.0, and we nearly doubled our performance by increasing the default Ruby garbage collection limit. From a future performance perspective, Ruby is nothing but upside.

• Ruby isn't cool any more. Yeah, you heard me. It's not cool to write Ruby code any more. All the cool people moved on to slinging Scala and Node.js years ago. Our project isn't cool, it's just a bunch of boring old Ruby code. Personally, I'm thrilled that Ruby is now mature enough that the community no longer needs to bother with the pretense of being the coolest kid on the block. That means the rest of us who just like to Get Shit Done can roll up our sleeves and focus on the mission of building stuff with our peers rather than frantically running around trying to suss out the next shiny thing.

And of course the Ruby community is, and always has been, amazing. We never want for great open source gems and great open source contributors. Now is a fantastic time to get into Ruby, in my opinion, whatever your background is.

Even if done in good will and for the best interests of the project, it's still a little scary to totally change your programming stripes overnight after two decades. I've always believed that great programmers learn to love more than one language and programming environment – and I hope the Discourse project is an opportunity for everyone to learn and grow, not just me. So go fork us on GitHub already!

Occasionally, startups will ask me for advice. That's a shame, because I am a terrible person to ask for advice. The conversation usually goes something like this:

We'd love to get your expert advice on our thing.

I probably don't use your thing. Even if I tried your thing out and I gave you my so-called Expert advice, how would it matter? Anyway, why are you asking me? Why don't you ask your community what they think of your thing?

And if you don't have a community of users and customers around your thing, well, there's your problem right there. Go fix that.

Like I said, I don't get asked for advice too often. But for what it's worth, it is serious advice. And the next question they ask always strikes fear into my heart.

You're so right! We need a place for online community around our thing. What software should we use?

This is the part where I start playing sad trombone in my head. Because all your software options for online community are, quite frankly, terrible. Stack Exchange? We only do strict, focused Q&A there and you'd have to marshal your proposal through Area 51. Get Satisfaction, UserVoice, Desk, etcetera? Sorry, customer support isn't the same as community. Mailing lists? Just awful.

Forum software? Maybe. Let's see, it's 2013, has forum software advanced at all in the last ten years?

I'm thinking no.

Forums are the dark matter of the web, the B-movies of the Internet. But they matter. To this day I regularly get excellent search results on forum pages for stuff I'm interested in. Rarely a day goes by that I don't end up on some forum, somewhere, looking for some obscure bit of information. And more often than not, I find it there.

There's an amazing depth of information on forums.

• A 12 year old girl who finds a forum community of rabid enthusiasts willing to help her rebuild a Fiero from scratch? Check.
• The most obsessive breakdown of Lego collectible minifig kits you'll find anywhere on the Internet? Check.
• Some of the most practical information on stunt kiting in the world? Check.
• The only place I could find with scarily powerful squirt gun instructions and advice? Check.
• The underlying research for a New Yorker article outing a potential serial marathon cheater? Check.

I could go on and on. As much as existing forum software is inexplicably and terrifyingly awful after all these years, it is still the ongoing basis for a huge chunk of deeply interesting information on the Internet. These communities are incredibly passionate about incredibly obscure things. They aren't afraid to let their freak flag fly, and the world is a better place for it.

At Stack Exchange, one of the tricky things we learned about Q&A is that if your goal is to have an excellent signal to noise ratio, you must suppress discussion. Stack Exchange only supports the absolute minimum amount of discussion necessary to produce great questions and great answers. That's why answers get constantly re-ordered by votes, that's why comments have limited formatting and length and only a few display, and so forth. Almost every design decision we made was informed by our desire to push discussion down, to inhibit it in every way we could. Spare us the long-winded diatribe, just answer the damn question already.

After spending four solid years thinking of discussion as the established corrupt empire, and Stack Exchange as the scrappy rebel alliance, I began to wonder – what would it feel like to change sides? What if I became a champion of random, arbitrary discussion, of the very kind that I'd spent four years designing against and constantly lecturing users on the evil of?

I already built an X-Wing; could I build a better Tie Fighter?

If you're wondering what all those sly references to Tie Fighters were about in my previous blog posts and tweets, now you know. All hail the Emperor, and by the way, what's your favorite programming food?

Today we announce the launch of Discourse, a next-generation, 100% open source discussion platform built for the next decade of the Internet.

The goal of the company we formed, Civilized Discourse Construction Kit, Inc., is exactly that – to raise the standard of civilized discourse on the Internet through seeding it with better discussion software:

• 100% open source and free to the world, now and forever.
• Feels great to use. It's fun.
• Designed for hi-resolution tablets and advanced web browsers.
• Built in moderation and governance systems that let discussion communities protect themselves from trolls, spammers, and bad actors – even without official moderators.

Our amazingly talented team has been working on Discourse for almost a year now, and although like any open source software it's never entirely done, we believe it is already a generation ahead of any other forum software we've used.

I greatly admire what WordPress did for the web; to say that we want to be the WordPress of forums is not a stretch at all. We're also serious about this eventually being a viable open-source business, in the mold of WordPress. And we're not the only people who believe in the mission: I'm proud to announce that we have initial venture capital funding from First Round, Greylock, and SV Angel. We're embarking on a five year mission to improve the fabric of the Internet, and we're just getting started. Let a million discussions bloom!

So now, when someone says to me …

You're so right! We need a place for community around our thing. What software should we use?

I can reply without hesitation.

And hopefully, so can you.

When Joel Spolsky, my business partner on Stack Overflow and Stack Exchange, asked me what I wanted to do after I left Stack Exchange, I distinctly remember mentioning Aaron Swartz. That's what Aaron was to us hackers: an exemplar of the noble, selfless behavior and positive action that all hackers aspire to – but very few actually achieve.

And now, tragically, Aaron is gone at the tender age of 26. He won't be achieving anything any more.

I never knew Aaron, but I knew Aaron.

Most of all, I am disappointed.

I'm deeply disappointed in myself, for not understanding just how bitterly unfair the government charges were against Aaron. Perhaps the full, grotesque details couldn't be revealed for a pending legal case. But we should have been outraged. I am gutted that I did not contribute to his defense in any way, either financially or by writing about it here. I blindly assumed he would prevail, as powerful activists on the side of fairness, openness, and freedom are fortunate enough to often do in our country. I was wrong.

I'm disappointed in our government, for going to such lengths to make an example of someone who was so obviously a positive force. Someone who actively worked to change the world for the better in everything he did, starting from the age of 12. There was no evil in this man. And yet the absurd government case against him was cited by his family as directly contributing to his death.

I'm frustrated by the idea that martyrdom works. The death of Aaron Swartz is now turning into an effective tool for change, a rallying cry, proving the perverse lesson that nobody takes an issue seriously until a great person dies for the cause. The idea that Aaron killing himself was a viable strategy, more than going on to prevail in this matter and so many more in his lifetime, makes me incredibly angry.

But also, I must admit that I am a little disappointed in Aaron. I understand that depression is a serious disease that can fell any person, however strong. But he chose the path of the activist long ago. And the path of the activist is to fight, for as long and as hard as it takes, to effect change. Aaron had powerful friends, a powerful support network, and a keen sense of moral cause that put him in the right. That's how he got that support network of powerful friends and fellow activists in the first place.

It is appropriate to write about Aaron on Martin Luther King day, because he too was a tireless activist for moral causes.

I hope you are able to see the distinction I am trying to point out. In no sense do I advocate evading or defying the law, as would the rabid segregationist. That would lead to anarchy. One who breaks an unjust law must do so openly, lovingly, and with a willingness to accept the penalty. I submit that an individual who breaks a law that conscience tells him is unjust, and who willingly accepts the penalty of imprisonment in order to arouse the conscience of the community over its injustice, is in reality expressing the highest respect for law.

Let's be clear that the penalty in Aaron's case was grossly unfair, bordering on corrupt. I've been a part of exactly one trial, but I can't even imagine having the full resources of the US Government brought to bear against me, with extreme prejudice, for a year or more. His defense was estimated to cost millions. The idea that such an engaged citizen would be forever branded a felon – serving at least some jail time and stripped of the most fundamental citizenship right, the ability to vote – must have weighed heavily on Aaron. And Aaron was no stranger to depresson, having written about it on his blog many times, even penning a public will of sorts on his blog all the way back in 2002.

I think about ragequitting a lot.

Rage Quit, also seen as RageQuit in one word, is Internet slang commonly used to describe the act of suddenly quitting a game or chatroom after either an argument, extreme frustration, or loss of the game.

At least one user ragequits Stack Exchange every six months, because our rules are strict. Some people don't like rules, and can respond poorly when confronted by the rules of the game they choose to play. It came up often enough that we had to create even more rules to deal with it. I was forced to think about ragequitting.

I say this not as a person who wishes to judge Aaron Swartz. I say it as a fellow gamer who has also considered playing the same move quite recently. To the point that I – like Aaron himself, I am sure – was actively researching it. But the more I researched, the more I thought about it, the more it felt like what it really was: giving up. And the toll on friends and family would be unimaginably, unbearably heavy.

What happened to Aaron was not fair. Not even a little. But this is the path of the activist. The greater the injustice, the greater wrong undone when you ultimately prevail. And I am convinced, absolutely and utterly convinced, that Aaron would have prevailed. He would have gone on to do so many other great things. It is our great failing that we did not provide Aaron the support network he needed to see this. All we can do now is continue the mission he started and lobby for change to our corrupt government practices of forcing plea bargains.

It gets dark sometimes. I know it does. I'm right there with you. But do not, under any circumstances, give anyone the satisfaction of seeing you ragequit. They don't deserve it. Play other, better moves – and consider your long game.

It's been six years since I wrote Discussions: Flat or Threaded? and, despite a bunch of evolution on the web since then, my opinion on this has not fundamentally changed.

If anything, my opinion has strengthened based on the observed data: precious few threaded discussion models survive on the web. Putting aside Usenet as a relic and artifact of the past, it is rare to find threaded discussions of any kind on the web today; for web discussion communities that are more than ten years old, the vast majority are flat as a pancake.

I'm game for trying anything new, I mean, I even tried Google Wave. But the more I've used threaded discussions of any variety, the less I like them. I find precious few redeeming qualities, while threading tends to break crucial parts of discussion like reading and replying in deep, fundamental, unfixable ways. I have yet to discover a threaded discussion design that I can tolerate long term.

A part of me says this is software Darwinism in action: threaded discussion is ultimately too complex to survive on the public Internet.

Before threaded discussion fans bring out their pitchforks and torches, I fully acknowledge that aspects of threading can be useful in certain specific situations. I will get to that. I know I'm probably wasting my time even attempting to say this, but please: keep reading before commenting. Ideally, read the whole article before commenting. Like Parappa, I gotta believe!

Before I defend threaded discussion, let's enumerate the many problems it brings to the table:

1. It's a tree.
   

   Poems about trees are indeed lovely, as Joyce Kilmer promised us, but data of any kind represented as a tree … isn't. Rigid hierarchy is generally not how the human mind works, and the strict parent-child relationship it enforces is particularly terrible for fluid human group discussion. Browsing a tree is complicated, because you have to constantly think about what level you're at, what's expanded, what's collapsed … there's always this looming existential crisis of where the heck am I? Discussion trees force me to spend too much time mentally managing that two-dimensional tree more than the underlying discussion.

2. Where did that reply go?
   

   In a threaded discussion, replies can arrive any place in the tree at any time. How do you know if there are new replies? Where do you find them? Only if you happen to be browsing the tree at the right place at the right time. It's annoying to follow discussions over time when new posts keep popping up anywhere in the middle of the big reply tree. And God help you if you accidentally reply at the wrong level of the tree; then you're suddenly talking to the wrong person, or maybe nobody at all. It absolutely kills me that there might be amazing, insightful responses buried somewhere in the middle of a reply chain that I will never be able to find.

3. It pushes discussion off your screen.
   

   So the first reply is indented under the post. Fair enough; how else would you know that one post is a reply to another post? But this indentation game doesn't ever end. Reply long and hard enough and you've either made the content column impossibly narrow, or you've pushed the content to exit, stage right. That's how endless pedantic responses-to-responses ruin the discussion for everyone. When we play the "indent everything to the right" game, everyone loses. It is natural to scroll down on the web, but it is utterly unnatural to scroll right. Indentation takes the discussion in the wrong direction.

4. You're talking to everyone.
   

   You think because you clicked "reply" and your post is indented under the person you're replying to, that your post is talking only to that person? That's so romantic. Maybe the two of you should get a room. A special, private room at the far, far, far, far, far right of that threaded discussion. This illusion that you are talking to one other person ends up harming the discussion for everyone by polluting the tree with these massive narrow branches that are constantly in the way.

   At an absolute minimum you're addressing everyone else in that discussion, but in reality, you're talking to anyone who will listen, for all time. Composing your reply as if it is a reply to just one person is a quaint artifact of a world that doesn't exist any more. Every public post you make on the Internet, reply or not, is actually talking to everyone who will ever read it. It'd be helpful if the systems we used for discussion made that clear, rather than maintaining this harmful pretense of private conversations in a public space.

5. I just want to scroll down.

   Reddit (and to a lesser extent, Hacker News) are probably the best known examples of threaded comments applied to a large audience. While I find Reddit so much more tolerable than the bad old days of Digg, I can still barely force myself to wade through the discussions there, because it's so much darn work. As a lazy reader, I feel I've already done my part by deciding to enter the thread; after that all I should need to do is scroll or swipe down.

   Take what's on the top of reddit right now. It's a cool picture; who wouldn't want to meet Steve Martin and Morgan Freeman? But what's the context? Who is this kid? How did he get so lucky? To find out, I need to collapse and suppress dozens of random meaningless tangents, and the replies-to-tangents, by clicking the little minus symbol next to each one. So that's what I'm doing: reading a little, deciding that tangent is not useful or interesting, and clicking it to get rid of it. Then I arrive at the end and find out that information wasn't even in the topic, or at least I couldn't find it. I'm OK with scrolling down to find information and/or entertainment, to a point. What I object to is the menial labor of collapsing and expanding threaded portions of the topic as I read. Despite what the people posting them might think, those tangents aren't so terribly important that they're worth making me, and every other reader, act on them.

Full bore, no-holds-barred threading is an unmitigated usability disaster for discussion, everywhere I've encountered it. But what if we didn't commit to this idea of threaded discussion quite so wholeheartedly?

The most important guidance for non-destructive use of threading is to put a hard cap on the level of replies that you allow. Although Stack Exchange is not a discussion system – it's actually the opposite of a discussion system, which we have to explain to people all the time – we did allow, in essence, one level of threading. There are questions and answers, yes, but underneath each of those, in smaller type, are the comments.

Now there's a bunch of hard-core discussion sociology here that I don't want to get into, like different rules for comments, special limitations for comments, only showing the top n of comments by default, and so forth. What matters is that we allow one level of replies and that's it. Want to reply to a comment? You can, but it'll be at the same level. You can go no deeper. This is by design, but remember: Stack Exchange is not a discussion system. It's a question and answer system. If you build your Q&A system like a discussion system, it will devolve into Yahoo Answers, or even worse, Quora. Just kidding Quora. You're great.

Would Hacker News be a better place for discussion if they capped reply level? Would Reddit? From my perspective as a poor, harried reader and very occasional participant, absolutely. There are many chronic problems with threaded discussion, but capping reply depth is the easiest way to take a giant step in the right direction.

Another idea is to let posts bring their context with them. This is one of the things that Twitter, the company that always does everything wrong and succeeds anyway, gets … shockingly right out of the gate. When I view one of my tweets, it can stand alone, as it should. But it can also bring some context along with it on demand:

Here you can see how my tweet can be expanded with a direct link or click to show the necessary context for the conversation. But it'll only show three levels: the post, my reply to the post, and replies to my post. This idea that tweets – and thus, conversations – should be mostly standalone is not well understood, but it illustrates how Twitter got the original concept so fundamentally right. I guess that's why they can get away with the terrible execution.

I believe selective and judicious use of threading is the only way it can work for discussion. You should be wary of threading as a general purpose solution for human discussions. Always favor simple, flat discussions instead.

In the go-go world of software development, we're so consumed with learning new things, so fascinated with the procession of shiny new objects that I think we sometimes lose sight of our history. I don't mean the big era-defining successes. Everyone knows those stories. I'm talking about the things we've tried before that … didn't quite work out. The failures. The also-rans. The noble experiments. The crazy plans.

I'm all for reinventing the wheel, because it's one of the best ways to learn. But you shouldn't even think about reinventing a damn thing until you've exhaustively researched every single last wheel, old or new, working or broken, that you can lay your hands on. Do your homework.

That's why I love unearthing stories like The Lessons of Lucasfilm's Habitat. It's basically World of Warcraft … in 1985.

Habitat is "a multi-participant online virtual environment," a cyberspace.

Each participant ("player") uses a home computer (Commodore 64) as an intelligent, interactive client, communicating via modem and telephone over a commercial packet-switching network to a centralized, mainframe host system. The client software provides the user interface, generating a real-time animated display of what is going on and translating input from the player into messages to the host. The host maintains the system's world model enforcing the rules and keeping each player's client informed about the constantly changing state of the universe.

This was the dark ages of home computing. In 1985, that 64k of memory in a Commodore 64 was a lot. The entirety of Turbo Pascal 3.02 for DOS, released a year later in 1986, was barely 40k.

The technical information in The Lessons of Lucasfilm's Habitat is incredibly dated, as you'd expect, and barely useful even as trivia. But the sociological lessons of Habitat cut to the bone. They're as fresh today as they were in 1985. Computers have radically changed in the intervening 27 years, whereas people's behavior hasn't. At all. This particular passage hit home:

Again and again we found that activities based on often unconscious assumptions about player behavior had completely unexpected outcomes (when they were not simply outright failures). It was clear that we were not in control. The more people we involved in something, the less in control we were. We could influence things, we could set up interesting situations, we could provide opportunities for things to happen, but we could not predict nor dictate the outcome. Social engineering is, at best, an inexact science, even in proto-cyberspaces. Or, as some wag once said, "in the most carefully constructed experiment under the most carefully controlled conditions, the organism will do whatever it damn well pleases."

Even more specifically:

Propelled by these experiences, we shifted into a style of operations in which we let the players themselves drive the direction of the design. This proved far more effective. Instead of trying to push the community in the direction we thought it should go, an exercise rather like herding mice, we tried to observe what people were doing and aid them in it. We became facilitators as much as designers and implementors. This often meant adding new features and new regions to the system at a frantic pace, but almost all of what we added was used and appreciated, since it was well matched to people's needs and desires. As the experts on how the system worked, we could often suggest new activities for people to try or ways of doing things that people might not have thought of. In this way we were able to have considerable influence on the system's development in spite of the fact that we didn't really hold the steering wheel -- more influence, in fact, than we had had when we were operating under the delusion that we controlled everything.

That's exactly what I was trying to say in Listen to Your Community, But Don't Let Them Tell You What to Do. Unfortunately, because I hadn't read this essay until a few months ago, I figured this important lesson out 25 years later than Randy Farmer and Chip Morningstar. So many Stack Overflow features were the direct result of observing what the community was doing, then attempting to aid them in it:

• We noticed early in the Stack Overflow beta that users desperately wanted to reply to each other, and were cluttering up the system with "answers" that were, well, not answers to the question. Rather than chastize them for doing it wrong – stupid users! – we added the commenting system to give them a method of annotating answers and questions for clarifications, updates, and improvements.
  
  
• I didn't think it was necessary to have a place to discuss Stack Overflow. And I was … kind of a jerk about it. The community was on the verge of creating a phpBB forum instance to discuss Stack Overflow. Faced with a nuclear ultimatum, I relented, and you know what? They were right. And I was wrong.
  
  
• The community came up with an interesting convention for handling duplicate questions, by manually editing a blockquote into the top of the question with a link to the authoritative question that it was a duplicate of. This little user editing convention eventually became the template for the official implementation.

I could go on and on, but I won't bore you. I'd say for every 3 features we introduced on Stack Overflow, at least two of them came more or less directly from observing the community, then trying to run alongside them, building tools that helped them do what they wanted to do with less fuss and effort. That was my job for the last four years. And I loved it, until I had to stop loving it.

Randy Farmer, one of the primary designers of Habitat at Lucasfilm, went on to work on a bunch of things that you may recognize: with Douglas Crockford on JSON, The Sims Online, Second Life, Yahoo 360°, Yahoo Answers, Answers.com, and so forth. He eventually condensed some of his experience into a book, Building Web Reputation Systems, which I bought in April 2011 as a Kindle edition. I didn't know who Mr. Farmer was at this time. I just saw a new O'Reilly book on an area of interest, and I thought I'd check it out.

As the co-founder of Stack Overflow, I know a thing or two about web reputation systems! Out of curiosity, I looked up the author on my own site. And I found him, with a tiny reputation. So I sent this friendly jibe on Twitter:

But the last laugh was on Randy, as it should be, because I didn't realize he had over 6,000 reputation on rpg.stackexchange.com. Turns out, Randy Farmer was already an avid Stack Exchange user. And, as you might guess given his background, a rather expert Stack Exchange user at that. The Stack Exchange ruleset is complex, strict, and requires discipline to understand. Kind of like.. maybe a certain role playing game, if you will.

Randy is the sort of dad who had his first edition Dungeons & Dragons books bound into a single leather tome and handed it down to his son as a family heirloom. How awesome is that?

If we've learned anything in the last 25 years since Habitat, it is that people are the source of, and solution to, all the problems you'll run into when building social software. Are you looking to (dungeon) master the art of guiding and nudging your online community through their collective adventure, without violating the continuity of your own little universe? If so, you could do a whole heck of lot worse than reading Building Web Reputation Systems and following @FRandallFarmer on Twitter.