Coding Horror

programming and human factors

Twitter: How Not To Crash Responsibly

In yesterday's post on Crashing Responsibly, I outlined a few ways to improve your application's crash behavior. In the event that your application crashes -- and oh, it will -- why not turn that crash into something that:

  • Records lots of diagnostic information developers can use to improve the application over time.
  • Reassures users and provides them with helpful information.

With that in mind, let's take a look at the Twitter crash page. How does it serve developers and users?

Twitter: Something is Technically Wrong

I don't mean to pick on Twitter; their bouts of downtime are near legendary at this point. Frankly, it's been discussed to death.

It's unfortunate, because I love Twitter. Like Michael Lopp, I'm dangerously close to being a Twitter fanboy.

The answer comes down to value. In the time that I've been using Twitter, it's transformed from a curiosity to an essential service. What were seemingly random status updates have now become organized into organic conversational threads that bring a steady flow of relevant content across my desktop.

An "essential service" is exactly the kind of thing you don't want to see error pages on. So, then, how does the Twitter error page fare?

Not so badly at first glance. It's an attractive error page, styled to match Twitter, with some basic links and navigational elements. Let's be generous and assume that the notification and logging of errors behind the scenes is taken care of. The Twitter developers must have access to a voluminous set of error logs by now.

But Twitter's error page is conspicuously lacking any real information. As an enthusiastic Twitter user presented with this error page, I am anything but reassured. Instead, I have some nagging questions:

  • Is this an ephemeral, temporary error or some kind of scheduled downtime? How do I tell the difference?
  • If this is scheduled downtime, when will it be over? Can I view the maintenance schedule, or the current status of the maintenance work?
  • Is Twitter down for everyone, or just me? Is there a place I can go to check Twitter's current system health?
  • Twitter has a reputation for unreliability. Where can I find out about Twitter's ongoing efforts to improve their reliability?

There's absolutely no mention of any of these things on the error page, the exact place I would care the most. Clicking through to the blog provides no relief, no mention of any availability work or maintenance schedules.

Furthermore, it's difficult to take the glib claim that "we're going to fix it up and have things back to normal soon" seriously. I've seen so much of the Twitter error page in the last year that I've lost confidence that these errors mean anything to anyone -- or that they're even recorded. This is the static error page that cried wolf. Where's the improvement over time from the collection and analysis of these errors?

I understand that Twitter has scaling problems I can only dream of. I don't envy the amount of work they'll have to undertake to fix this pernicious, systemic problem of massive scale.

But I sure wish they could be a lot more transparent about it.

Isn't that what crashing responsibly is all about -- establishing an honest, open dialog between users and developers, even at the worst possible moment of that relationship?

Discussion

Crash Responsibly

As programmers, it is our responsibility to ensure that when something goes horribly wrong with our software, the user has a reasonable escape plan. It's an issue of fundamental safety in software error handling that I liken to those ubiquitous airline safety cards.

error-safety-airline-good.png   error-safety-airline-bad.png

Which one accurately depicts the way your software treats the user in the event of an emergency?

If I've learned anything in the last thirty years, it's that I write shitty software -- with bugs. I not only need to protect my users from my errors, I need to protect myself from my errors, too. That's why the first thing I do on any new project is set up an error handling framework. Errors are inevitable, but ignorance shouldn't be. If you know about the problems, you can fix them and respond to them.

Note that when I say "errors", I don't mean mundane, workaday problems like empty form values, no results, or file not found. Those kinds of errors are covered quite well in 37 Signals' Defensive Design for the Web: How to Improve Error Messages, Help, Forms, and Other Crisis Points.

defensive-design-for-the-web.png

It's a great book; a quick read with lots of visual do's and don'ts side by side. Despite the giant exclamation point icon on the cover, however, it's mostly about fundamental web usability, not error handling per se.

I'm talking about catastrophic errors -- real disasters. Cases where a previously unknown bug in your code causes the application to crash and burn in spectacular fashion. It happens in all applications, whether they're websites or traditional executables.

windows error reporting dialog

web error reporting page

The situation is pretty dire at this point, but some disaster recovery is possible, if you plan ahead.

  1. It is not the user's job to tell you about errors in your software!

    If users have to tell you when your app crashes, and why, you have utterly failed your users. I cannot emphasize this enough.

    It's bad enough that the user has to use our crashy software; are we really going to add insult to injury by pressing them into service as QA staff, too? If you're relying on users to tell you about problems with your software, you'll only see a tiny fraction of the overall errors. Most users won't bother telling you about problems. They'll just quietly stop using your application.

    Whatever error handling solution you choose, it should automatically log everything necessary to troubleshoot the crash -- and ideally send a complete set of diagnostic information back to your server. This is fundamental. If you don't have something like this in place yet, do so immediately.

  2. Don't expose users to the default screen of death.

    It's true that we can't do much to recover from these kinds of crashes, but relying on the underlying operating system or webserver to deliver the generic bad news to the user is rude and thoughtless. Override the default crash screen and provide something customized, something relevant to your application and your users. Here are a few ideas:

    • Let users know that it's our fault, not theirs.
    • Inform the user that the error was logged and dispatched.
    • If possible, suggest some workarounds and troubleshooting options.
    • Perhaps even provide direct contact information if they're really stuck and desperately need to get something done.

  3. Have a detailed public record of your application's errors.

    In my experience, nothing motivates a team better than a detailed public record of all crashes. There should of course be a searchable, sortable database of errors somewhere, but active notifications are also a good idea. Crashes are incredibly annoying to your users. It's only fair that the team behind the software share a little of that pain for each crash. You could broadcast an error email, text message, or instant message to everyone on the team. Or maybe have every crash automatically open a bug ticket in your bug tracking software. Tired of dealing with all those error emails and/or bug tickets? Fix the software so you don't have to!

  4. Leverage the 80/20 rule.

    Once you have a comprehensive record of every crash, you can sort that data by frequency and spend your coding effort resolving the most common problems. Microsoft, based on data from their Windows Error Reporting Service, found that fixing 20 percent of the top reported bugs solved 80 percent of customer issues, and fixing 1 percent of the top reported bugs solved 50 percent of customer issues. That's huge! Let the Pareto principle work for you, not against you.

As software professionals, we should protect our users -- and ourselves -- from our mistakes. Crash responsibly!

Discussion

Oh Yeah? Fork You!

In Where Are All The Open Source Billionaires? I used this chart as an illustration:

Linux Distro timeline, 1991-2007

Because open source code is freely distributable, anyone can take that code and create their own unique mutant mashup version of it any time they feel like it. Whether anyone else in the world will care about their crazy new version of the code is not at all clear, but that's not the point. If someone wants it bad enough, they can create it -- or pay someone else to create it for them. This is known as "forking". It's the very embodiment of freedom zero, and it's an essential part of every open source license.

But there are forks, and there are forks:

What is different about a fork is intent. In a fork, the person(s) creating the fork intend for the fork to replace or compete with the original project they are forking.

That's exactly what happened to the Pidgin project recently.

In their 2.4 release they changed the GUI action of the text field where the user types their IM from a manually re-sizable window, to a fixed size window that auto-re-sizes based on the amount of text typed. On the surface, this sounds like a minor change, but it triggered a massive user revolt! Why?

This is what they're up in arms about:

Pidgin input window

The developers, for whatever reason, dug in their heels on this one and refused to budge. You can read through some of the commentary on the bug ticket to get an idea, but the general tenor was combatative bordering on hostile. The bug was eventually closed as "won't fix".

The community's response was swift: Oh yeah? Fork you!

Funpidgin is a fork of the popular open source client Pidgin which allows instant messaging with over twenty different protocols.

What makes us different from the official client is that we work for you. Unlike the Pidgin developers, we believe the user should have the final say in what goes into the program.

So far five new features have been added to Funpidgin upon requests from users, and all of them are optional. It is these options that make the use of Funpidgin enjoyable to a diverse range of people.

Funpidgin is a fork in the truest sense; the developers intend to replace Pidgin. But will it? Who knows. There are four possible outcomes from any fork:

  1. The fork dies
    Funpidgin languishes due to lack of attention from developers and users. Funpidgin eventually dies.
  2. The fork merges
    Funpidgin and Pidgin reach a consensus. The Funpidgin changes are folded back into Pidgin.
  3. The original dies
    Funpidgin becomes so popular that it draws developers and users away from Pidgin. Pidgin eventually dies.
  4. Both original and fork survive
    Funpidgin and Pidgin both succeed on their own terms, perhaps by attracting different audiences or meeting different user needs.

You can find examples of all four outcomes peppered throughout the history of open source software. You might think that the adoption of open source software licenses would lead to dozens if not hundreds of incompatible, slightly-different versions of the same stuff -- bewildering users and developers alike. I'm not so sure. There's a tremendous amount of inertia around the open source projects that survive long enough to become popular. Consider the challenges the newly forked Funpidgin project now faces:

  • A divided community of users and developers.
  • Siphoning enough energy and attention away from an established project to remain viable.
  • Differentiating themselves enough from Pidgin so that they aren't viewed as useless or irrelevant.
  • The original Pidgin project is free to take whatever parts of the Funpidgin open source code they deem appropriate and fold that into Pidgin, thus undermining the fork.

Forking is incredibly difficult to pull off. It is a painful, but necessary part of the evolution of open source software. Just as in real evolution, I suspect that most forks die in vast, nameless numbers, before they become strong enough to engender any forked progeny of their own. Forking is the absolute bedrock of open source software -- but it is also not a path to be chosen lightly.

Discussion

Is HTML a Humane Markup Language?

One of the things we're thinking about while building stackoverflow.com is how to let users style the questions and answers they're entering on the site. Nothing's decided at this point, but we definitely won't be giving users one of those friendly-but-irritating HTML GUI browser layout controls.

an example HTML GUI editor

I have one iron-clad design guide: this is a site for programmers, so they should be comfortable with basic markup. None of that nancy-boy GUI toolbar handholding nonsense for us, thankyouverymuch. If you can sling code, a little bit of presentation markup is child's play.

We will support some sort of markup language to style the questions and answers. But what markup language?

I mentioned in podcast #4 that we consider Wikipedia a defining influence. Let's see how Wikipedia handles markup syntax. This is what the edit page for Joel Spolsky's Wikipedia entry looks like:

Wikipedia Edit page for Joel Spolsky entry

It's an effective markup language, but I think you'll agree that it's more intimidating than humane. Wikipedia's How to Edit a Page and the accompanying Wikipedia syntax cheatsheet helps. Some. I'd argue that writing a Wikipedia entry is a step beyond mere presentational markup; it's almost like coding, as you weave the article into the Wikipedia gestalt. (Incidentally, if you haven't ever edited a Wikipedia article, you should. I consider it a rite of passage, a sort of internet merit badge for anyone who is serious about their online presence.)

Let's consider a simpler example. What we're looking for is some kind of middle ground, a humane text format. Let's start with some basic HTML.

Lightweight Markup Languages

According to Wikipedia:

A lightweight markup language is a markup language with a simple syntax, designed to be easy for a human to enter with a simple text editor, and easy to read in its raw form.

Some examples are:

  • Markdown
  • Textile
  • BBCode
  • Wikipedia

Markup should also extend to code:

10 PRINT "I ROCK AT BASIC!"
20 GOTO 10

Here's what that looks like expressed in a variety of lightweight markup languages. Bear in mind that each of these will produce HTML equivalent to the above.

Textile Markdown
h1. Lightweight Markup Languages
According to *Wikipedia*:
bq. A "lightweight markup language":http://is.gd/gns
is a markup language with a simple syntax, designed
to be easy for a human to enter with a simple text
editor, and easy to read in its raw form.
Some examples are:
* Markdown
* Textile
* BBCode
* Wikipedia
Markup should also extend to _code_:
pre. 10 PRINT "I ROCK AT BASIC!"
20 GOTO 10
Lightweight Markup Languages
============================
According to **Wikipedia**:
> A [lightweight markup language](http://is.gd/gns)
is a markup language with a simple syntax, designed
to be easy for a human to enter with a simple text
editor, and easy to read in its raw form.
Some examples are:
* Markdown
* Textile
* BBCode
* Wikipedia
Markup should also extend to _code_:
10 PRINT "I ROCK AT BASIC!"
20 GOTO 10
Wikipedia BBCode
==Lightweight Markup Languages==
According to '''Wikipedia''':
:A [[lightweight markup language]]
is a markup language with a simple syntax, designed
to be easy for a human to enter with a simple text
editor, and easy to read in its raw form.
Some examples are:
* Markdown
* Textile
* BBCode
* Wikipedia
Markup should also extend to ''code'':
<source lang=qbasic>
10 PRINT "I ROCK AT BASIC!"
20 GOTO 10
</source>
[size=150]Lightweight Markup Languages[/size]
According to [b]Wikipedia[/b]:
[quote]
A [url=http://is.gd/gns]lightweight markup language[/url]
is a markup language with a simple syntax, designed
to be easy for a human to enter with a simple text
editor, and easy to read in its raw form.
[/quote]
Some examples are:
[list]
[*]Markdown
[*]Textile
[*]BBCode
[*]Wikipedia
[/list]
Markup should also extend to [i]code[/i]:
[code]
10 PRINT "I ROCK AT BASIC!"
20 GOTO 10
[/code]

None of these lightweight markup languages are particularly difficult to understand -- and they're easy on the eyes, as promised. But I still had to look up the reference syntax for each one and map it to the HTML that I already know by heart. I also found them disturbingly close to "magic" for some of the formatting rules, to the point that I wished I could just write literal HTML and get exactly what I want without guessing how the parser is going to interpret my fake-plain-text.

Which leads directly to this question: why not just stick with what we already know and use HTML? This c2 wiki page titled Why Doesn't Wiki Do HTML? makes the case that -- at least for Wiki content -- you're better off leaving HTML behind:

  1. In a Wiki, the emphasis is on content, not presentation. Simple Wiki markup rules let people focus on expressing their ideas.
  2. Why not use a domain-specific markup language designed to do "the simplest thing that could possibly work"?
  3. Some HTML tags are difficult to work with and can break the flow of your thoughts. The table tag, for example.
  4. Does the average user really need total HTML and CSS layout power?
  5. Allowing the full range of HTML tags can lead to major security vulnerabilities.
  6. Many people don't know HTML. A simple Wiki markup language is easier to learn.

I'm not sure I agree with all of this, but it can make sense in the context of a full-blown Wiki. It's worth considering.

After all this research on humane markup languages, much to my chagrin, I've come full circle. I now no longer think humane markup languages make sense for most uses. I agree with the guy at fileformat.info -- HTML is generally the better choice:

  • Simplicity

    If the source and destination are the web, why not use the native markup language of the web?

  • Readability

    HTML is a bit less readable than the lightweight markup languages, it's true. But basic HTML is not onerous to read, particularly if we hide the repetitive paragraph tags.

  • Security

    With a bit of careful coding, it is possible to whitelist specific HTML tags that you will allow. This way you avoid exposing yourself to risky/vulnerable tags.

  • Conversion

    It's not at all clear that any existing lightweight markup language has critical mass, with the possible exception of Wikipedia's flavor. On the other hand, text parsers and tools will always understand HTML.

  • What people know

    A lot more people know HTML than any given flavor of humane text. If you're a programmer, you damn well better know HTML. For the handful of wiki-like functions we may need, it's possible to add some optional attributes to the HTML tags. And wouldn't that be easier to learn than some weird, pseudo-ASCII derivation of HTML?

I do think we'll adopt some of the cleverer functions of Textile and Markdown, insofar as they remove mundane HTML markup scutwork. But in general, I'd much rather rely on a subset of trusty old HTML than expend brain cells trying to remember the fake-HTML way to make something bold, or create a hyperlink. HTML isn't perfect, but it's an eminently reasonable humane markup language.

Discussion

Cleaning Your Display and Keyboard

Let's say, just as a hypothetical, you're sitting at your computer, casually chatting with a fellow programmer. You begin to describe some bit of code, then bring it up on your display to illustrate. You want to highlight some particular part of the code. Perhaps you move the cursor invitingly over the area to bring it to their attention, or gesture towards it with your hand.

What happens next?

When I said there were two types of programmers, here's what I really meant:

  1. Programmers who touch displays with their greasy, disgusting, bacteria-addled fingers.
  2. Programmers who don't.

I am incredibly anal about people not touching my displays. I'm not even going to apologize. If you touch my display, I'll kill you. Displays are for viewing, not touching. Put down your damn sticky bun and go touch your own filthy display. Here's my mental image of everyone who has ever touched my screen:

a baby eating

You know that's you. You know it. You do. And you just can't resist touching my display, can you?

Every time it happens, I replay it in slow motion, desperately trying to insert some part of my body between the toucher and my monitor. But I rarely succeed.

Not everyone considers displays inviolate and untouchable as I do. They should. But keyboards are another matter. They're designed to be touched. And boy, are they ever disgusting. They're literally dirtier than a toilet.

Out of 33 keyboards swabbed, four were regarded as a potential health hazard and one harboured five times more germs than one of the office's toilet seats.

Microbiologist Dr Peter Wilson said a keyboard was often "a reflection of what is in your nose and in your gut".

During the Which? tests in January this year, a microbiologist deemed one of the office's keyboards to be so dirty he ordered it to be removed, quarantined and cleaned.

It had 150 times the recommended limit for bacteria - five times as filthy as a lavatory seat tested at the same time, the research found.

After reading that, I'm not sure I want to touch my own keyboard any more, much less someone else's.

So then, how do we clean our screens and keyboards that are so casually defiled by our coworkers, family, and friends? This is apparently not a big concern for some. I am continually amazed by the horrifying state of many programmer's computer workstation keyboards and monitors. I'm not talking about dust, but utter and total neglect resulting in devices I'm afraid to touch. Given the data, maybe that's a good thing.

Cleaning screens is fairly straightforward.

Most manufacturers recommend basic soap and water -- no harsh detergents -- along with a soft cloth. I've used the Monster ScreenClean kit for a while with good results.

Monster ScreenClean

You don't have to buy a kit, of course, but I definitely recommend some kind of microfiber cloth like the one bundled here. Microfiber is a generic name for any synthetic fiber that's finer than silk, and the stuff is amazing. It works well on all kinds of displays: televisions, computer monitors, laptops -- I even use the kit to clean my glasses.

Cleaning keyboards is a much more challenging task.

Despite what you may have been told, compressed air dusters aren't just for sneaking up behind your unsuspecting coworkers and friends and spraying them in the neck and ears. I mean, yes, that's the ideal use, but it's also quite good at cleaning up computer equipment. Including keyboards. You can remove most of the dust and a substantial amount of the unmentionable gunk that builds up under the keys with a generous application of compressed air.

line of compressed air dusters

Compressed air is a reasonable first line of defense. But it does nothing to actually clean the keyboard. Sure, you could methodically disassemble your keyboard, or if you're hard core enough, even disassemble your laptop's keyboard, and painstakingly clean every part of it. But is all that work really worth it to clean a lousy keyboard? Short of buying a new keyboard every few years, is there a better way?

Maybe. Have you considered putting your keyboard in the dishwasher? It's not as crazy as it sounds; based on the volume of reader feedback to an old BoingBoing post on the topic, I'd say it works. It certainly seemed to work for Austin Matzko.

But lately the years of dirt build-up [on his 10 year old keyboard] have been really disgusting, so I decided to try something I read about a long time ago: cleaning the keyboard in the dishwasher.

keyboard in dishwasher

Everything washed up beautifully and dried out by the next morning; check out the before and after pics. Total time disassembling and reassembling the keyboard was probably five minutes, which is a lot less than you'd spend trying to clean the thing with Q-tips. If that's too much work for you, just stick the whole thing in there, but give it several days to dry out.

Note that Austin removed the circuitry from the keyboard first, while some people stick the whole keyboard in the dishwasher as-is. There is a followup NPR article that toes the keyboard manufacturer party line and advises against doing this, so obviously, try at your own risk. Personally, I can't wait to give it a shot. I'll buy a new keyboard first, just in case something goes horribly wrong -- and because I need a second keyboard to use while the first one dries for a week.

If that's too radical an approach, you can fall back on using the old reliable soap-and-water damp rag to scrub your keyboard clean. There's even a neat Mac utility program, Keyboard Cleaner, which will lock out your keyboard while you're thoroughly wiping it down.

I'm no germophobe, but I like using clean keyboards and displays, and I'd prefer to see other people using clean equipment too. But remember -- just because I can clean my display doesn't mean you should be touching it, Poky McSmudgypants.

Discussion