Coding Horror

A little over a year ago, I wrote about the importance of version control for databases.

When I ask development teams whether their database is under version control, I usually get blank stares.

The database is a critical part of your application. If you deploy version 2.0 of your application against version 1.0 of your database, what do you get? A broken application, that's what. That's why your database should always be under source control, right next to your application code. You deploy the app, and you deploy the database. Like peanut butter and chocolate, they are two great tastes that taste great together.

When it comes to version control, the database is often a second or even third-class citizen. From what I've seen, teams that would never think of writing code without version control in a million years-- and rightly so-- can somehow be completely oblivious to the need for version control around the critical databases their applications rely on. I don't know how you can call yourself a software engineer and maintain a straight face when your database isn't under exactly the same rigorous level of source control as the rest of your code. Don't let this happen to you. Get your database under version control.

I was thinking about this again because my friend and co-author K. Scott Allen just wrote a brilliant five part series on the philosophy and practice of database version control:

1. Three rules for database work
2. The Baseline
3. Change Scripts
4. Views, Stored Procedures and the Like
5. Branching and Merging

K is one of the smartest software developers I know. Read it all; even if you currently have your database under version control (and bully for you if you do), there is much food for peanut buttery chocolatey thought here. It doesn't matter what tools you use-- per the agile manifesto, individuals and interactions are more important than processes and tools. Just get your database under version control already.

Heidi Adkisson notes that features sell products, but the people buying those products often don't use the very features they bought the product for in the first place.

A few years ago I did an extensive in-home study observing use of a particular computer hardware peripheral. Most people had high-end models with many features. But in my observation of use, only one "power user" went beyond using anything but the core, basic features. These people had paid a premium for features they didn't use. However, when describing their purchase experience, it was clear they aspired to using these features and sincerely intended to. But, once the product was out of the box, the paradox of the active user took over. They never invested even the smallest amount of time to go beyond the basics (even though the extra features could have saved them time).

In my experience it's people's aspiration for the experience that drives purchasing decisions. Ultimately, their aspirations may be different than the reality.

It's interesting that Heidi used the hedge phrase "may be different" when her own study data showed that users' aspirations and reality were almost always different. Maybe she aspires to live in a world where aspirations and reality aren't so wildly divergent. I don't blame her. It'd be nice.

That disparity is why it's so important to observe how users actually behave versus the way they tell you they behave. People who do this professionally are called "economists". Observation is a powerful skill, and so is learning to disregard what people tell you in favor of judging them by their actions. No actions are more carefully considered than those that result in money flowing out of your pocket. That's why you owe it to yourself to read books like Freakonomics, and maybe even The Economist magazine. It's also why the Freakonomics blog should be a part of your regular reading routine if it isn't already.

People lie not because they're all evil liars (although a few inevitably will be), but because they're usually lying to themselves in some way. Some lies are useful. Small social "white" lies grease the skids of social reality. Penetrating this veil of lies and intentions is one of the central themes of the excellent television show House, M.D. :

The show plays up subtle connections between the House character and Sherlock Holmes, which is appropriate, because it's very much a detective show at heart. The character Gregory House, as played by the brilliant Hugh Laurie, is fond of stating "Everybody lies." Parsing through all the irrational human behavior, and the inevitable lies – white or otherwise – makes for a gripping detective story indeed when lives are at stake.

Heidi referenced the Paradox of the Active User (pdf), which has been around as a concept since 1987. I highly recommend reading the original paper, but if you don't have time, Jakob Nielsen summarizes:

Users never read manuals but start using the software immediately. They are motivated to get started and to get their immediate task done: they don't care about the system as such and don't want to spend time up front on getting established, set up, or going through learning packages.

The "paradox of the active user" is a paradox because users would save time in the long term by learning more about the system. But that's not how people behave in the real world, so we cannot allow engineers to build products for an idealized rational user when real humans are irrational. We must design for the way users actually behave.

There are a bunch of ways to restate the paradox of the active user. Cooper calls it perpetual intermediacy. I think the easiest way to explain it is this: every user lies. Instead of asking users if they love your software – of course they love your software, it'd be rude to tell the people responsible just how mind-numbingly awful it really is – do what Gregory House does. Observe whether or not they use the software, and how they use it. Rely on that behavioral data to design your software, and not the lies of your users, however well intentioned they may be.

You may think of Steve Martin as a stereotypical family friendly comedian today-- the center of saccharine movies like Parenthood and Father of the Bride. But it wasn't always this way. Steve hit his stride in the early 80's. At that time, I don't think there were any popular comedians exploring the ragged edge of humor in quite the same way Steve Martin was. I'll forever remember finding a copy of his book Cruel Shoes as an impressionable teenager. It's a collection of very strange short stories. At my tender young age, I had certainly never read anything like it. It's hard to explain. Read for yourself. Here's the complete text of the eponymous Cruel Shoes short story:

Anna knew she had to have some new shoes today, and Carlo had helped her try on every pair in the store. Carlo spoke wearily, "Well, that's every pair of shoes in the place."

"Oh, you must have one more pair ..."

"No, not one more pair...Well, we have the cruel shoes, but no one would want..."

Anna interrupted, "Oh yes, let me see the cruel shoes!"

Carlo looked incredulous. "No, Anna, you don't understand, you see, the cruel shoes are..."

"Get them!"

Carlo disappeared into the back room for a moment, then returned with an ordinary shoebox. He opened the lid and removed a hideous pair of black and white pumps. But these were not an ordinary pair of black and white pumps; both were left feet, one had a right angle turn with seperate compartments that pointed the toes in impossible directions. The other shoe was six inches long and was curved inward like a rocking chair with a vise and razor blades to hold the foot in place. Carlo spoke hesitantly, "...Now you see why...they're not fit for humans..."

"Put them on me."

"But..."

"Put them on me!"

Carlo knew all arguments were useless. He knelt down before her and forced the feet into the shoes.

The screams were incredible.

Anna crawled to the mirror and held her bloody feet up where she could see.

"I like them."

She paid Carlo and crawled out of the store into the street.

Later that day, Carlo was overheard saying to a new customer, "Well, that's every shoe in the place. Unless, of course, you'd like to try the cruel shoes."

Funny, yes, but also disturbing-- you feel vaguely uncomfortable while laughing at Cruel Shoes. That awkward feeling is what makes Steve's humor so subversive and thought provoking. I previously cited my lifelong fascination with the subversive humor of Mad Magazine, and I'd hold Steve Martin's brand of strange, experimental comedy right up there next to it. I distinctly remember seeing The Jerk in 1979. I could only make sense of about half of it at the time, but the half I did understand blew my young mind. I've been slowly deciphering the genius of this Steve Martin movie ever since.

In a recent interview with Smithsonian Magazine, Steve explains some of the philosophy behind his unique comedy. It's a fantastic article, but one particular passage stood out:

At the end of my closing-night show at the Troubadour, I stood onstage and took out five bananas. I peeled them, put one on my head, one in each pocket and squeezed one in each hand. Then I read the last line of my latest bad review: "Sharing the bill with Poco this week is comedian Steve Martin ... his 25-minute routine failed to establish any comic identity that would make the audience remember him or the material." Then I walked off the stage.

The consistent work enhanced my act. I learned a lesson: it was easy to be great. Every entertainer has a night when everything is clicking. These nights are accidental and statistical: like lucky cards in poker, you can count on them occurring over time. What was hard was to be good, consistently good, night after night, no matter what the circumstances.

Steve's insistence that greatness isn't something you can count on, or even something you should strive for, resonates deeply for me. Greatness is far too difficult, too abstract, too daunting. Being good-- consistently good-- is the real goal, and that takes hard work and discipline. Being good-- that's something concrete you can roll up your sleeves and accomplish. Forget greatness. Can we even define what greatness truly is? Like Steve Martin, you become great through applying yourself at being reliably good, night after night, venue after venue, time after time.

Voltaire originally said better is the enemy of good; at the risk of creating another snowclone, Steve's advice is essentially that great is the enemy of good. It's not exactly a message about software development, but it strongly reminds me of worse is better, at least from where I'm sitting. There's a fascinating history behind this classic essay that I glossed over in my original post on the topic. After doing a bit more research, I found that Richard Gabriel wrote a detailed article explaining the rich history of worse is better:

The concept known as "worse is better" holds that in software making (and perhaps in other arenas as well) it is better to start with a minimal creation and grow it as needed. Christopher Alexander might call this "piecemeal growth." This is the story of the evolution of that concept.

From 1984 until 1994 I had a Lisp company called "Lucid, Inc." In 1989 it was clear that the Lisp business was not going well, partly because the AI companies were floundering and partly because those AI companies were starting to blame Lisp and its implementations for the failures of AI. One day in Spring 1989, I was sitting out on the Lucid porch with some of the hackers, and someone asked me why I thought people believed C and Unix were better than Lisp. I jokingly answered, "because, well, worse is better." We laughed over it for a while as I tried to make up an argument for why something clearly lousy could be good.

The idea being, of course, that enough goodness slowly accreted over time usually trumps any epic (and usually ill-fated) plans to create any brand new great thing. I completely agree, and I think history bears this lesson out a hundredfold. But even the original author, Richard Gabriel, can't decide if worse really is better. He's written a slew of pro and con articles over the years, vacillating back and forth, playing Devil's Advocate to his own position, representing both the "worse" and "better" sides in equal measure:

• Lisp: Good News, Bad News, How to Win Big (original "Worse is Better" talk, 1991)
• Worse Is Better is Worse (pdf, 1991)
• Is Worse Really Better? (pdf, 1992)
• Models of Software Acceptance: How Winners Win (pdf presentation, 1995)
• Money Through Innovation Reconsidered (pdf, 1995)
• Back to the Future: Is Worse (Still) Better? (pdf, 2000)
• Back to the Future: Worse (Still) is Better! (pdf, 2000)

Richard's final statement on the matter is a bit of a cop out: decide for yourselves. I'm not sure if worse is better or not. Personally, I'm inclined to follow Steve Martin's advice here: strive to be consistently good, and the greatness takes care of itself.

Jamie Zawinski's public service backup announcement starts off with a bang:

Option 1: Learn not to care about your data. Don't save any old email, use a film camera, and only listen to physical CDs and not MP3s. If you have no possessions, you have nothing to lose.

This is obviously meant as satire, but it's disturbingly close to reality for me. I suppose everything in my life that's worth capturing... well, let me put it this way: you're reading it. When I said make it public, I really meant it. Still, I'm fairly sure Jamie was kidding, and while Google may be a great service, it's only a so-so backup mechanism. Let's proceed to option 2, which goes something like this:

1) You have a computer. It came with a hard drive in it. Go buy two more drives of the same size or larger. If the drive in your computer is SATA2, get SATA2. If it's a 2.5" laptop drive, get two of those. Brand doesn't matter, but physical measurements and connectors should match.

2) Get external enclosures for both of them. The enclosures are under $30.

3) Put one of these drives in its enclosure on your desk. Name it something clever like "Backup". If you are using a Mac, the command you use to back up is this:

sudo rsync -vaxE --delete --ignore-errors / /Volumes/Backup/

If you're using Linux, it's something a lot like that. If you're using Windows, go f*ck yourself.

Yeah! Take that, Windows users! Hey, wait a second. I use Windows. Did I mention that Jamie is a funny guy? Moving on.

I've long been a fan of inexpensive hard drive enclosures. Jamie's advice confirms my long held opinion that multiple hard drives are the most effective and easy backup process you'll ever find. The rsync command is more than a simple copy; it actually does a block-by-block comparison, only copying the differences. So instead of backing up the entire contents of your hard drive (again), you only back up the parts that changed since your last backup. This is commonly known as incremental backup.

Incremental backups only have value if you're doing them regularly, so it's only natural to schedule this as a recurring task.

4) If you have a desktop computer, have this happen every morning at 5AM by creating a temporary text file containing this line:

0 5 * * * rsync -vaxE --delete --ignore-errors / /Volumes/Backup/

and then doing sudo crontab -u root that-file

If you have a laptop, do that before you go to bed. Really. Every night when you plug your laptop in to charge.

5) If you're on a Mac, that backup drive will be bootable. That means that when (WHEN) your internal drive scorches itself, you can just take your backup drive and put it in your computer and go. This is nice.

6) When (WHEN) your backup drive goes bad, which you will notice because your last backup failed, replace it immediately. This is your number one priority. Don't wait until the weekend when you have time, do it now, before you so much as touch your computer again. Do it before goddamned breakfast. The universe tends toward maximum irony. Don't push it.

7) That third drive? Do a backup onto it the same way, then take that to your office and lock it in a desk. Every few months, bring it home, do a backup, and immediately take it away again. This is your "my house burned down" backup.

What I like about Jamie's approach is that it's totally KISS, yet it touches all the cornerstones of a solid backup strategy:

1. Pick a simple backup strategy you can live with.
2. Make incremental backups a part of your daily routine.
3. Include an off-site backup in your strategy.

And for the dissenters, although I can't imagine too many with the minimalist backup process Jamie outlined, there's this bon mot:

"OMG, three drives is so expensive! That sounds like a hassle!" Shut up. I know things. You will listen to me. Do it anyway.

I'm not sure Windows users have a direct equivalent of rsync. There is, of course, RoboCopy, and it looks like someone has ported rsync to Windows. But let's face it. I'm a Windows user. When I have a problem, I buy software. That's why, after hearing so many great things about it, I recently purchased a copy of Acronis True Image.

Acronis does a lot of things, but most of all it's drive imaging software, a fancy GUI over the rsync command. With Jamie's recommended two external hard drives in tow, I can use Acronis to create a bootable mirror image of my hard drive. If anything at all goes wrong, I simply swap hard drives, and I'm back in business. I can even create those backup images incrementally and on a schedule. You don't even technically need a second or third hard drive; if you have a large enough primary drive, Acronis will allow you to create a new, hidden partition to store a complete backup image. You can restore these disk images from within Windows proper, during pre-boot, or from bootable USB or optical media. It is very cool, a logical evolution of the more primitive drive imaging products I've used for years.

Of course, as much as I am enamored of it, you don't have to spend thirty bucks on Acronis and even more for two external hard drives to have a decent backup strategy. Lots of people use completely internet based backup services, like Mozy, Carbonite, or JungleDisk, with varying degrees of success. One thing's for sure: until you have a backup strategy of some kind, you're screwed, you just don't know it yet. If backing up your data sounds like a hassle, that's because it is. Shut up. I know things. You will listen to me. Do it anyway.

I never quite made the transition from the Apple II series to the Mac. Instead, I migrated from my Apple II to a PC. I always thought the PC ecosystem, although deeply flawed, was more naturally analogous to the eclectic third party hardware and software hacker ecosystem that grew up around the semi-open Apple II hardware platform. This, to me, was the most enduring and beloved quality of the early Apple community. The Mac, in contrast, was underwritten and driven by primarily Apple software running on completely locked down Apple hardware. It's almost first party only – about as close as you can get to a console platform and still call yourself a computer. I guess you'd say I chose Woz over Jobs. The way Jobs ruthlessly crushed the fledgling clone market in 1997 only reinforced this lesson for me.

So let's be completely clear: when you buy a new Mac, you're buying a giant hardware dongle that allows you to run OS X software.

You know, a dongle:

A dongle is a small hardware device that connects to a computer, often to authenticate a piece of software. When the dongle is not present, the software runs in a restricted mode or refuses to run. Dongles are used by some proprietary vendors as a form of copy prevention or digital rights management because it is much harder to copy the dongle than to copy the software it authenticates. Vendors of software protection dongles (and dongle-controlled software) often use terms such as hardware key, hardware token, or security device in their written literature. In day-to-day use however, the jargon word "dongle" is much more commonly used.

There's nothing harder to copy than an entire MacBook. When the dongle – or, if you prefer, the "Apple Mac" – is present, OS X and Apple software runs. It's a remarkably pretty, well-designed machine, to be sure. But let's not kid ourselves: it's also one hell of a dongle.

If the above sounds disapproving in tone, perhaps it is. There's something distasteful to me about dongles, no matter how cool they may be. But it's seductive, too. I wonder if the console model that Jobs is aping isn't some temporary evolutionary dead end, but in fact, the model for all future computing. People buy consoles like the Xbox 360 and Wii because they work with a minimum of fuss. Similarly, people buy Apple hardware because of the perfect synergy between the Apple hardware, OS X, iTunes, iLife, iMovie, iPhoto, and countless other software packages expressly designed to run on a closed hardware platform. "It just works." And why wouldn't it? There are no crude, selfish third parties to screw the experience up behind your back. No oddball hardware, no incompatible drivers, no more software which has to deal with a combinatorial explosion of potential configurations. Choosing to run proprietary software and hardware is just that, a choice. If it's working for consumers, who am I to judge?

I find Apple's brand of hardware lock-in particularly egregious. On the other hand, I run Windows, so I'm subject to my own flavor of self imposed software lock-in. Others have made different choices. In of canaries and coal mines, Mark Pilgrim revisits his choice to abandon Apple's proprietary software model for the world of free software.

18 months later, Apple has sold 4 million crippled phones, billions of crippled songs, and people are predicting that Mac sales are up 40% year over year. And I wouldn't bet against their new movie rental venture either.

So after 18 months, I think we can safely say that no, Cory and I were not "canaries in the coal mine." There are not hordes of fed-up consumers rejecting Apple's vision of cryptographic lock-in. There are not mass graves where people ceremoniously dump their crippled, non-general-purpose computing devices. Outside of Planet Debian and my own personal echo chamber, nobody gives a sh*t about Freedom 0.

You knew this, of course, but I just wanted to let you know that I knew, too.

Maybe I'm a hypocrite. Maybe the issue cuts philosophically deeper than mere dongles. Maybe it's not only about the freedom to run your operating system on whatever hardware you wish, but also the freedom to run whatever software you want for whatever purpose you need, in perpetuity. That's Freedom Zero:

Freedom 0 is the freedom to run the program, for any purpose. WordPress gives me that freedom; Movable Type does not. It never really did, but it was free enough so we all looked the other way, myself included. But Movable Type 3.0 changes the rules, and prices me right out of the market. I do not have the freedom to run the program for any purpose; I only have the limited set of freedoms that Six Apart chooses to bestow upon me, and every new version seems to bestow fewer and fewer freedoms. With Movable Type 2.6, I was allowed to run 11 sites. In 3.0, that right will cost me $535.

WordPress is Free Software. Its rules will never change. In the event that the WordPress community disbands and development stops, a new community can form around the orphaned code. It's happened once already. In the extremely unlikely event that every single contributor (including every contributor to the original b2) agrees to relicense the code under a more restrictive license, I can still fork the current GPL-licensed code and start a new community around it. There is always a path forward. There are no dead ends.

Movable Type is a dead end. In the long run, the utility of all non-Free software approaches zero. All non-Free software is a dead end.

It's compelling rhetoric. As a software developer, there's no denying that open source software is a powerful and transformative force in modern software development.

The console model, and Apple's de-facto first party development model, are about as far as you can get from Mark's freedom zero – instead, you get zero freedom. You hand the vendor a pile of cash and they allow you to do a handful of specific things with their device, for only so long as they're inclined to do so. It's hardly fair. In fact it's completely unfair; they can legally pull the rug out from under you at any time. But it can still result in some incredibly useful relationships with products that solve very real problems for the user. As Jaron Lanier notes, the iPhone was not a product of freedom zero:

Twenty-five years later, that concern seems to have been justified. Open wisdom-of-crowds software movements have become influential, but they haven't promoted the kind of radical creativity I love most in computer science. If anything, they've been hindrances. Some of the youngest, brightest minds have been trapped in a 1970s intellectual framework because they are hypnotized into accepting old software designs as if they were facts of nature. Linux is a superbly polished copy of an antique, shinier than the original, perhaps, but still defined by it.

Before you write me that angry e-mail, please know I'm not anti-open source. I frequently argue for it in various specific projects. But a politically correct dogma holds that open source is automatically the best path to creativity and innovation, and that claim is not borne out by the facts.

Why are so many of the more sophisticated examples of code in the online world – like the page-rank algorithms in the top search engines or like Adobe's Flash – the results of proprietary development? Why did the adored iPhone come out of what many regard as the most closed, tyrannically managed software-development shop on Earth? An honest empiricist must conclude that while the open approach has been able to create lovely, polished copies, it hasn't been so good at creating notable originals. Even though the open-source movement has a stinging countercultural rhetoric, it has in practice been a conservative force.

So I'll ask again, since Mark brought it up: why doesn't anyone give a crap about freedom zero?