Coding Horror

Everywhere I look, programmers and programming tools seem to have standardized on XML. Configuration files, build scripts, local data storage, code comments, project files, you name it -- if it's stored in a text file and needs to be retrieved and parsed, it's probably XML. I realize that we have to use something to represent reasonably human readable data stored in a text file, but XML sometimes feels an awful lot like using an enormous sledgehammer to drive common household nails.

I'm deeply ambivalent about XML. I'm reminded of this Winston Churchill quote:

It has been said that democracy is the worst form of government except all the others that have been tried.

XML is like democracy. Sometimes it even works. On the other hand, it also means we end up with stuff like this:

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<SOAP-ENV:Body>
<m:GetLastTradePrice xmlns:m="Some-URI">
<symbol>DIS</symbol>
</m:GetLastTradePrice>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

How much actual information is communicated here? Precious little, and it's buried in an astounding amount of noise. I don't mean to pick on SOAP. This blanket criticism applies to XML, in whatever form it appears. I spend a disproportionate amount of my time wading through an endless sea of angle brackets and verbose tags desperately searching for the vaguest hint of actual information. It feels wrong.

You could argue, like Derek Denny-Brown, that XML has been misappropriated and misapplied.

I find it so interesting that XML has become so popular for such things as SOAP. XML was not designed with the SOAP scenarios in mind. Other examples of popular scenarios which deviate XML's original goals are configuration files, quick-n-dirty databases, and [RSS]. I'll call these 'data' scenarios, as opposed to the 'document' scenarios for which XML was originally intended. In fact, I think it is safe to say that there is more usage of XML for 'data' scenarios than for 'document' scenarios, today.

Given its prevalence, you might decide that XML is technologically terrible, but you have to use it anyway. It sure feels like, for any given representation of data in XML, there was a better, simpler choice out there somewhere. But it wasn't pursued, because, well, XML can represent anything. Right?

Consider the following XML fragment:

<memo date="2008-02-14">
<from>
<name>The Whole World</name><email>us@world.org</email>
</from>
<to>
<name>Dawg</name><email>dawg158@aol.com</email>
</to>
<message>
Dear sir, you won the internet. http://is.gd/fh0
</message>
</memo>

Because XML purports to represent everything, it ends up representing nothing particularly well.

Wouldn't this information be easier to read and understand -- and only nominally harder to parse -- when expressed in its native format?

Date: Thu, 14 Feb 2008 16:55:03 +0800 (PST)
From: The Whole World <us@world.org>
To: Dawg <dawg158@aol.com>
Dear sir, you won the internet. http://is.gd/fh0

You might argue that XML was never intended to be human readable, that XML should be automagically generated via friendly tools behind the scenes, never exposed to a single living human eye. It's a spectacularly grand vision. I hope one day our great-grandchildren can live in a world like that. Until that glorious day arrives, I'd sure enjoy reading text files that don't make me suffer through the XML angle bracket tax.

So what, then, are the alternatives to XML? One popular choice is YAML. I could explain it, but it's easier to show you. Which, I think, is entirely the point.

<club>
<players>
<player id="kramnik"
name="Vladimir Kramnik"
rating="2700"
status="GM" />
<player id="fritz"
name="Deep Fritz"
rating="2700"
status="Computer" />
<player id="mertz"
name="David Mertz"
rating="1400"
status="Amateur" />
</players>
<matches>
<match>
<Date>2002-10-04</Date>
<White refid="fritz" />
<Black refid="kramnik" />
<Result>Draw</Result>
</match>
<match>
<Date>2002-10-06</Date>
<White refid="kramnik" />
<Black refid="fritz" />
<Result>White</Result>
</match>
</matches>
</club>

players:
Vladimir Kramnik: &kramnik
rating: 2700
status: GM
Deep Fritz: &fritz
rating: 2700
status: Computer
David Mertz: &mertz
rating: 1400
status: Amateur
matches:
-
Date: 2002-10-04
White: *fritz
Black: *kramnik
Result: Draw
-
Date: 2002-10-06
White: *kramnik
Black: *fritz
Result: White

There's also JSON notation, which some call the new, fat-free alternative to XML, though this is still hotly debated.

You could do worse than XML. It's a reasonable choice, and if you're going to use XML, then at least learn to use it correctly. But consider:

1. Should XML be the default choice?
2. Is XML the simplest possible thing that can work for your intended use?
3. Do you know what the XML alternatives are?
4. Wouldn't it be nice to have easily readable, understandable data and configuration files, without all those sharp, pointy angle brackets jabbing you directly in your ever-lovin' eyeballs?

I don't necessarily think XML sucks, but the mindless, blanket application of XML as a dessert topping and a floor wax certainly does. Like all tools, it's a question of how you use it. Please think twice before subjecting yourself, your fellow programmers, and your users to the XML angle bracket tax. <CleverEndQuote>Again.</CleverEndQuote>

You've probably read this classic boner of an iPod quote at some point:

No wireless. Less space than a nomad. Lame.

It's from the Slashdot article on the introduction of the original Apple iPod back in 2001. I had always assumed this particular quote was written by a random Slashdot user in the comments. But in fact, that quote is part of the body of the news entry, and it came directly from Rob Malda, the founder of Slashdot.

Rob's pithy dismissal of the iPod at its introduction has become virtually synonymous with how out of touch the Slashdot crowd is with the rest of the world. It's ripe for parody, as Andy Baio explains:

This is nothing new. It's as old as communication itself. I'm sure that the moment man discovered fire, there was some guy nearby saying, "Too smoky. Can burn you. Lame."

The success of the iPod was anything but a foregone conclusion back in 2001. A quick peek at the first iPod ad provides a little context to how rough that first generation really was compared to the competely polished product we enjoy on store shelves today. But the iPod, and the companion iTunes Store, have been hugely successful:

• The iTunes Store is the number one music retailer in the US
• Over 50 million customers
• Over 4 billion songs sold
• Music catalog of over 6 million songs

Clearly Apple is doing something right. Except there's one small problem. Music purchased from the Apple store comes encumbered with Apple's flavor of Digital Rights Management, known as FairPlay:

1. Users can make a maximum of seven CD copies of any particular playlist containing songs purchased from the iTunes Store.
2. Users can access their purchased songs on a maximum of five computers.
3. Songs can only be played on a computer with iTunes or an iPod; other mp3 devices do not support FairPlay encoded tracks.

(EMI and independent artists are also offered in "iTunes Plus" DRM-free format -- at last count, around 2 million songs. These songs were originally sold at a 30 cent premium, but later reduced to the standard 99 cents.)

Now, I'm a pragmatist. I'm no fan of DRM, but I do accept that sometimes it is a necessary evil. FairPlay was indeed an acceptable tradeoff when Apple's iTunes Store was one of the few easy and legal ways to get digital music of any kind.

But that's no longer true today. You can generally get the same music for the same price, or less, at Amazon's MP3 store -- completely free of any form of DRM! Reg Braithwaite provides an example:

For example, Bach: Goldberg Variations, BWV 988 on 256-bit DRM-free MP3 is just $9.99 from Amazon. The same album is also $9.99 from Apple, but you get DRM. And there are tons of tracks on Amazon that are actually less expensive than on iTMS, so you get better music for less money without the DRM hassle.

Better quality. Less money. And no evil, consumer hostile DRM! It's almost unbelievable. Needless to say, I've been buying as much music as I can from Amazon to vote with my wallet and demonstrate to the music labels that yes, giving the customer what they want does pay. And you should too. Every purchase of DRM-ed music, in the face of Amazon's excellent alternative, is an implicit vote for more useless, aggravating DRM on your music.

If it seems a little odd to you that Amazon is somehow able to offer all this music up DRM-free, while the majority of Apple's iTunes Store catalog is still stuck in the old testament world of DRM customer punishment, you're not alone.

The reason you can find more music on Amazon at a lower price is that the Record Labels want it that way. Do you think they charge Apple and Amazon the same price for each track and Apple simply charges you more and pockets the difference as a higher markup? The labels would like you to think that, but they actually charge Amazon less for each track, and that's how Amazon can charge you less.

Do you think Apple insists on the DRM but Amazon has the vision to see that the future of music is DRM-free? Do you think Jeff Bezos is a better negotiator and he was able to get a better price per track than Steve Jobs? Without putting up with DRM?

The major labels want nothing more than to break Apple's dominance of the digital music business. They spin it as a good thing. More retailers means more competition, which is good for consumers.

The record labels now view the massive iTunes juggernaut as a threat. Thus, the offer of DRM-free music exclusively to Amazon, and at lower prices than Apple can offer, is a direct attack by the record labels on the increasing power of Apple's iTunes. The irony of the record labels attacking a Frankenstein monster of their very own creation is almost overwhelming -- who do you think demanded that all the music on iTunes have DRM in the first place?

But here's where Reg Braithwaite and I differ: he argues that poor Apple is getting a raw deal.

And if the whole world can sell DRM-free music, then Amazon and the like would have to compete with iTMS by building a better music store. Except, of course, they don't have to compete with iTMS because the labels are colluding to place Apple at a disadvantage.

I'll certainly agree that the stunning success of the iTunes Store is what led us to this competitive situation in the first place, and that's entirely to Apple's credit. Correct me if I'm wrong, but I believe they've made quite the handsome profit along the way, too.

But to argue that the competition is "unfair" smacks of the absolute worst kind of Apple advocacy. Unfair? Unfair to whom? The customers who are getting DRM-free music officially blessed by the major record labels?

Yeah, that's terrible. Just awful.

Hold on for a minute while I wipe this tear out of my eye. Try to imagine me playing my DRM-free MP3 of REM's "Everybody Hurts" while I'm doing it, for maximum effect.

As soon as they can break this pesky iPod-iTMS-iPhone nonsense, the labels want to get back to dictating what you pay and how often you pay.

You'll get no argument from me that the RIAA and the major record labels are as close as you can get to pure evil while not actively killing small children, puppies, and kittens. Well, not in public, anyway. I'm sure they'd be charging us a trillion dollars per song -- no, per byte of the song -- if they could get away with it.

But clearly, they can't. There are certain market realities at work here. There's absolutely no historical evidence that a type of media, once it is officially sold DRM free, can somehow revert back to the DRM model. I am somehow reminded of software developers who desperately try to "revoke" the GPL after they've adopted it.

So if the labels want to disrupt the power of the iTunes machine by doing the right thing for customers and irrevocably breaking the back of DRM on music, that is the beauty of pure competition working for us, the users. This is a level of progress on the DRM front that I thought we would never see.

If it takes "the labels break[ing] Apple", as Reg says, to get us this far, then so be it. That kind of invective may be difficult to read if you're emotionally involved with Apple. But let me tell you, I've been emotionally involved with companies before, and it rarely ends well. I find that corporations never reciprocate your love in quite the same way.

Personally, I'd much rather be an advocate for my fellow users than an advocate of any one particular company. For now, that means supporting Amazon's DRM-free MP3 store. I'm pretty sure the good folks at 1 Infinite Loop will survive, one way or another.

Like everything else in software engineering, it seems, the concept of Model-View-Controller was originally invented by Smalltalk programmers.

More specifically, it was invented by one Smalltalk programmer, Trygve Reenskaug. Trygve maintains a page that explains the history of MVC in his own words. He arrives at these definitions in a paper he published on December 10th, 1979:

1. Models

   Models represent knowledge. A model could be a single object (rather uninteresting), or it could be some structure of objects.

   There should be a one-to-one correspondence between the model and its parts on the one hand, and the represented world as perceived by the owner of the model on the other hand.

2. Views

   A view is a (visual) representation of its model. It would ordinarily highlight certain attributes of the model and suppress others. It is thus acting as a presentation filter.

   A view is attached to its model (or model part) and gets the data necessary for the presentation from the model by asking questions. It may also update the model by sending appropriate messages. All these questions and messages have to be in the terminology of the model, the view will therefore have to know the semantics of the attributes of the model it represents.

3. Controllers

   A controller is the link between a user and the system. It provides the user with input by arranging for relevant views to present themselves in appropriate places on the screen. It provides means for user output by presenting the user with menus or other means of giving commands and data. The controller receives such user output, translates it into the appropriate messages and pass these messages on to one or more of the views.

It may seem like we're deep in Architecture Astronaut territory now, but bear with me. The MVC concepts are a little abstract, it's true, but it's an incredibly common pattern. It is literally all around you. In fact, let me bring it back down to Earth this way: you're looking at MVC right now.

Model = HTML	View = CSS	Controller = Browser

This ubiquitous trifecta represents MVC almost perfectly.

1. Model

   The HTML is the "skeleton" of bedrock content. Text that communicates information to the reader.

2. View

   The CSS adds visual style to the content. It is the "skin" that we use to flesh out our skeleton and give it a particular look. We can swap in different skins via CSS without altering the original content in any way. They are relatively, but not completely, independent.

3. Controller

   The browser is responsible for combining and rendering the CSS and HTML into a set of final, manipulatible pixels on the screen. It gathers input from the user and marshals it to any JavaScript code necessary for the page to function. But here, too, we have flexibility: we can plug in a different brower and get comparable results. Some browsers might render it faster, or with more fidelity, or with more bells and whistles.

So if you believe the web has been at all successful -- most signs I've seen point to yes -- then you also have to acknowledge the incredible power of Model-View-Controller.

It's no coincidence that many of the most popular web programming frameworks also encapsulate MVC principles: Django, Ruby on Rails, CakePHP, Struts, and so forth. It's also officially creeping into ASP.NET under the fledgling ASP.NET MVC project.

Just take a gander at the project layout in a sample ASP.NET MVC project:

It's almost self-explanatory, if you've ever built an application of any kind:

1. Model

   The classes which are used to store and manipulate state, typically in a database of some kind.

2. View

   The user interface bits (in this case, HTML) necessary to render the model to the user.

3. Controller

   The brains of the application. The controller decides what the user's input was, how the model needs to change as a result of that input, and which resulting view should be used.

It's beautiful in its simplicity, as Terence Parr notes:

For the "MVC" of a web app, I make a direct analogy with the Smalltalk notion of MVC. The model is any of the logic or the database or any of the data itself. The view is simply how you lay the data out, how it is displayed. If you want a subset of some data, for example, my opinion is that is a responsibility of the model. The model knows how to make a subset. You should not be asking your graphics designer to filter a list according to age or some other criteria.

The controller in a web app is a bit more complicated, because it has two parts. The first part is the web server (such as a servlet container) that maps incoming HTTP URL requests to a particular handler for that request. The second part is those handlers themselves, which are in fact often called "controllers." So the C in a web app MVC includes both the web server "overlord" that routes requests to handlers and the logic of those handlers themselves, which pull the data from the database and push it into the template. This controller also receives HTTP POST requests and processes these, sometimes updating the database.

I look at a website as nothing but a graph with edges with POSTs and GETs that routes pages.

Here's one quick way to test if your application has properly segregated itself between the Model, View, and Controller roles: is your app skinnable?

My experience is that designers don't understand loops or any kind of state. They do understand templates with holes in them. Everybody understands mail merge. And if you say, "Apply the bold template to this hole," they kind of get that, too. So separating model and view addresses this very important practical problem of how to have designers work with coders.

The other problem is there is no way to do multiple site skins properly if you don't have proper separation of concerns. If you are doing code generation or sites with different skins on them, there is no way to properly make a new skin by simply copying and pasting the old skin and changing it. If you have the view and the logic together, when you make a copy of the view you copy the logic as well. That breaks one of our primary rules as developers: have only one place to change anything.

Skinnability cuts to the very heart of the MVC pattern. If your app isn't "skinnable", that means you've probably gotten your model's chocolate in your view's peanut butter, quite by accident. You should refactor your code so that only the controller is responsible for poking the model data through the relatively static templates represented by the view.

The power and simplicity of properly implemented MVC is undeniable. But the first step to harnessing MVC is to understand why it works, both on the web, and also within your own applications.

The Garmin Nuvi GPS first got my attention when it came not just recommended, but insanely recommended by Jason Fried in late 2005.

So, back to the 350… Oh wow. The Nuvi 350 is insanely good. Next to the iPod it's the the best piece of consumer electronics I've purchased in the last 5 years. It really is that good. It's perfectly executed.

In early 2006, Jan Miksovsky heaped similar levels of ebullient praise on the the Nuvi:

Garmin and other manufacturers have been making GPS units since the late 1980s, and during that time have continually made incremental improvements in size, form factor, performance, and UI. From time to time I've looked at the category, but beyond the flat-out magic of finding your way using satellites, I found little captivating about the products themselves. GPS units have suffered from a wide range of UI problems, such as the heavy use of jargon, awkward use of a few buttons to accomplish complex tasks (such as entering an address), and cumbersome systems for transferring maps to a device with limited memory.

Sometimes you encounter a product and get the strong feeling its the first one in its category to really be Designed, with a capital "D". In my case, TomTom had the first GPS with that distinction. From the branding to the startup sound to the UI, they had clearly thought about the product as a consumer experience. Despite breaking that ground, I still felt that the TomTom product I saw came up short.

The Garmin Nuvi is the first GPS I've seen that meets my bar for a good user experience. They've given a lot of thought to an overall package of functionality a traveler might want in a single pocket device. In addition to the GPS, the Nuvi unit includes an MP3 player, a photo vault, a currency converter, a world clock, a foreign language dictionary, and a travel guide. This is a good sign that Garmin's considering the overall user experience of the device, not just trying to make a housing for a satellite receiver.

I've used Microsoft Streets and Trips and an external USB GPS device to navigate unfamiliar areas for years. I've sort of ignored GPS devices until now, because at $500+ I figured my laptop was "good enough." It's the same solution I relied on when I moved to California in 2005. The laptop is unwieldy and sometimes frankly even a little dangerous as a navigation device, but people -- myself included -- will do crazy things to save a buck.

It's amazing the difference two years can make.

Prices have crashed on the Nuvi GPS models. I picked up a refurbished Nuvi 200W for $170 shipped. I'm a proud member of the Nuvi club at last.

After using it for about two weeks, I feel like an idiot for waiting so long.

I agree with Jan and David. The Nuvi offers an outstanding consumer experience in every possible way. I'd rate it up there with the original Tivo. Everything about it just works, to the point that it completely breaks you of the old ways of navigation. I'm so late to the party that I'm not sure I can add much more than Jan and David did in their excellent reviews, not to mention the thousands of other rave reviews on the internet. All I can offer is a belated confirmation that, yes, it is that good.

Bear in mind that I am an utter and complete GPS newbie. I don't geocache and to be completely honest, I don't even like to leave the house that much if I can avoid it. I'm what you might call an indoor enthusiast. There may be other GPS devices with better features or more functionality. I've been told the Nuvi 205W is an updated version of this older model I have, which partially explains why it's such a great deal. I do know enough to recommend the widescreen models; the pseudo-3D presentation scales much better on wider screens.

I also have one bit of accessory advice. You'll want a good universal mount for your GPS, and I can highly recommend this $12 Bracketron GPS friction mount.

This clever little add-on converts your default suction mount into a super flexible go-anywhere dashboard mount. Taking the GPS along in any vehicle becomes a no brainer. Just lay it anywhere on the dash, plug it into the cigarette lighter port for power, and off you go. It's rock solid.

I used to think of GPS devices as high-end geeky toys. Based on my Nuvi experience, that's no longer true. They're now so inexpensive, practical, and easy to use that -- exactly like Tivo did for DVRs -- everyone should have one. Really. If you don't have a GPS device, or if you think your parents or family could use one, but have been turned off by high prices in the past, check prices on the Nuvi series and see. I think you'll be pleasantly surprised.

And I strongly suspect that once you have a Nuvi, like me, you'll wonder how anyone ever lived without it.

Like Donald Knuth, I think much of the current multicore hype is overrated.

The machine I use today has dual processors. I get to use them both only when I'm running two independent jobs at the same time; that's nice, but it happens only a few minutes every week. If I had four processors, or eight, or more, I still wouldn't be any better off, considering the kind of work I do -- even though I'm using my computer almost every day during most of the day. So why should I be so happy about the future that hardware vendors promise? They think a magic bullet will come along to make multicores speed up my kind of work; I think it's a pipe dream. (No -- that's the wrong metaphor! "Pipelines" actually work for me, but threads don't. Maybe the word I want is "bubble.")

Despite that, I've acknowledged all along there are certain narrow tasks that the proliferation of CPU cores will make dramatically faster. And one of my very favorite tasks in that niche is encoding your DVD collection.

I bought my first DVD about 10 years ago. At the time, they were a technical marvel:

• 8.5 Gigabytes per side
• 720 x 480 MPEG-2 video at 30 frames per second
• Dolby Digital (AC-3) or Digital Theater System (DTS) digital multichannel sound

Today, those specs are rapidly becoming pedestrian in the face of high definition cable, broadcast, and Blu-Ray discs. A few of the video sharing websites offer something perilously close to DVD quality already.

I say the DVD is the new MP3. We're going to start tossing these things around like candy.

Unlike audio CDs, DVDs are already compressed digital data. You could extract the files from the DVD as-is, and play them back to your heart's content. No re-encoding required. But like The Six Million Dollar Man, we can rebuild them better than they were before. Video codecs have advanced tremendously since the heady days of MPEG-2. These new codecs take a lot more playback horsepower than MPEG-2, but offer comparable quality in about one-fourth the size. We can turn our digital DVDs into better digital DVDs through superior computer science.

But if you thought the playback performance demands of these new codecs were severe, wait until you see the encoding performance demands. It's only in the last year or two that typical CPUs could encode new-fangled MPEG-4 or VC1 at anything even approaching real time. But now, with extremely fast dual cores trickling all the way down to the mainstream, and quad-core CPUs carving out a decent share for themselves-- the average user could potentially rip and encode a typical DVD in less than 30 minutes. Per a recent H.264 benchmark dataset analysis, you can statistically expect to halve your encode time when going from 2 to 4 cores.. and almost do it again when you go from 4 to 8!

It helps, too, that there's great free software like Handbrake which makes it easy to harness that embarassment of desktop CPU power to encode your DVDs.

Yes, there are an intimidating number of knobs and dials to potentially tweak here, but what I like about handbrake is that I largely don't have to. There are some logical presets on the right (clipped out of the screenshot for size, unfortunately) which will get you headed in the right direction: AppleTV, iPod, Film, Xbox 360, and so forth. I do set a handful of variables, like the overall bitrate -- I prefer something between 900 and 1200 -- and whether I need to deinterlace the source. But I mostly let Handbrake use its "auto" defaults, and get excellent results. If you're curious about the details, there's a well-written description of many of the Handbrake settings.

I had the most luck with the H.264 codec, which is aggressively multithreaded. I achieved 30-40 fps on my modest new power efficient dual-core HTPC processor, and upwards of 100 fps on my overclocked 4 GHz dual-core. Fortunately, Handbrake supports a batch encode mode, so you can queue up a bunch of jobs to run overnight.

I was particularly excited to find that I can pass the digital audio directly through using the "AC3" setting for the audio encoder. That means, when playing these files back on a home theater PC, the digital audio arrives at your receiver in exactly the same way it would from a DVD, with a few minor adjustments in ffdshow. There is no audio degradation or conversion whatsoever! As something of an audiophile, I suffered mightily through many a re-encoded DVD that was downmixed to plain vanilla stereo over the years, so this is a huge step forward in my book.

So, in summary -- nearly the same digital video quality, exactly the same digital audio quality, all wrapped up in a single file less than a quarter of the original size of the DVD. Seriously, I get chills. It's geek nirvana. What's not to love about encoding your DVD collection? It's also a perfect use of those four CPU cores, which would otherwise lay idle 99% of the time.

Take, for example, one of my favorite movies, Idiocracy. I used Handbrake to convert this DVD from a set of files totalling 4.15 GB to a single 995 MB file, at almost no quality loss. See for yourself.

Still image captured from original DVD:

Still image captured from H.264 encoded video of DVD:

The above still was used in an earlier post, A World of Endless Advertising. I love being able to grab a movie file over the network and quickly get the exact still I need for a blog post. Yes, there is a tiny loss of fidelity -- particularly in the chair shadows on the bottom left, and the grain of the wall texture at the upper left. But I'm willing to live with that compromise if it means I don't have to pull ginormous 8 GB ISO images files over the network.

Why re-encode DVDs you already own? For convenience, mostly -- so you can watch them on your mobile devices, on your PCs, laptops, and anything else that vaguely resembles a computer in your home. Might as well put all those CPU horses to proper use. A re-encoded DVD is so much more flexible than those physical hunks of round, reflective plastic.