Coding Horror

I was tickled to see that James Hague chose The Zen of Assembly Language Programming as one of five memorable books about programming. I wholeheartedly agree. Even if you never plan to touch a lick of assembly code in your entire professional career, this book is a fantastic and thoroughly useful read. I was a mere Visual Basic programmer when I found this book (along with The Zen of Code Optimization), picked it up on a lark, and I could barely put it down. It's that good.

Abrash isn't just a seminal figure in the software engineering community, he's also one of the best technical writers you'll ever find. That's why he's one of my programming heroes, directly alongside Steve McConnell.

His Graphics Programming Black Book is similarly great, and covers topics so general and wide ranging that the title becomes a bit of a misnomer. Best of all, it's available online for free courtesy of Byte, so you can sample it yourself.

I know what you're thinking. "This book is about graphics. And assembly language. Plus it's from, like, 1996, which is approximately 1928 in computer years. It's of no interest to me as a programmer." Admit it. You are. But you know what you're going to do? You're going to click through anyway and read some of it. Just like in college, the class topic doesn't matter when the instructor is a brilliant teacher. And that's exactly what Abrash is.

Abrash is a world class coder and technical writer, but he's also not shy about explaining the perils and dangers of our craft, including the biggest problem of all-- the one that sits behind the keyboard. Allow me to illustrate with one of my very favorite Abrash passages, from Chapter 16 of the Graphics Programming Black Book.

Not so long ago, Terje Mathisen, who I introduced earlier in this book, wrote a very fast word-counting program, and posted it on BIX. When I say it was fast, I mean fast; this code was optimized like nobody's business. We're talking top-quality code here.

When the topic of optimizing came up in one of the BIX conferences, Terje's program was mentioned, and he posted the following message: "I challenge BIXens (and especially mabrash!) to speed it up significantly. I would consider 5 percent a good result." The clear implication was, "That code is as fast as it can possibly be."

Naturally, it wasn't; there ain't no such thing as the fastest code (TANSTATFC? I agree, it doesn't have the ring of TANSTAAFL).

[assembly language tricks and useful optimization approaches elided -- see PDF for full detail]

The biggest optimization barrier that Terje faced was that he thought he had the fastest code possible. Once he opened up the possibility that there were faster approaches, and looked beyond the specific approach that he had so carefully optimized, he was able to come up with code that was a lot faster. Consider the incongruity of Terje's willingness to consider a 5 percent speedup significant in light of his later near-doubling of performance.

In the same chapter, Mr. Abrash relates a similar anecdote based on a word counting program. It was published as a challenge in his "Pushing the Envelope" column:

That initial challenge was sparked by a column David Gerrold wrote concerning the matter of counting the number of words in a document; David turned up some pretty interesting optimization issues along the way. David did all his coding in Pascal, pointing out that while an assembly language version would probably be faster, his Pascal utility worked properly and was fast enough for him.

It wasn't, however, fast enough for me. The logical starting place for speeding up word counting would be David's original Pascal code, but I'm much more comfortable with C, [so I created] a loose approximation of David's word count program, translated to C.

Mike proceeds to do what he does best-- optimize the word count program into assembly and explain along the way in an easy going, highly articulate way. His results are as follows:

He then posted his program as a challenge for readers of PC Techniques-- can this optimized assembly word count program, from an acclaimed industry expert on assembly optimization, be made even faster? Well, I think you can guess what happened next.

So how did the entrants in this particular challenge stack up? More than one claimed a speed-up over my assembly word-counting code of more than three times. On top of the three-times speedup over the original C code that I had already realized, we're almost up to an order of magnitude faster. You are, of course, entitled to your own opinion, but I consider an order of magnitude to be significant.

Truth to tell, I didn't expect a three-times speedup; around two times was what I had in mind. Which just goes to show that any code can be made faster than you'd expect, if you think about it long enough and from many different perspectives.

Like Mike said, there ain't no such thing as the fastest code. If you think there is, you're probably the barrier standing in the way of further performance, not the code itself.

The original Tivo was one of the finest out of box experiences I've ever had as a consumer. I remember how exciting it was to tell friends about our newfound ability to pause live television, and how liberating it felt to be freed from the tyranny of television schedules. Imagine watching whatever you want, whenever you want! Of course, digital video recorders are no longer the rare, expensive creatures they were back in 2002, so some of that original Tivo luster is irretrievably lost.

But Tivo was more than a garden variety DVR. Its true beauty was the synthesis of an intuitive user interface, cool hardware, and an elegant remote. It fired on all cylinders. I loved my series one unit with a passion most people reserve for their newborn children. I upgraded it with an ethernet connector, and plopped in a larger hard drive so I could store a week's worth of television. Never mind that I'd have to quit my job to actually have time enough to watch all those shows. Somehow, the mere knowledge that I had a zillion episodes of Seinfeld or the Simpsons available for immediate viewing gave me a warm, fuzzy feeling in the nether regions of my geek brain. It was a kind of beautiful, pure technological freedom.

Under the hood, Tivo was based on Linux. With a series 1 box, you could access all the standard *nix tools, and install some cool hacks including a web UI. But all that ended when I upgraded from my Series 1 Tivo box to a Series 2. This epic 2003 Tivo community forum thread explains why:

Can the stand alone Series2 running 3.2 software be hacked to get a BASH prompt over the USB/Ethernet bridge?

No. The config files are protected by hashes, and the list and hash check program are in the kernel initrd, which is signed and checked by the boot ROM. Any changes and it will either replace the file or not boot at all.

Can the method used by people using DirectTivos running 3.1 be used on a stand alone 3.2?

No. Unlocked software exists for the DTivos (before Tivo updated it), but all the Series2 stand alones always had the protected OS and ROMs.

If it is not possible, is there any hope that it will be possible in the future?

Who knows? There was an initial hack (set a BASH_ENV variable that makes bash run a script), but Tivo now checks for that. Changing the boot ROM is difficult, as it's soldered to the system board. If it's flashable (I haven't seen a definate yes or no), you still have to get into the system to run a flash program.

In other words, Tivo added hardware protection in the Series 2 to prevent anyone from modifying the Tivo software. Tivo became another Xbox or Playstation console-- a locked-down, hardware protected platform. You'd somehow have to defeat the hardware protection (eg, install a modchip or flash ROMs) before you can modify anything. It looks like Series 2 protection was finally defeated by 2005, but I was long gone from the Tivo ecosystem by then, so I never saw it happen.

It's fair to ask "so what?" at this point. So we can't hack Tivo's series 2 hardware. I may not like it, but it's their hardware, not mine. They can build whatever protections into it they want, right? Ah, but there's the rub. Tivo is based on Linux, and Linux is licensed under the GPL. The GPL uses a "copyleft" license:

The simplest way to make a program free software is to put it in the public domain, uncopyrighted. This allows people to share the program and their improvements, if they are so minded. But it also allows uncooperative people to convert the program into proprietary software. They can make changes, many or few, and distribute the result as a proprietary product. People who receive the program in that modified form do not have the freedom that the original author gave them; the middleman has stripped it away.

In the GNU project, our aim is to give all users the freedom to redistribute and change GNU software. If middlemen could strip off the freedom, we might have many users, but those users would not have freedom. So instead of putting GNU software in the public domain, we "copyleft" it. Copyleft says that anyone who redistributes the software, with or without changes, must pass along the freedom to further copy and change it. Copyleft guarantees that every user has freedom.

This concept of software freedom is embedded deeply into the GPL. As promised, we indeed have the freedom to see the Tivo source code, copy it, and change it. Since the Tivo's hardware validates the software, access to the Tivo software becomes effectively meaningless. You can modify the software all you want, but you'll never be able to run it on your own Tivo!

You might, in fact, argue that Tivo subverted the very principles of the GPL they built their business on. Richard Stallman certainly did. He felt so strongly about this perceived subversion of the GPL that he literally went back and rebuilt the GPL license to prevent what Tivo did:

However, there are those that want to use GPL-covered software for this purpose, and they want to do so by turning freedom number one into a sham, a facade. So they plan to do something like, make a modified version of the GPL-covered program, which contains code to restrict you, and distribute that to you and somehow arrange that you can't really modify it, or if you modify it it won't run, or if you modify it and operate it, it won't operate on the same data.

They do this in various ways. This is known as Tivoization because this is what the Tivo does. The Tivo includes some GPL-covered software. It includes a GNU+Linux system, a small one, but it does, and you can get the source code for that, as required by the GPL because many parts of GNU+Linux are under the GPL, and once you get the source code, you can modify it, and there are ways to install the modified software in your Tivo and if you do that, it won't run, period. It does a checksum of the software and it verifies that it's a version from them and if it's your version, it won't run at all. This is what we are forbidding, with the text we have written for GPL version three. It says that the source code they must give you includes whatever signature keys, or codes that are necessary to make your modified version run.

Businesses can no longer adopt GPL software, then incorporate hardware protections to effectively prevent the software freedoms that the GPL specifically guarantees. This is nothing less than a full frontal assault on everyone's favorite technology, Digital Rights Management. Note that there is a sizable loophole, however. The final version of GPL v3 (section 6) states that the signing key does not have to be provided when the software is distributed to businesses. But this was clearly done only grudgingly, and after intense resistance; the FAQ chides us: "we think it's unfortunate that people would be willing to give up their freedom like this."

The way Tivo built their business around the GPL and then completely subverted it with hardware protection does rankle. But I also wonder how a company like Tivo could make money if users could simply recompile the Tivo software to stop phoning home and billing them. Like consoles, the Tivo hardware is typically sold at a big loss to subsidize the platform. If that hardware could be easily formatted and the software rebuilt, you've created a permanent loss leader. So I can empathize with their desire to control the platform.

I'm not sure where I stand on the tivoization clause in GPL v3. For what it's worth, as much as I adored my Tivo, I abandoned the platform years ago as a lost cause. I've already said that I think a compelling product can make me overlook the DRM, so maybe my opinion is already suspect.

We programmers are obsessive by nature. But I often get frustrated with the depth of our obsession over things like code coverage. Unit testing and code coverage are good things. But perfectly executed code coverage doesn't mean users will use your program. Or that it's even worth using in the first place. When users can't figure out how to use your app, when users pass over your app in favor of something easier or simpler to use, that's the ultimate unit test failure. That's the problem you should be trying to solve.

I want to run up to my fellow programmers and physically shake them: think bigger!

A perfect example of thinking bigger is Alan Cooper's Interaction 08 keynote, An Insurgency of Quality.

There's a transcript of his keynote available, or you can view a video of his keynote video with the slides in place.

Alan is a well known interaction designer , and the author of several classic books in the field, such as About Face, and a few others that are on my recommended reading list. In the Q&A after the talk, he had this to say:

"We are not very important because we don't cut code." (A boo and hiss from the audience.) In the world of high-technology, if you cut code, you control things. It's the power to destroy the spice, it's the power to control the spice. It's not a fine kind of control: it's bruce-force kind of things. [Interaction designers are] largely marginalized. We're constantly asking for permission from the folks who shouldn't be in a position to grant permission. We should be working with business folks and marshalling the technology to meet the solutions to business problems.

But when it comes time to marshal the solution to the problems, we find ourselves slamming into this kind of Stay-Puft Marshmallow Man of software development.

We don't need to change interaction design; we need to re-orient organizations to build things right. When we come to programmers and say, "Look at the people I've talked to; look at the personas I've created" and present them with research, programmers understand that, and that's how we will influence.

It pains me to hear that Cooper considers most programmers twenty-story marshmallow barriers to good interaction design. Please don't be one of those programmers. Learn about the science of interaction design. Pick up a copy of Don't Make Me Think as an introduction if you haven't already. There's a reason this book is at the top of my recommended reading list. Keep your unit testing and code coverage in perspective -- the ultimate unit test is whether or not users want to use your application. All the other tests you write are totally irrelevant until you can get that one to pass.

In Where the Heck is My Focus, I wondered why web developers don't pay attention to basic keyboard accessibility issues. I don't want to navigate the entire web with my keyboard. That's unrealistic. I was specifically referring to login pages, which tend to be quite spartan and minimal. On a simple login web page, the standard keyboard tab, enter, focus order navigation scheme is quite useful and much more efficient than using the mouse.

But why is keyboard navigation so unrealistic for the rest of the web? Probably because the existing keyboard navigation paradigm was developed for the earliest GUIs, where forms had at most dozens of selectable items.

Compare with the modern web, where pages regularly have hundreds of selectable items.

Presented with a 20-fold increase in selectable items, it's not too difficult to see why traditional keyboard navigation techniques completely break down. They were never designed to handle such complex navigation scenarios. Jan Miksovsky explains:

The first issue is one of scale: the page above has twenty times the number of focusable controls as the simple dialog. A user trying to use the keyboard to reach a link in the middle of the page might have to press the Tab key 125 times to reach it. (Or, if they were exceptionally efficient, they could tab around the other direction and only have to press Shift+Tab 75 times.) The second issue is that the page has a much more complex two-dimensional columnar layout that the dialog, but that layout cannot be captured in the one-dimensional tab order. To the user, the behavior of the Tab key is therefore quite unpredictable.

The other standard keyboard navigation technique-- explicit keyboard shortcuts-- are also inadequate for complex user interfaces. Microsoft Windows allows users to move the focus directly to a control on the dialog by pressing a keyboard shortcut, generally the Alt key plus a single letter in the control's label. (OS/X does this too, although I find it less discoverable and generally weaker in execution.) This system is workable for dialogs with a small number of controls and a reasonable distribution of letter frequencies in control labels, but is obviously unable to scale well beyond a handful of controls.

Incremental search is one way to find what you're looking for on a complex web page. Safari does incremental searching extraordinarily well, Firefox reasonably well, and IE not at all unless you install a third-party plugin. As useful as incremental search is, it can be a jarring navigational technique.

Jan describes an alternate navigational technique that can scale to hundreds of selectable items. It's not even new. You've probably used it before, but not on your desktop or laptop PC. That technique is spatial navigation.

A much better user interface for navigating screens with lots of elements is already ubiquitous-- but not on PCs. It's found on mobile phone web browsers, which of necessity do a good job at keyboard navigation. They support two-dimensional directional navigation by using Left, Right, Up and Down arrow keys (or a joystick) to move to the "nearest" element in the corresponding direction. For example, if you press the Right key, heuristics determine whether there's an element you might be trying to reach towards the right, and if there are multiple elements, which element you probably want.

Significantly, these heuristics respect the rendered visual representation of the page, not the structure of the document's object model or the original location of elements at design time. This is necessary to account for the fact that the user may be viewing the page at a different width than the designer used, with different fonts, at different sizes, etc. Directional navigation UIs also tightly connect keyboard focus and scroll position, allowing someone to continually press the Up and Down keys to move through focusable controls and to page over large blocks of text.

Jan said "directional navigation works so well on mobile devices, I'm hoping it will get built into a browser someday." What he apparently didn't realize is that at least one browser already implements spatial navigation. That browser is Opera. In Opera, you can press shift+arrow to move the focus to the next logical selection in that direction.

Opera's spatial navigation is fun to play with. Combined with the space bar and arrow keys to scroll the page, it's a surprisingly effective navigation technique even outside the constraints of mobile browsers where you usually see it. But there are some quirks. It isn't always obvious what selectable item is next in any particular direction. Also, heavy use of JavaScript page manipulation appears to interfere with spatial navigation in some cases.

Try it yourself. Despite the quirks, spatial navigation is worlds better than the insanity of pressing tab 125 times.

It's too bad Opera doesn't get more respect. I'm as guilty as anyone; when I'm testing something, I'll use IE, Firefox, and Safari in that order. Opera isn't even on my radar. That's a shame, because as you can see, it's quite innovative in some areas. It's also historically one of the fastest browsers on the market. Opera is the default browser on the Nintendo DS and Wii, and I've heard nothing but raves for Opera Mini and Opera Mobile on mobile phones. Yet Opera's PC market share remains vanishingly small, on the order of 1 percent-- a very distant fourth behind the big three. I suppose part of that is Opera's fault; Opera was sold as a product long after browsers were given away for free. That certainly didn't help their market share.

Regardless, it's worth checking out Opera for spatial navigation and other innovations it brings to the table. Now that the browser wars have heated up again, I hope there will be more cross-pollination of innovative features so everyone can benefit.

As far back as I can remember-- which admittedly isn't very far-- GUI toolkits have included a special type of text entry field for passwords. As you type, the password field displays a generic character, usually a dot or asterisk, instead of the character you actually typed.

I've criticized the login dialog before, but I definitely understand the need to obfuscate password entry, even if you're using fancy two-factor authentication with smart cards and the like. If password entry was treated as plain old text entry, you'd reveal your password (or PIN code) to anyone who casually happened to be looking at the screen while you're typing. So instead of seeing:

**************

Everyone in your meeting or presentation would instead see:

IHeartBunnies!

Which would be sort of traumatic on several levels. Not to mention the security implications.

I can't talk about login dialogs without bringing up one in Lotus Notes 6.0. Like everything else in Notes, it's a massive trainwreck.

This dialog box contains several security "features":

• The hieroglyphics on the left of the dialog box are supposed to distract anyone who is peering over your shoulder trying to learn your password as you type.
• The number of characters you type is hidden; a random number of X's appear instead of one asterisk per character.

Is any of this nonsense really necessary? If I want to learn someone's password as he or she types it, I will look at the keyboard, not the screen!

I actually had to use that exact login dialog for my job at the time, and I can tell you from personal experience exactly how mind-bendingly, appallingly awful it truly was. Who reinvents a perfectly standard dialog-- and makes it so much worse? On second thought, perhaps "how can we make this worse?" was the design goal for Notes. It certainly felt that way while I was using it.

But I digress. As much as we worry about password obfuscation, at least one dialog in Vista bucks this long-standing GUI trend. Specifically, the dialog where you enter your wireless network password.

Checking the "display characters" checkbox overrides the password obfuscation and reveals the password. At first I was appalled. Reveal my password? Imagine the security implications! The chutzpah of Microsoft's developers, putting my password at risk in such a careless, haphazard manner! What were they thinking?

I'm guessing they implemented the reveal option here because network passwords can be unusually long and complex-- and troubleshooting network connectivity is difficult enough even without factoring in the inevitable password typos. But are network passwords really so different from any other type of password? After using this dialog a few times, I began to see how useful the reveal password option truly was. If you think you've made a mistake entering your password, tick the reveal box and find out. It's quite a time saver compared to typing in your password in blindly two, three, or even four times before getting it right. I don't know about you, but that happens to me at least a few times a day on average.

I've come full circle. I now think the password reveal option should be available on all login dialogs.

It's awfully convienient, and it doesn't seem particularly risky to me. Nobody leaves their password typed in and waiting to be revealed on the login screen. If you're in a public place, you simply refrain from using the reveal option. But at home or in a private work area, why not opt to reveal your password? Traditional GUI password obfuscation is a nice convention, but it's not the alpha and omega of password security. Far from it. If criminals really want to get your password, they'll be watching your fingers on the keyboard or using keylogger hardware.