Coding Horror

In Why Isn't My Encryption.. Encrypting? we learned that your encryption is only as good as your understanding of the encryption code. And that the best encryption of all is no encryption, because you kept everything on the server, away from the prying eyes of the client.

In The Bathroom Wall of Code we learned the potential danger of copy-pasting code from the internet, and the continued importance of regular peer review for every line of code that enters your codebase, from whatever source.

I didn't anticipate this series becoming a trilogy, but apparently it has, because Thomas Ptacek of Matsano Security wrote a long blog entry about it. A blog entry masquerading as an overly dramatic college screenplay, but still. These guys, unlike us, are real security experts, so it's worth reading.

But you don't have to read that screenplay, because I'm going to reveal the twist in the final act right here.

1. The root problem wasn't failing to understand the encryption.
2. The root problem wasn't copy and pasting code from the internet.
3. The root problem wasn't failing to peer review the code.

Mr. Ptacek is absolutely right. The root problem was that we were working at the wrong layer of abstraction.

Rather than construct code from the low-level cryptography primitives provided in .NET, we should have used a library to handle our encryption needs. I'm reminded of a common Stack Overflow joke:

Q: How do I write this in JavaScript?

A: You don't. You use JQuery.

You can save a tremendous amount of time and effort by using the browser-independent framework that JQuery has spent untold man-hours testing, debugging, and proving in the field. While there's nothing wrong with writing JavaScript, why not speed your development time by writing to the library instead? As I've always said, don't reinvent the wheel, unless you plan on learning more about wheels.

Abstractions are important. You could view most of computer programming history as slowly, painfully clawing our way up the evolutionary tree of abstraction -- from assembly language, to C, to Java, to JavaScript, all the way up to JQuery, where the air starts to get pretty darn thin. We've already layered an operating system, web browser, and interpreted scripting language on top of each other to get to this point. It's a testament to the power of abstraction that any of it works at all.

Getting back to specifics: how can you stop programmers from working at the wrong layer of abstraction? One solution would be to disallow the .NET encryption primitives entirely. This is akin to Steve Gibson's holy crusade against raw socket programming in Windows XP. That's one way to do it, I suppose. But putting roadblocks in front of programmers is tantamount to a challenge; why not offer them more attractive alternatives, instead?

Hiding the low-level encryption primitives feels like a temporary solution. That said, I'd strongly recommend marking some of the older encryption methods as deprecated, so programmers who do stumble down some dusty old code path at least have some warning sign that they're using an algorithm with a lot of known vulnerabilities. I'm envisioning a Clippy that pops up with something like:

"Hey! It looks like you're using a method of encryption that's widely regarded as insecure by security experts! Would you like to see alternatives?"

One of those alternatives would be a full-blown library, perhaps something like Bouncy Castle, or Keyczar, or cryptlib. What could be easier than a EncryptStringForBrowser() method which has security and tamper-resistance built in, that's part of a proven, domain-expert-tested set of code that thousands if not millions of developers already rely on?

Using encryption libraries doesn't mean that crucial encryption mistakes will magically disappear overnight. But these libraries, because they force developers to work at a higher level of abstraction, do make it harder to misuse cryptography. And perhaps more importantly, usability improvements to the library can be better handled by the specialists who created the library, rather than the generalists working on the .NET framework itself.

So the next time you set out to write code -- not just encryption code, any code -- ask yourself: am I working at the right level of abstraction?

If you've followed my blog for any length of time, you know that I am a total regular expression fanboy. It's almost embarrassing how much I love the damn things. I'm pretty sure my teammates roll their eyes every time they see yet another class I've touched that has using System.Text.RegularExpressions at the top. You might as well rename it to JeffHasBeenHere.

I say that because I end up writing a lot of string handling code, even when people tell me I shouldn't. Now, I only advocate responsible and judicious use of regular expressions when you happen to be dealing with strings. In the wrong hands, regular expressions can be dangerous. You might end up wondering if Q*Bert just vomited all over your source code. Or you might be programming in Perl. Is there any difference? (instant rimshot)

But I digress. Although I love regex, I've never been a fan of the classic regular expression reference book, Friedl's Mastering Regular Expressions. I found it dry, a bit academic, and lacking in practical real world examples. It just didn't speak to me as a working programmer in the way that regular expressions themselves did, and that was disappointing.

That's why I was so excited to discover that two of the gnarliest regex gurus I knew – Jan Goyvaerts (author of RegexBuddy and regular-expressions.info) and Steven Levithan (author of XRegExp and RegexPal) – were putting their heads together to create a regular expression reference for the rest of us. I immediately pre-ordered it sight unseen.

That book is Regular Expressions Cookbook. It arrived a few days ago, and although my expectations were high, I think this book has exceeded even the loftiest expectations I had. It is outstanding.

What I love about this book is twothree things:

1. It's filled with practical, real world examples of RegEx use. At every step of the way, from beginner to master level, you're building regular expressions that are actually useful in the wild, and not just abstract, obtuse academic exercises in solving string matching puzzles.
   
   
2. It covers all the common gotchas that you inevitably run into when you start building non-trivial regular expressions. Things like the sometimes massive (and painful) differences between regex libraries in various languages, subtle regex flavor quirks, catastrophic backtracking, unicode support, and so forth. These are all presented in context of the solutions, exactly as you'd encounter them in real programming. I know because I have the scars to prove it.
   
   
3. It was updated and revised in the second edition to reflect current flavor differences with new recipes, plus a whole new chapter on source code and log files. It also covers XRegExp, an emerging standard, fully-featured regex library for JavaScript.

Regular Expressions Cookbook manages to be simultaneously accessible and almost ridiculously comprehensive. I consider myself a fairly advanced regex user and about 50 pages in I've already had three big "oh, wow, I didn't realize that" moments. In my mind, at least, this completely replaces the Friedl book as the go-to reference for programmers of any skill level or background who seek regular expression enlightenment.

Needless to say, recommended.

Unix turns 40: The past, present and future of a revolutionary OS is fascinating reading.

Forty years ago this summer, a programmer sat down and knocked out in one month what would become one of the most important pieces of software ever created.

In August 1969, Ken Thompson (pictured at left), a programmer at AT&T subsidiary Bell Laboratories, saw the month-long departure of his wife and young son as an opportunity to put his ideas for a new operating system into practice. He wrote the first version of Unix in assembly language for a wimpy Digital Equipment Corp. (DEC) PDP-7 minicomputer, spending one week each on the operating system, a shell, an editor and an assembler.

The article is accompanied by a graph from wikipedia, illustrating the lineage of the Unix family.

To me, Unix has become synonymous with Linux, and the open source movement in general. The last *nixes standing shake out as follows:

I didn't realize there were that many closed source Unix variants still surviving in the wild. It's also odd how OS X brings us full circle with the original Unics and BSD licensing. If it's lonely in the "Closed" column, imagine the existential angst of being the only vendor in the "Mixed / Shared Source" column. (NB: I think the currently tiny category Apple occupies represents the future of commercial software, but that's a topic for another blog post.)

I've been primarily a Windows developer since the early 90s, but over time, I've developed a grudging respect for Unix. I think Michael Feathers summarized it best:

There's something deep in software development that not everyone gets but the people at Bell Labs did. It's the undercurrent of "the New Jersey Style", "Worse is Better", and "the Unix philosophy" - and it's not just a feature of Bell Labs software either. You see it in the original Ethernet specification where packet collision was considered normal.. and the same sort of idea is deep in the internet protocol. It's deep awareness of design ramification - a willingness to live with a little less to avoid the bigger mess and a willingness to see elegance in the real rather than the vision.

I find this to be deeply and profoundly true in everything I've ever worked on as a programmer, and to the extent that Unix reflects these philosophies, it is undeniably on the right path. Unlike Rich Skrenta, I didn't grow up as a Unix developer, so I have come late in life to this appreciation. Joel Spolsky's take on the Unix / Windows divide, after reading The Art of UNIX Programming, is this:

What are the cultural differences between Unix and Windows programmers? There are many details and subtleties, but for the most part it comes down to one thing: Unix culture values code which is useful to other programmers, while Windows culture values code which is useful to non-programmers. This is, of course, a major simplification, but really, that's the big difference: are we programming for programmers or end users? Everything else is commentary.

So on one side, you have hundreds of command line applications, built in wildly different styles, with thousands of arcane command line parameters, all of which can be flexibly combined together to accomplish almost anything. And on the other side, you have the windows registry and MFC.

Sometimes, you just can't win.

So, yes, I'm a fan of Unix. And I'm also a fan of Windows. I think it's worth studying what both are getting right and wrong, because as a programmer, I'm a fan of whatever the heck works.

Everybody loves BitTorrent. And rightfully so.

With BitTorrent, you also start by placing your large file on a central server. But once the downloading begins, something magical happens: as clients download the file, they share whatever parts of the file they have with each other. Clients can opportunistically connect with any other client to obtain multiple parts of the file at once. And it scales perfectly: as file size and audience size increases, the bandwidth of the BitTorrent distribution network also increases. Your server does less and less work with each connected client. It's an elegant, egalitarian way of sharing large files with large audiences.

BitTorrent radically shifts the economics of distribution. It's one of the most miraculous ideas ever conceived on the internet. As far as I'm concerned, there should be a Nobel prize for computing, and the inventor of BitTorrent should be its first recipient.

I've been a happy consumer of files distributed via BitTorrent for years; it was only natural that I would turn to BitTorrent to distribute our cc-wiki licensed Stack Overflow data. I figured serving a several-hundred megabyte file with BitTorrent wouldn't be much harder than downloading one. Boy, was I ever wrong. Sharing files with BitTorrent is way more complicated than downloading them! After two frustrating hours, I finally came up with a relatively straightforward way to share a file via BitTorrent, and in the interests of saving future readers a little time, I'm documenting it here.

Now, I'm going to show you an easy way, but it isn't technically the easiest way. The easiest way is to let someone else do the sharing for you. If you own content that you want to share, LegalTorrents is the obvious choice:

LegalTorrents^tm is an online digital media community.

We discover and distribute high quality open-license (Creative Commons) digital media and art, and provide support to Content Creators. We host creative content in its entirety, ensure fast, reliable downloads, and enable users to directly sponsor Content Creators and their work.

We distribute content with the full permission of the rights holders and use the peer-2-peer file-sharing technology called Bittorrent.

The site is still in beta, but signup is a snap, because they support OpenID! I encourage anyone interested to check it out. If nothing else, get the furtive thrill of actually downloading legal content through BitTorrent for once! Yes, it can happen. Shocking, I know. Don't worry, you crazy kids can get right back to your regular non-copyright-respecting torrenting ways immediately afterwards.

Anyway, you can't start sharing files on LegalTorrents without some kind of special email-us-please permission, and I was in a hurry. I wanted to share files via BitTorrent right now. I did, and you can too! But you'll need a few things first:

1. A copy of uTorrent (it's free!)
   
   
2. Your external IP address; if you don't know what it is, use http://www.whatismyipaddress.com to find out.
   
   
3. The uTorrent listen port. This is under Options | Preferences | Connection. This is typically set randomly every time uTorrent starts, so you may want to specify a more memorable value here.
   
   
4. You must have port forwarding properly configured so the outside world can get to your IP address and the port specified above. A full discussion of how to do this is outside the scope of this post, but it usually starts with your firewall settings and/or router configuration. uTorrent has a fairly nice help page at Options | Speed Guide that's a good start; just click the Test if port is forwarded properly button on that dialog to begin.

Here's where I hit a major roadblock: to share files via BitTorrent, you need a tracker.

A BitTorrent tracker is a server that assists in the communication between peers using the BitTorrent protocol. It is also, in the absence of extensions to the original protocol, the only major critical point, as clients are required to communicate with the tracker to initiate downloads. Clients that have already begun downloading also communicate with the tracker periodically to negotiate with newer peers and provide statistics; however, after the initial reception of peer data, peer communication can continue without a tracker.

Without a tracker, you're sort of hosed, as clients will never be able to find your file, much less each other. Unfortunately, most of the freely open, public trackers out there are sort of.. disreputable. And the LegalTorrents tracker won't track files unless they are on its creator whitelist, which involves that manual sign-up process. You've got precious few legit options for tracking, unless you're willing to take a trip to the wrong side of town, and associate yourself and your files with that kind of .. neighborhood. I wasn't.

Fortunately, uTorrent has a solution: you can become your own tracker!

1. in uTorrent, go to Options | Preferences | Advanced.
2. Scroll down to bt.enable_tracker and set it to True
3. Restart uTorrent.

Now, let's create the torrent for the file we want to host, which will point to our newly created tracker.

1. In uTorrent, click the Create New Torrent button.
2. Select the file or directory you want to share.
3. Enter your tracker in this format: http://my-ip-address:my-port/announce
4. That's it! Click Create and save the new *.torrent file you've created.

Now go forth and share your *.torrent file with the world. Share it with anyone and everyone! The more the merrier! Any client that opens your *.torrent file will attempt to connect to your tracker, download your file, and share it with other downloading clients in classic BitTorrent stylee. Pat yourself on the back; you just shared a file with the world using the transformative distribution power of BitTorrent!

But you do have to keep uTorrent running as a desktop application all the time, which is sort of a bummer. What if you wanted to share your file on a server, or via a silent background process? No problem. It's just a few more steps:

1. Enable the uTorrent web interface under Preferences, Web UI. Note that the URL for it is, by default, http://my-ip-address:my-port/gui/, and it requires a username and password to be set here.
   
   
2. Obtain a copy of the user-defined service utilities, srvany.exe and instsrv.exe. Copy them to the same folder as uTorrent.exe.
   
   
3. Issue this command to make uTorrent run as a service:
   
   

   instsrv uTorrent "C:uTorrentsrvany.exe"

   
   
4. Enter this registry file to set the path for the service named "uTorrent" you just created in the previous step:
   
   

   Windows Registry Editor Version 5.00
   [HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesuTorrentParameters]
   "Application"="C:uTorrentuTorrent.exe"
   

   
   
5. In Control Panel, Services, set the account that the uTorrent service will run under. Note that you must use the same account that you set uTorrent options with if you want the service to respect those settings, so plan accordingly.
   
   
   
   
6. Start the uTorrent service.

(Obviously, replace the above paths with the actual paths that you installed uTorrent to.)

Bam -- you're sharing files with the world using BitTorrent, even when you're not logged in. You can control everything remotely, too, by navigating your browser to the WebUI URL.

Like so many things in Windows, it ain't pretty, but it gets the job done. It's ironic that BitTorrent, which is justly famous for equalizing the highly asymmetric nature of most people's internet connections, is itself so asymmetric when it comes to sharing: trivially easy to consume, but awkward and confusing to share. That's too bad, because BitTorrent is such a powerful tool for sharing. Hopefully this post demystifies the process a bit!

Emrah Diril recently asked me this via email:

Steve Yegge mentioned in the comments of his last post that he gets quite a bit of hate directed his way.

Fake51: you underestimate the ability of people to get mad. Some people start mad and just take it out on you. The hating has gradually become a little too much for me.

I read the guy's blog too, but don't understand where this is coming from. Some people just have this tendency I suppose.

Do you have a similar experience? I don't see you wanting to quit blogging, so how do you deal with this? Is it just a matter of personality? Are you better able to ignore this stuff?

I answered with one of my favorite quotes from Randy Pausch's Last Lecture:

And when it was all over, one of the other assistant coaches came over and said, yeah, Coach Graham rode you pretty hard, didn't he? I said, yeah. He said, that's a good thing. He said, when you're screwing up and nobody's saying anything to you anymore, that means they gave up. And that's a lesson that stuck with me my whole life: when you see yourself doing something badly and nobody's bothering to tell you anymore, that's a very bad place to be. Your critics are your ones telling you they still love you and care.

Welcoming and appreciating reasonable criticism is the right attitude to have, but it's not the full story. Do I love criticism? Do I seek it out? No. I have many personality deficiencies, but masochism isn't one of them. I don't have fantasies of waking up every day to an R. Lee Ermey browbeating from commenters. Or, maybe I do. I should blog about that.

Criticism, painful though it may be, is still a conversation. It means your readers and listeners are engaging with you, and there's something to learn from following that conversation. Those messages you're broadcasting out into the world are being received, in some form, by someone on the planet. Even if that person is, well .. this guy:

I stopped reading the blog a while a go. Joel explains my reasoning nicely in his latest post.

The mystery of the non-reading Coding Horror reader. Another NP-complete problem, I guess.

If you think something sucks to the extent that it's actively harming the world and you want it to go away, leaving comments to that effect is not the way. I know, because I bear the psychic scars of a million online flamewars, dating all the way back to 300 baud dualup modems and BBSes. I've been doing this a very long time. I've seen what works, and what doesn't.

I'm here to tell you that there is something much more powerful than criticism that you can bring to bear in these situations. Something almost unimaginably powerful in its ability to shape human behavior.

The "just don't look" strategy [is] effective in any situation where someone or something runs on attention. On the web attention comes in the form of links and pageviews so "just don't look" translates roughly into "just don't link or read". If you don't like who's on the cover of Wired, just don't look. If no one talks about her, she'll go away. Think media gossip sites are ruining the web? Don't read them. Leggy blonde conservative got your knickers in a knot? Just don't look. Commenters ruining the internet? Moderate your comments or close them up. If some Web 2.0 blowhard says something stupid, just don't look. Hate blonde socialites? Just. Don't. Look.

I am absolutely sick to death of hearing about Susan Boyle, both in the traditional media and online. Nothing personal, you understand, I'm sure she's a perfectly lovely person. But I don't talk about Susan Boyle, because talking about her gives Susan Boyle power and currency. I just ignore Susan Boyle. I wish I had two brains so I could ignore her twice as hard. I. Just. Don't. Look. And if we could convince enough people to ignore her, she .. disappears. Poof. Like magic.

One of my favorite books as a child was the Great Brain series, the story of a family in rural Utah, set in the late 1800s.

In these books, the parents doled out a strange punishment to their children when they seriously misbehaved. For a period of a week, or longer – depending on the severity of the misbehavior – nobody in the family would talk to, acknowledge, or address in any way, that particular boy. It was called "The Silent Treatment".

Tom and I were the most unfortunate kids in town.

When any other kids did something wrong, all they got was a whipping. When Tom and I did something wrong, we got the silent treatment, which was ten times worse than a whipping. It meant that Papa and Mamma wouldn't speak to us, and if we spoke to them, they would pretend they didn't hear us. It was as if Tom and I didn't exist as far as they were concerned during the silent treatment. How I wished they would act like normal parents and give us a whipping, and get it all over with in a hurry.

This didn't seem like much of a punishment to me. In fact, as an introverted kid who loved solitary activities like computers and reading more than anything, it seemed kind of like a .. reward. I couldn't reconcile this feeling with the semi-biographical reality depicted in the books. To the Fitzgerald boys, the silent treatment was the worst possible punishment, far worse than a physical beating. They would go to incredible lengths to avoid getting the silent treatment. As punishments go, it must have been a doozy, though I couldn't quite wrap my geeky, socially maladjusted young head around exactly why.

The silent treatment was a punishment I didn't fully understand until years later in life. That's how you change the world. Not by arguing with people. Certainly not by screaming at them. You do it by ignoring them.

And if you feel strongly enough about me and what I do here, you can begin by ignoring this.