Coding Horror

Are there any programming jobs you wouldn't take? Not because the jobs didn't pay enough, had poor benefits, or limited upside -- but because the work itself made you uncomfortable? Consider the tale of one freshmeat.net writer:

Back in the old days (let's say 1996), I was just another Perl coder writing CGI scripts for a living. Well, pocket money's more like it, but okay. I wrote scripts for fun, I wrote them to make some cash, and I wrote them because I'm a geek and I love programming. Then, one day, I got a phone call from this company. A friend of mine had referred them to me, and they wanted me to write a CGI script. The gentleman I spoke with was very well mannered, very well educated -- the typical likeable manager.

After some talking, he came to the point. The CGI script I was to create was supposed to take an archive of images and make them searchable by topic. In itself nothing amazing, but when I asked, out of curiosity, what kind of images we were talking about, I was surprised to find out it was porn. Yes, porn.

I accepted the job, and life changed dramatically. Instead of friends saying "cool" or some coders I knew saying "nice script", they shied away, refused to talk to me, refused to look at the script. For a long time, I wondered why. This year, I went to a convention. I was just out there looking for new cool stuff, not much else. Everyone I talked to was friendly, and downright nice, right up until the point when I told them what I did for a living. Then they suddenly remembered they had something better to do.

And why? Does working on the adult part of the net mean I'm a scumbag? Does it mean I'm sleazy? Does it mean I'm untrustworthy? Does it mean my code is bad?

That was eight years ago. I wonder how the now over-thirty author of the original article is getting on in his career. Does he still write code for the adult industry? Somehow, I doubt it.

This isn't just random noodling on my part. I've almost been in the same situation. About ten years ago, I had an interview for a programming position with a prominent North Carolina based purveyor of adult products. After the interview, I asked my girlfriend (now my wife) how she would feel if I took a job that was, more or less, in the adult industry. Although she's flexible on almost every topic, this is one area where she had serious reservations. I think the operative words were "what will we tell our parents?" It's a fair question. For that matter, what do you tell your friends when they ask where you work? Your peers? It was enough to keep me from taking the job.

Years later, I encountered one of my previous coworkers who had taken a programming job there. It turns out I made the right choice, but not for the reasons you might think. There were technical and managerial problems on the job that far outstripped any effects from the unusual choice of industry. That said, when I asked him what the environment was like, working daily with adult products, he had a one word response: weird.

The adult industry does present interesting technical and scaling challenges, perhaps more interesting than building yet another line of business CRUD application for Yet Another MegaCorp. A recent Reddit discussion thread which asked if you designed a porn site, would you put it on your resume? had some excellent examples.

[Adult] sites have oodles of top-quality attributes to them; payment processing, secured content, username and password maintenance (especially self-service maintenance) rapid updates and, if your site was successful, some interesting scaling problems to engineer around.

I work for an [adult] site. It's going on my resume. Anybody I've told in a professional or semi-professional setting has been impressed and wanted to know the technical details about our server setup and bandwidth. I have yet to meet anybody, friend or prospective employer who was turned off by the thought of my serving up [adult content]. And I'm willing to say that I wouldn't work for someone who would judge me negatively because of it. I got into it because of the interesting scaling problems and potential for wisdom-of-crowds filtering and selection. And you know what, it's kinda fun.

I've worked in the [adult] industry for over 8 years. If I were to apply for a new job, you bet I would include it on my resume. I don't think I'd want to work for a company that couldn't see the benefit of having someone onboard that's worked with systems that must always work, have a known cost-per-minute of downtime, and are on a permanently continuous release cycle.

The general tone of the advice is that, if you choose to work in the adult industry, you have to tell white lies about your work -- small evasions about what your work is, and who it is for, depending on the audience. Invoke vague NDAs. Describe things in broad, general terms.

What if you saw this programming job listing:

• Work from home on a large, high-traffic website with lots of challenging scaling problems.
• Use the latest frameworks and technologies.
• Set your own hours.
• Excellent pay through wire transfer from an offshore account, with full benefits.

(I am not making any of this up, I'm actually summarizing a real job listing.) Seems like a fantastic programming job, right? But what if I told you this job listing was for an adult website? Would you still consider it?

I bring this up because I recently read a great in the trenches story about continuous deployment.

Our tests suite takes nine minutes to run (distributed across 30-40 machines). Our code pushes take another six minutes. Since these two steps are pipelined that means at peak we're pushing a new revision of the code to the website every nine minutes. That's 6 deploys an hour. Even at that pace we're often batching multiple commits into a single test/push cycle. On average we deploy new code fifty times a day.

My enthusiasm for this supreme feat of software engineering was tempered by the fact that, when I clicked through to find out more about the company that was doing such sophisticated software engineering, I learned that it's a 3D chat avatar system. A very.. sexy.. 3D chat avatar system. Just look at their ads to see what I mean:

What is being sold here? I've even seen similar "sexy" IMVU ads with female 3D avatars in skimpy lingerie come up organically on my Fake Plastic Rock blog, of all places. I'm not the first person to make this connection, either.

A reader expressed their irritation with the IMVU ads that have been running in the sidebar recently. I was actually glad to see I wasn't the only one. They have a trashy, lowest-common-denominator feel to them. Kind of a "Welcome to Hoochie World" vibe. The ad has been running for over a month, and I've never seen a picture of a single male avatar. It's either the quasi-jailbait in a bikini, or a couple of skanks in a pseudosapphic embrace. Using a pretty girl to sell your stuff is perfectly reasonable, but doing it with such a lack of class gets on my nerves. I've never used the software, but the ads make me think their chat software is a world inhabited by l337-speaking teenage boys that would make the average FARK thread sound like the Mclaughlin Group by comparison.

The profile for IMVU user "hottiepie4life" makes it abundantly clear that IMVU, while not quite part of the adult industry per se, skirts awfully close to the edge of it. Enough to make me, personally, uncomfortable about working there, or talking to anyone who worked there. And it certainly colors and devalues my impression of the technical work going on there.

Maybe this is my problem. Does the subject matter dilute the excellent technical work the IMVU team might be doing? No. But at the same time, I can't help questioning the ultimate value of that work. I'm no prude. And I don't expect every programmer to be doing noble, selfless work for the good of humanity. All the same, it's difficult for me to respect software engineering in the service of such least common denominator interests.

I'll always remember WordPerfect as the quintessential white text on blue screen application.

For a period from about 1985 to 1992, WordPerfect was the most popular word processing program in the world on virtually every computing platform. I remember it well; the very concept of word processing was synonymous with WordPerfect.

And now I can't even recall the last time I encountered a WordPerfect document, much less anyone who still uses WordPerfect. The software is still limping along, barely, under the auspices of Corel corporation, as WordPerfect Office X4. I guess it's a testament to how quickly things change in the world of software; you can dominate the world for years, only to be relegated to little more than a dimly remembered footnote in computing history a decade later.

Perhaps that's why the online book Almost Perfect, which documents the rise and fall of WordPerfect, is such a gripping read. I clicked through, read the first chapter, read the second chapter, and ... I couldn't stop reading it!

Some of the story is predictable. WordPerfect, like many other companies at the time, never really made the transition from DOS to Windows. They didn't just bet on the wrong horse, they institutionalized a software culture that lived and died on character mode assumptions. That, plus an almost fanatical dedication to cross-platform parity -- even when the platforms they supported made little business sense -- makes the final outcome almost inevitable.

Still, there's something intriguing about the fledgling SSI corporation. For one thing, the entire business was run in an agile, almost by-the-seat-of-their-pants way. It's a bit of a David and Goliath story, how they clawed their way to success over so many formidable opponents by simply dedicating themselves to creating an excellent product, and cultivating a vibrant community around that product both inside and outside the company. But perhaps the best part is that the company was run by programmers:

We spent a lot of time in meetings going over what had to be in the product and how things should work. These decisions were made by the programmers, who sometimes had very heated discussions about what was needed. At times the three of us on the Board had to assume the role of referee. Some of the issues were very complicated, so by the time the arguments were finished and a decision was made, I usually had a headache.

It was somewhat unusual for a software company to let the programmers decide the future of its products. We were, however, a company founded and owned by programmers, where programmers were treated with an extra measure of respect. The marketing department was used primarily to sell products once they were developed, and only rarely did it get involved early enough to perform the traditional marketing role of identifying a need and defining a product to fill that need. At times this put us in the position of developing solutions before we identified problems, but it was hard to be too critical of the programmers when the company was so successful. To their credit, the programmers tried very hard to listen to our customers and to those of us in the marketing department. The programmers were smart and thoughtful and very good at protecting the best interests of the company. At times, however, they were prone to manipulate some of the data they received to fit what it was they wanted to do.

The story sort of fizzles out toward the end, as the author, W. E. Pete Peterson, is unceremoniously kicked out of the company just as WordPerfect Corporation begins to lose its hold on the market. Perhaps this is fitting: like WordPerfect itself, he doesn't go out with a bang, but a whimper.

At any rate, Almost Perfect is fantastic reading. Long out of print, it finds its own audience when self-published on the web. It's reminiscent of the very best early computer industry tales from one of my favorite books, Accidental Empires.

I was paging through the book again after I was reminded of it, and I found this passage:

Of course, companies don't have to grow. Electric Pencil, the first word processing program for the Apple II, was the archetype for all word processing packages that followed, but its developer, a former Hollywood screenwriter, just got tired of all the support hassles and finally shut his company down. In 1978, Electric Pencil had 250,000 users. By 1981, it was forgotten.

The book was also made into a documentary, Triumph of the Nerds. I recommend both highly.

I'm not sure if all the lessons from Almost Perfect are relevant today -- but some failure patterns are timeless, and I certainly admired the way SSI bootstrapped itself while letting the employees (and more specifically, the programmers) run the company.

I've been following Michael Abrash for more than 10 years now; he's one of my programming heroes. So I was fascinated to discover that Mr. Abrash wrote an article extolling the virtures of Intel's upcoming Larrabee. What's Larrabee? It's a weird little unreleased beast that sits somewhere in the vague no man's land between CPU and GPU:

[Larrabee] is first and foremost NOT a GPU. It's a CPU. A many-core CPU that is optimized for data-parallel processing. What's the difference? Well, there is very little fixed function hardware, and the hardware is targeted to run general purpose code as easily as possible. The bottom lines is that Intel can make this very wide many-core CPU look like a GPU by implementing software libraries to handle DirectX and OpenGL.

We know that GPUs generally deliver one or two more orders of magnitude more performance than a general purpose CPUs at the things they are good at. That's what I would expect for dedicated hardware devoted to a specific and highly parallizable task.

Michael Abrash has already attempted what most people said was impossible -- to build a full software 3D renderer that runs modern games at reasonable framerates. In other words, to make a general purpose CPU compete in a completely unfair fight against a highly specialized GPU. He's effectively accomplished that, and his company sells it as a product called Pixomatic:

In this three-part article, I discuss the process of optimizing Pixomatic, an x86 3D software rasterizer for Windows and Linux written by Mike Sartain and myself. Pixomatic was perhaps the greatest performance challenge I've ever encountered, certainly right up there with Quake. When we started on Pixomatic, we weren't even sure we'd be able to get DirectX 6 features and performance, the minimum for a viable rasterizer. I'm pleased to report that we succeeded. On a 3 GHz Pentium 4, Pixomatic can run Unreal Tournament 2004 at 640â—Š480, with bilinear filtering enabled. On slower processors, performance is of course lower, but by rendering at 320â—Š240 and stretching up to 640â—Š480, Unreal Tournament 2004 runs adequately well -- even on a 733-MHz Pentium III.

Pixomatic is documented in an excellent series of Dr. Dobbs articles. It's fascinating reading; even though I know zero about assembly language, Michael's language of choice, he's a fantastic writer. That old adage about the subject not mattering when you have a great teacher has never been truer.

I remember trying out Pixomatic briefly four years ago. Those CPUs he's talking about seem awfully quaint now, and that made me curious: how fast is the Pixomatic software renderer on today's CPUs? My current box is a Core 2 Duo (wolfdale) running at 3.8 GHz. So I downloaded the Unreal Tournament 2004 demo (still fun, by the way!), and followed the brief, easy instructions provided to enable the Pixomatic software renderer. It's not complicated:

ut2004.exe -software

One word of warning. Be sure you have an appropriate resolution set before doing this! I was playing at 1920x1200 initially, and that's what the software renderer defaulted to. And here's the shocker: it was actually playable! I couldn't believe it. It wasn't great, mind you, but it was hardly a slideshow. I tweaked the resolution down to something I felt was realistic: 1024x768. I turned on framerate display by pressing ...

~
stat fps

... from within the game. This Pixomatic software rendered version of the game delivered a solid 40-60 fps experience in capture the flag mode. It ran so well, in fact, that I decided to bump up the detail -- I enabled 32-bit color and bilinear filtering by editing the ut2004.ini file:

[PixoDrv.PixoRenderDevice]
FogEnabled=True
Zoom2X=False
SimpleMaterials=True
LimitTextureSize=True
LowQualityTerrain=False
TerrainLOD=10
SkyboxHack=True
FilterQuality3D=3
FilterQualityHUD=1
HighDetailActors=False
SuperHighDetailActors=False
ReduceMouseLag=True
DesiredRefreshRate=0
DetailTexMipBias=0.000000
Use16bitTextures=False
Use16bit=False
UseStencil=False
UseCompressedLightmaps=False
DetailTextures=False
UsePrecaching=True

Once I did this, the game looked totally respectable. Eerily reminiscent in visuals and performance to the classic, early Voodoo and Voodoo 2 cards, actually.

(If you think this looks bad, check out Doom 3 running on an ancient Voodoo 2 setup. It's certainly better than that!)

The frame rate took a big hit, dropping to 30fps, but I found it was an uncannily stable 30fps. The only achilles heel of the Pixomatic software renderer is places with lots of alpha blending, such as when you fire a sniper rifle, obscuring the entire screen with a puff of muzzle smoke, or if you're standing near a teleportation portal.

Pretty amazing, right? It is!

And utterly pointless.

My current video card renders Unreal Tournament 2004 at the highest possible resolution with every possible quality option set to maximum, at somewhere between 200 and 300 frames per second. Despite the miraculously efficient assembly Abrash and Sartain created to make this possible at all, it's at best a carnival oddity; even the crappiest onboard laptop 3D (assuming a laptop of recent vintage) could outperform Pixomatic without even breaking a sweat.

We know that the game is far more enjoyable to play with a real GPU, on a real video card. And we're hip deep in real GPUs on every platform; even the iPhone has one. Perhaps Pixomatic made some business sense back in 2003, but it didn't take a genius analyst back then to see that it would make no business sense at all today. At the same time, I can't help admiring the engineering effort that went into building a viable 3D software renderer, something that seemed virtually impossible bordering on foolish.

In short, it will be possible to get major speedups from [Larrabee] without heroic programming, and that surely is A Good Thing. Of course, nothing's ever that easy; as with any new technology, only time will tell exactly how well automatic vectorization will work, and at the least it will take time for the tools to come fully up to speed. Regardless, it will equally surely be possible to get even greater speedups by getting your hands dirty with intrinsics and assembly language; besides, I happen to like heroic coding.

Ditto.

We'll have to wait and see if Intel's efforts to push GPU functionality into their x86 architecture makes any of this heroic coding more relevant in the future. Either way, it remains impressive.

Have you ever gotten that classic job interview question, "where do you see yourself in five years?" When asked, I'm always mentally transported back to a certain Twisted Sister video from 1984.

I want you to tell me – no, better yet, stand up and tell the class –

what do you wanna do with your life?

You want to rock, naturally! Or at least be a rockstar programmer. It's not a question that typically gets a serious answer – sort of like that other old groan-inducing interview chestnut, "what's your greatest weakness?" It's that you sometimes rock too hard, right? Innocent bystanders could get hurt.

But I think this is a different and more serious class of question, one that deserves real consideration. Not for the interviewer's benefit, but for your own benefit.

The "where do you see yourself in five years" question is sort of glib, and most people have a pat answer they give to interviewers. But it does raise some deeper concerns: what is the potential career path for a software developer? Sure, we do this stuff because we love it, and we're very fortunate in that regard. But will you be sitting in front of your computer programming when you're 50? When you're 60? What is the best possible career outcome for a programmer who aspires to be.. well, a programmer?

What if I told you, with tongue firmly planted in cheek, that there were Eight Levels of Programmers?

1. Dead Programmer

   This is the highest level. Your code has survived and transcended your death. You are a part of the permanent historical record of computing. Other programmers study your work and writing. You may have won a Turing Award, or written influential papers, or invented one or more pieces of fundamental technology that have affected the course of programming as we know it. You don't just have a wikipedia entry – there are entire websites dedicated to studying your life and work.

   Very few programmers ever achieve this level in their own lifetimes.

   Examples: Dijkstra, Knuth, Kay

2. Successful Programmer

   Programmers who are both well known and have created entire businesses – perhaps even whole industries – around their code. These programmers have given themselves the real freedom zero: the freedom to decide for themselves what they want to work on. And to share that freedom with their fellow programmers.

   This is the level to which most programmers should aspire. Getting to this level often depends more on business skills than programming.

   Examples: Gates, Carmack, DHH

3. Famous Programmer

   This is also a good place to be, but not unless you also have a day job.

   You're famous in programming circles. But being famous doesn't necessarily mean you can turn a profit and support yourself. Famous is good, but successful is better. You probably work for a large, well known technology company, an influential small company, or you're a part of a modest startup team. Either way, other programmers have heard of you, and you're having a positive impact on the field.

4. Working Programmer

   You have a successful career as a software developer. Your skills are always in demand and you never have to look very long or hard to find a great job. Your peers respect you. Every company you work with is improved and enriched in some way by your presence.

   But where do you go from there?

5. Average Programmer

   At this level you are a good enough programmer to realize that you're not a great programmer. And you might never be.

   Talent often has little do do with success. You can be very successful if you have business and people skills. If you are an average programmer but manage to make a living at it then you are talented, just not necessarily at coding.

   Don't knock the value of self-awareness. It's more rare than you realize. There's nothing wrong with lacking talent. Be bold. Figure out what you're good at, and pursue it. Aggressively.

6. Amateur Programmer

   An amateur programmer loves to code, and it shows: they might be a promising student or intern, or perhaps they're contributing to open source projects, or building interesting "just for fun" applications or websites in their spare time. Their code and ideas show promise and enthusiasm.

   Being an amateur is a good thing; from this level one can rapidly rise to become a working programmer.

7. Unknown Programmer

   The proverbial typical programmer. Joe Coder. Competent (usually) but unremarkable. Probably works for a large, anonymous MegaCorp. It's just a job, not their entire life. Nothing wrong with that, either.

8. Bad Programmer

   People who somehow fell into the programmer role without an iota of skill or ability. Everything they touch turns into pain and suffering for their fellow programmers – with the possible exception of other Bad Programmers, who lack even the rudimentary skill required to tell that they're working with another Bad Programmer.

   Which is, perhaps, the hallmark of all Bad Programmers. These people have no business writing code of any kind – but they do, anyway.

These levels aren't entirely serious. Not every programmer aspires to the same things in their career. But it's illuminating to consider what a programmer could accomplish in ten years, twenty years, or thirty years – perhaps even a lifetime. Which notable programmers do you admire the most? What did they accomplish to earn your admiration?

In short, what do you wanna do with your life?

One of the more famous Edsger Dijkstra quotes is from his 1972 Turing award lecture, How do we tell truths that might hurt?

Besides a mathematical inclination, an exceptionally good mastery of one's native tongue is the most vital asset of a competent programmer.

Note that he specifically says native tongue, not English. Which makes me wonder why all of Dijkstra's most important writings were in English, not his native Dutch.

But I digress. Let's consider the first part of the Dijkstra quote. Should competent programmers be "mathematically inclined"? It might be instructive to think of programming as a form of mathematics, at least for one reason: to resist localization. Although there have been attempts to create localized programming languages, as far as I know, nobody has ever tried to localize π or the number 3. They're universal. So in that sense, programming languages bear a vague, passing resemblance to mathematics. Learn the symbols once and use them everywhere in the world, no matter what your native tongue is.

On the other hand, I have not found in practice that programmers need to be mathematically inclined to become great software developers. Quite the opposite, in fact. This does depend heavily on what kind of code you're writing, but the vast bulk of code that I've seen consists mostly of the "balancing your checkbook" sort of math, nothing remotely like what you'd find in the average college calculus textbook, even.

{
i = j++ / (x + v);
}

Not exactly the stuff mathletes are made of.

I never understood the desire to formally equate skill at mathematics with skill at programming. While being a math wonk certainly won't hurt you as a programmer, it's very hard for me to draw a direct line from "good at math" to "good at programming". Like Rory, I believe that software development requires some markedly right-brained sensibilities.

When I was growing up, I remember hearing people say things like, "If you like computer programming, then you'll love math." I always thought that these people were absolutely nuts. While there is something intrinsically similar about certain types of math and computer programming, the two are different in many more ways than they are similar.

With math, and I'm not talking about the crazy number-theory math philosophy "Do numbers really exist?" side of things, but with the applied stuff, there are correct answers. You're either correct or you're incorrect.

With coding, the best you can hope for is to do something well. With so many different ways to effect a single outcome, it's up to some very right-brained sensibilities to determine if you've met your goal, as there isn't anybody (except [another more experienced developer]) who can tell you if you're right or not.

If you ignore your right brain, and I'm talking generally about abstraction and aesthetics, then you can slap some code together that might work, but it also might be one hell of a maintenance nightmare. If you focus only on the right brain, you might have something that works, but is so utterly inefficient and personalized that you're the only person on Earth who could make sense of the code and maintain it.

All those caveats aside, people still advocate the idea that math alone has the power to make you a better programmer. Steve Yegge makes the best case I've read for the programmer-mathematician, with his five points:

1. Math is a lot easier to pick up after you know how to program. In fact, if you're a halfway decent programmer, you'll find it's almost a snap.


2. They teach math all wrong in school. Way, WAY wrong. If you teach yourself math the right way, you'll learn faster, remember it longer, and it'll be much more valuable to you as a programmer.


3. Knowing even a little of the right kinds of math can enable you do write some pretty interesting programs that would otherwise be too hard. In other words, math is something you can pick up a little at a time, whenever you have free time.


4. Nobody knows all of math, not even the best mathematicians. The field is constantly expanding, as people invent new formalisms to solve their own problems. And with any given math problem, just like in programming, there's more than one way to do it. You can pick the one you like best.


5. Math is... ummm, please don't tell anyone I said this; I'll never get invited to another party as long as I live. But math, well... I'd better whisper this, so listen up: (it's actually kinda fun.)

To me, this reads like a broad recipe for becoming intellectually curious and building skill at solving abstract problems. Important programming skills, to be sure, but not necessarily exclusive to the study of math. If math is your preferred way to sharpen your saw, then have at it -- but it's hardly the only way.

I recently received this email:

I run a small (4 people) web dev shop and I'm finding that younger coders haven't had the pleasure of writing assembler or managing without library functions. I've always found strong math skills to be one of the most useful skills for coding, and when one has Google and a massive library of functions, one doesn't have to be good at math to get things working, until it either breaks, has edge cases, or brings out OS or library bugs.

Some quick examples: simplifying tricky equations to determine array indicies or memory offsets; trigonometry to help with physical calculations; mental hex/bin/dec conversion; logic equalities such as DeMorgan's theorem.

He's got the right idea; if we're going to talk about math, let's get out of the abstract and into the specific. Let's talk details. Examples. What could be more math-y than that?

What code have you personally written where a detailed knowledge of math made your work easier? I can think of some broad categories. Writing a 3D game. Or a physics simulation. Or low-level image filters. Or compression algorithms. The list goes on and on. But if you're in those situations, you'll know it.

Maybe I'm a hopeless optimist, but I think most programmers are smart enough to learn whatever math they need just in time to attack the problem at hand.