The Reality of Failure

How can you tell experienced programmers from beginners? New programmers think if they work hard, they might succeed. Experienced programmers know that if they work really hard, they might not fail. Allow me to elaborate with an excerpt from an interview with Steve McConnell:

SM: One of the points I make in Rapid Development is that all kinds of different things that can go wrong. I enumerate three dozen classic mistakes in Rapid Development. There are lots more mistakes than that that can come up on a project. Actually succeeding in a software project depends a whole lot less on not doing a few things wrong but on doing almost everything right.

TNC: Now, I've been developing software for about ten years and I don't think I exaggerate when I say that 90 percent of the projects I've worked on were not successful. And when I say not successful, the problem was not that they didn't complete on time or they didn't include the features that were originally intended. But they didn't complete at all. Is my experience representative?

SM: I think your experience might be worse than average. 90 percent not completed at all is higher than any number I've seen reported for industry data. But a number like 25 percent not completed at all would be much more in line. And that's still pretty awful. If you look at the remaining 75 percent of projects that are completed, you see that a very tiny percentage are completed on time, on budget and with the originally expected feature set. If you look at the various surveys that have been done, and I'm using some very round numbers, a quarter of the projects are canceled before they're completed and another 50 percent are completed over budget or behind schedule. And 25 percent are completed on time and on schedule, but the piece that's always missing from those surveys is how much of the original functionality was delivered in that 25 percent that was on time and on budget.

The depressing reality of software development is that your project has to be almost supernaturally well run to avoid failure, much less succeed. And it still happens all the time; this article is dated last week:

The FBI's Virtual Case File project -- on which the bureau has spent almost $170 million since June 2001 -- won't succeed. That's the conclusion the Justice Department's inspector general has reached. The auditors' gloomy forecast, contained in a draft report obtained by GCN, echoes criticism from systems specialists within and outside the bureau who cited the project's technical and management flaws.

The bureau likely will not be able to recoup the money it has given contractor Science Applications International Corp. of San Diego for VCF work, sources said.

Via today's yahoo news:

A $170 million computer overhaul intended to give FBI agents and analysts an instantaneous and paperless way to manage criminal and terrorism cases is headed back to the drawing board, probably at a much steeper cost to taxpayers.

The FBI is hoping to salvage some parts of the project, known as Virtual Case File. But officials acknowledged Thursday that it is possible the entire system, designed by Science Applications International Corp. of San Diego, is so inadequate and outdated that one will have to be built from scratch.

In my experience, only developers who have been through a number of failure cycles can appreciate the pain they're about to inflict on users. They work much harder to understand the pitfalls and avoid the classic mistakes. I'd seriously question the credentials-- or at least the intellectual honesty-- of any developer who denied being a part of any software disasters.