The Killer Quest
BY PHILLIP MANNING
“A train wreck in slow motion,” is how Senator Patrick Leahy described an FBI project that wasted $170 million in an unsuccessful attempt to modernize the bureau’s computer software. McDonalds spent the same amount trying to install a real-time network for its outlets before finally admitting failure. The IRS spent even more in a botched attempt to upgrade its software and computer systems. The saddest part about such tales is that they are so common. Failed, late and over-budget software projects are so routine that they aren’t news.
A 1995 survey revealed that technology managers rated 31 percent of the software projects undertaken as “total failures.” Nine years later, a similar survey showed that total failures had been almost halved to 18 percent — however the same survey found that only 29 percent of the projects were rated as successes.
In his revealing book “Dreaming in Code,” Scott Rosenberg, a cofounder of the on-line magazine Salon, asks: Why is writing code for a computer so difficult to do right and on time? To answer the question, he hooks up with a software team headed by Silicon Valley legend Mitch Kapor to observe day-by-day how he and his group develop software and introduce it to the marketplace.
First some background. Software is, at bottom, a string of abstract symbols that tells the computer what to do. One error or bug — one [ where a ( needs to be — in a million lines of code can bring an entire system down. Finding those bugs is like trying to locate the bad bulb on a 50-mile long string of Christmas lights. Consequently, bug fixing is demanding, time-consuming work that can bring a project it to a standstill.
Another common reason for software failures is mushy or changing specifications. To create a usable system, programmers must know from the outset what the final software is supposed to do. Many a project has foundered because customers demanded changes after work was underway or worse yet nearing completion. Choosing Kapor’s project enabled Rosenberg to find yet another reason why software projects fall behind schedule and sometimes fail: Without the harsh discipline of the marketplace, programmers tend to spend too much time wallowing in software sink holes.
Mitch Kapor created one of the first “killer apps” in the 1980s. His spreadsheet program Lotus 1-2-3 was so desirable that customers bought PCs just to get the software. Kapor walked away from the company he founded in 1986 with $100 million in his jeans. He was, he said, “extricating myself from my own success.” Before he left, he began work on a software package called Agenda. Agenda was a PIM, a personal information manager, aimed at keeping busy people organized. Calendars and address books can handle some items, but Kapor wanted a system that would organize all of the barrage of details daily life throws at us.
Agenda was a modest success for Lotus, but not the world-changing software that Kapor hoped. So, after trying other careers from MIT professor to venture capitalist, Kapor returned to Agenda. He wanted to completely rewrite the program, make it more elegant, more useful. He named his software-to-be Chandler, not after the character on “Friends” but for the mystery writer Raymond Chandler.
Kapor assembled an outstanding team of software veterans: computer scientists from Stanford, software designers and programmers from Microsoft, Netscape and Apple. However, two peculiarities would slow down Kapor’s dream team. First, unlike most projects, which begin with a list of desired features, Chandler started as a vision of how the innards of a sophisticated PIM should work. But no one specified exactly what Chandler was supposed to do. What reports were to be issued? What screens would be needed? Second, the company that Kapor founded to develop Chandler was a nonprofit funded almost entirely by Kapor himself. Without investors or customers breathing down their necks, the team never had any hard-and-fast deadlines. What were the penalties for being late and what were the rewards for succeeding?
Unlike Chandler, most commercial software projects are profit motivated and limited by money or time or both. Furthermore, they usually start not as a software vision but as set of concrete features or specifications that customers want. Fear and greed and pride usually drive programmers in such undertakings, especially one-shot, entrepreneurial projects such as Chandler. Fear for your job or your company if the project comes in late or flops completely; greed for a big payoff if the project succeeds; pride in completing a tough job on time in the face of unyielding deadlines. Without those drivers, the Chandler team was free to engage in an outrageous amount of “axe sharpening.”
Give a person six hours to cut down a tree, the saying goes, and she will spend the first four sharpening the axe. In other words, most of us would rather spend time improving the tools that make a job easier than getting on with the job itself. One of many such axe sharpenings at Chandler came when the team, already a year behind schedule, decided to wrap its code in packages that would make it easy for nonprogrammers to add features sometime in the future. Hard-nosed software managers simply will not put up with this kind of axe sharpening. Let’s get something out the door, they intone sternly, before we worry too much about what someone might want in the distant future.
In structuring Chandler as a programmer’s paradise without the discipline of do-or-die deadlines, Kapor eliminated the all-nighters, the time’s-running-out frenzy that pushes a team to finish a job. And because Chandler is funded by a nonprofit organization, no big payoffs were coming. The consequences of Kapor’s good intentions were unfortunate but predictable.
The project began in the spring of 2001. A year later, little code had been written. “Optimistically,” Kapor wrote in his blog, we could have a fully functional version of Chandler available “by the end of 2003. Pessimistically, it will be 2004.”
Kapor’s projections have proved to be badly off the mark. After almost three years of living with a still-unfinished project that was supposed to be done in a year, Rosenberg realized that he could not stick around to see the project through as he had intended. He gave up and went home to write his book. However, he does provide the address of Chandler’s web site so readers can see how the project has progressed since the book ended. When I checked the site on January 1, 2007, the fully functional 1.0 release of Chandler was still not available.