In an intriguing post entitled Fact and Folklore in Software Engineering, Laurent Bossavit addressed one of the most persistent memes in software development:
One debate (if it can be called that) which has gone on for too long without a satisfactory resolution concerns programmer productivity and the often quoted “observation” or “fact” that the best programmers are 10 times better than the worst. I will come back to the origins of this observation, but it isn’t quite the topic of this article.
As Bossavit rightly notes, this “fact” about programmer productivity has been circulating for decades, acquiring authority via constant repetition. It is widely cited as evidence for the special genius of the so-called “super-programmer,” whose skills (innate, not acquired) are vastly superior to the merely average programmer. In my chapters on “The ‘Black Art’ of Programming” and “Chess players, Musicians, and Mathematicians” I devote a fair bit of space to the ways in which the idea that good programmers were “born, not made” played out in the emerging labor market for computer programmers. The goal was not, of course, to make any empirical claims about programmer performance, but rather to provide a historical context for a debate that has now spanned several decades.
What Bossavit does with this meme is interesting. Borrowing from the work of Bruno Latour (one of the patron saints of my own academic discipline), he describes the ways in which data from a single, very limited study gradually become a well-established scientific “fact.” The original study was undertaken at the Systems Development Corporation (SDC) by the personnel researchers Hal Sackman, W.J. Erickson, and E.E. Grant. It was published in 19681Sackman, Hal, W.J. Erickson, and E.E. Grant. “Exploratory Experimental Studies Comparing Online and Offline Programming Performance”. In: Communications of the ACM 11.1 (1968), pp. 3–11.
SDC was the Rand Corporation spin-off charged with developing the systems software for the SAGE air defense project. Over the course of the late 1950s and early 1960s, SDC trained (or recruited) thousands of computer programmers. As one point, SDC alone employed more than three-fifths of the programmers in the United States. They had, therefore, a huge stake in understanding programming expertise. “We trained the industry,” SDC executives were later fond of saying, and in many respects they were correct; for the next decade, at the very least, any programming department of any size was likely to contain at least two or three SDC alumni.
The 1968 study examined the performance of 12 (!?!) programmers split between two groups. Each groups worked in a different environment (one “online”, or interactive, another in a simulated “offline” environment) on a debugging task. It was not really a study of programmer performance per se, but among their other observations Sackman et al. noted “striking differences” in the performance of different programmers. The fastest programmers completed the task 28 times faster than the slowest programmers. The performance of the most efficient program was, on average, 10 times better than that of the least efficient. From this the authors concluded (borrowing from a familiar nursery rhyme) that “When a programmer is good, he is very, very good. But when he is bad, he is horrid.” Despite the obvious limits of the 1968 study, the 10x figure quickly became part of the lore of of the industry.
Follow- 1Sackman, Hal, W.J. Erickson, and E.E. Grant. “Exploratory Experimental Studies Comparing Online and Offline Programming Performance”. In: Communications of the ACM 11.1 (1968), pp. 3–11.
