No Silver Bullet

No Silver Bullet: Essence and Accident in Software Engineering, by Fred Brooks, is a landmark paper first published in 1986. I chose this paper to lead off the Classics in Computer Science series for a couple of reasons. First, it speaks directly to the raison d’etre of this blog: “How do we get better?” Another reason is that this article, although it appeared over 30 years ago, addresses issues in software development with which we continue to grapple.

In No Silver Bullet, Brooks compares software projects with werewolves, creatures that are normal people most of the time but at times transform into monsters that will rip your throat out. The only way to kill this beast is to shoot it with a silver bullet.

“So,” he asks, “where is the silver bullet with which we can slay these monstrous, out of control, hopelessly expensive and late software projects?” Well, I think we know what his answer is. After all, the title of the article is “No Silver Bullet.” He certainly did NOT bury the lead on this news story.

There is no silver bullet.

This is sad news. We don’t believe it. Just like Brooks’s law (“Adding personnel to a late project makes it later.”), we ignore its truth and blithely drive the bus over a cliff. Repeatedly.

There is not only no silver bullet, claims Brooks — there is nothing on the horizon that promises even an order of magnitude increase in programmer productivity. And there is certainly nothing that we can foresee that will improve our ability to deliver software on time and under budget, systems that are reliable and with few bugs.

Why so serious? Why the gloomy outlook? Is Brooks just a pessimist, an Eeyore under a perpetual dark cloud? Ok, Mister Technological Stud-Muffin, your experience on OS/360 wasn’t that great. Do you have to take it out on us?

Actually, he has some compelling reasons for maintaining his position. In the first part of the paper, he addresses why software is different than just about any other human artifact, examining its essence.

Essence and Accident

He then argues that the breakthroughs that have given us the greatest gains in productivity thus far have addressed, not the essence of software development, but its accidental attributes.

What is this nonsense about essence and accidents? What does an accident have to do with something that is essential? This can be confusing because he’s using the original meaning of these words. To us, something accidental is an event that is unanticipated and unintended, probably due to some mistake or a lapse in concentration.

That’s not what Brooks is talking about. He is using accident in a philosophical sense, harkening all the way back to Aristotle. For Aristotle, an essential quality is one that is inherent in the object, sometimes by definition. A chair has legs and a seat. This is its essence. Without these things, you would probably not consider the object a chair. In other words, an essential quality of an object is one that, if removed, will cause the object to cease to be. If you remove the legs of a chair, then whatever that object is, it is not a chair.

An accidental quality is one that can be removed or altered and the original object remains intact. For example, if you change a red chair to a brown chair, it is still a chair. The color is an “accidental” attribute. A better term these days would be incidental.

Does It Have To Be This Hard?

The author begins by asking, “Does software development have to be hard? Is there no silver bullet?”

His answer is: Yes, it has to be hard and no, there is no silver bullet.

He answers in this manner because of four essential difficulties in developing software systems.

1. Complexity.
2. Brooks argues that software is more complex, for it’s size, than any other human construct, mainly because, above the statement level, no two parts are alike. This leads to an inordinate number of states that the software can assume. And the complexity does not increase linearly with size. Larger systems are not composed of larger versions of the smaller components. They comprise more and more parts with more and more states. The actual growth in complexity is more exponential than linear.
3. Management becomes more complex as the project scales as well. There are more people with more communication paths, causing communication to be as difficult as it is vital. It increases the difficulty of managing all the details, so that inevitably important items are overlooked or given short shrift. “It creates the tremendous learning and understanding burden that makes personnel turnover a disaster.” (p. 3)
4. Conformity
5. “Much of the complexity he (the software engineer) must master is arbitrary complexity, forced without rhyme or reason by the many human institutions and systems to which his interfaces must conform.” (p.3–4) Einstein had faith that the laws of nature were regular and intelligible and could be discovered because they were designed by God and “God does not play dice with the universe.” No such faith, says Brooks, comforts the software developer. Our requirements are given, not by God, but by humans, usually many humans.
6. Changeability
7. “The software entity is constantly subject to pressures for change.” (p. 4) Other entities feel that pressure as well, but physical objects change less often because the costs and effort are well known. No one asks for a two-story house to become a three-story house by lifting up the second floor and inserting a floor in the gap. Because it is perceived that software is easier to change, these sorts of unnatural requests are commonplace in the software enterprise.
8. Invisibility
9. “Software is invisible and unvisualizable.” (p. 4) A building’s floor plan can be readily translated in the mind’s eye to a physical room or building. There is no such representation for a computer program. This is why, in my opinion, the many design diagrams that have been proposed over the years just are not helpful. They can capture one particular part of the software artifact, such as a class structure or and a process flow, but there is no representation that can help you manage the overall complexity of the complete system.

Accidental Improvements

Having outlined these four essential difficulties, Brooks then considers the areas where great improvements in productivity have taken place in the past. He concludes that these breakthroughs addressed accidental difficulties and not essential ones.

High-level languages abstract away the hardware architecture, so the developer doesn’t have to be concerned with registers and other low-level constructs. Though our level of abstraction continues to grow, this does not reduce the complexity of the program. It reduces the difficulty with which we represent the program and its constructs, but the issues of proliferations of states and management complexity remain.

Time-sharing systems, although a thing of the past, allowed programmers at the time to get immediate feedback on their programs. Rather than enter the program, submit a job, and then wait for the mainframe to run the job at some point, we can enter the program in an editor, compile it immediately, and run it. This made the process quicker, but, once again, did not reduce the essential complexity of the software.

Unified programming environments, what we would refer to now as an IDE, were also a great move forward for programmers. Programmers could develop in a single environment and not have to juggle multiple programs (editor, linker, loader, etc.). This also increased speed of development but did nothing to reduce essential complexity. To be fair, modern IDEs are powerful tools and generate a great amount of mundane, boilerplate code that we are very happy to have done for us. However, the inherent complexity of the problem at hand is unphased by generated code. In addition, we’ve added complexity in the form of package managers, bundling tools, and inversion of control containers, all of which require significant intellectual effort to configure and maintain. It’s similar to the adage about using regular expressions to solve a problem: now you have two problems.

Having demonstrated that the greatest gains in productivity to date have been accomplished by solving problems that are not essential to the software enterprise, the curmudgeonly Dr. Brooks examines several technologies that have been advertised as potential silver bullets. He mentions such things as continued advances in high-level languages, object-oriented programming, artificial intelligence, expert systems, “automatic” programming, graphical programming, program verification, improvements in programming environments, and workstations. While they are all useful to some degree and should not be ignored, none of these things, says Brooks, promise even a two-fold increase in productivity. For various reasons that the author gives, none of these items address the essential complexity of software development.

What’s the point, then? Nothing is going to help, so how can we possibly improve the way we do software? It’s a pretty gloomy, almost cynical picture Brooks paints of our craft.

No Silver Bullet? How About Some Garlic?

There is hope, according to the author, if not of a silver bullet, at least we can grab some garlic to fend off the werewolves (in some medieval myths, garlic was used to repel vampires, werewolves, and a host of nasty demons). Brooks gives a broad outline of four areas that promise improvements in our productivity. In each of these, he proves that he not only can identify the difficulties of developing software, he sees, in an eerily prescient fashion, the things that will actually help us write software.

1. Buy versus build. The best software is the software you don’t have to write. It’s cheaper, says he, to buy a solution than build it. Although he could not have known the details at the time, this has largely come true in the move to open-source development.
2. Requirements refinement and rapid prototyping. In this he presaged agile development.
3. Incremental development — grow, not build, software. Growing a system, a little at a time, is a way to deliver working software before it’s complete. Users see what they are getting and adjustments can be made without great rework and re-engineering. This has agile written all over it.
4. Great designers. This is the coupe de grace. In dramatic fashion, Brooks saves the best for last. He sucker-punches us with the obvious and most ignored truth of all. Great systems are built by great programmers. While we focus on methods and tools and automation, what we really should be concerned with is developing developers. Our efforts should be spent building into one another and raising the level of programming expertise of all the developers we know. Most of us are too busy with ourselves and are very happy to focus on new tools and sexy languages. What should occupy our best thoughts and our greatest effort is to become great programmers and make everyone around us great as well.

I hope you read this article. Much of it is dated, but those are the accidental attributes of the paper (See what I’m doing?). The essence is more applicable than ever. And if you take only one thing from it, take this: become a great developer. And help me become great as well.

Who is Fred Brooks?

Brooks is best-known for his classic book, The Mythical Man-Month, in which he offers software development wisdom gained as the head of IBM’s OS/360 operating system project. From this book comes what has become known as Brooks’s law: Adding personnel to a late project makes it later. It is one of those observations that everyone agrees with and no one practices. It’s also never, and I mean never, failed to hold true, at least in my experience. Throwing bodies at a late project may save a manager’s scrawny, sycophantic posterior, but it never helps the project.

Brooks has also garnered a plethora of awards over the years, probably the most prestigious being the 1999 ACM Alan J. Turing award for his “landmark contributions to computer architecture, operating systems, and software engineering.” He is attributed with inventing the IBM mainframe’s interrupt system (he holds the patent), along with coining the term “computer architecture” and establishing the 8-bit byte as the lowest addressable unit in IBM systems. Stud.