The Folly of Code Optimization

If you find yourself in a technical interview it is a near certainty that you will be asked to write some hyper-optimized code. The conditions of this demand will be grotesquely unsuited for thoughtful concentration; your interviewer will have no training in the art, he will sniff and fidget and offer misleading suggestions. The medium for coding will not be a coding editor, it will be a whiteboard unsuited to the task and using a marker pen whose chemical odor gives you a headache within seconds.

This demand to write screaming fast code leads to an expectation in development projects that all code be optimized for speed. This is a truly terrible idea. I am here to explain why I think optimization should be avoided except in the tiny percentage of the project that actually is performance-critical.

I had my first whiteboard interviews in 1988. The CPU in most common use was the 8088, a very slow processor with only realmode addressing and a maximum of 640K of RAM. Most music files today are many times this size. At this time optimization was important, even simple operations ran slowly if we followed all the rules of structured programming.

This has not been true for decades. Yet the compulsion is undiminished.

My core point here is that highly optimized code is significantly more prone to bugs than carefully structured, boring, unclever unoptimized code, which is what we should be writing probably 99% of the time. I take this as axiomatic, thirty years of programming have taught me that cleverness may add speed but it makes up for it a thousandfold with its instability. I am irretrievably convinced of this. The place for serious performance improvement is at the design level, not in low-level code.

Consider the software process. At the time of the 8088 very few companies had the development/QA process we have now. Bug reports may have been written on whiteboards and erased when fixed. The processes we have now

  • Developer writes a feature
  • Tester finds a bug
  • Tester spends 20 minutes making an entry in a bug tracking database; circumstances, reproduction scenario
  • The list of open bugs is triaged by a team of several people whose time would be better spent creating bugs of their own. It is prioritized and assigned back to the developer.
  • After days or weeks the developer has to refamiliarize himself with his own work because its cleverness makes it hard to understand and in any case he has cultivated an illegible “personal coding style.”
  • Developer fixes the bug (or thinks he has).
  • Back to QA, where if the bug is not actually fixed it goes back to triage. After another fifteen minutes of bug tracking updates.

This can take weeks. In the end dozens of man-hours have gone into this one bug whose cleverness saved a microsecond or so of CPU time, as opposed to tens of billions of microseconds spent in the process bulleted above.

Avoiding bugs should be a supremely high priority, and to accept bugs as an inevitable part of the process is no longer acceptable. Nothing delays release like fixing bugs.

Now: most of us write applications; I write servers; some write device drivers. In a device driver the bulk of code is performance-critical; in the application case only a tiny percentage of code is actually performance-critical; most of it is user interface code doing unremarkable things on user time scales, where shaving off a microsecond is nothing short of folly, even if it wasn’t destabilizing, which it is.

So if I find myself in my office dead tired at 3AM Sunday fixing a bug, that bug had damned well better be in code that matters, in code that adds competitive value to the product, code that distinguishes the project from others. If I find myself fixing bugs in perfectly ordinary and unremarkable code that does ordinary, commonplace operations but which has been “optimized” into oh-so-clever fragility, I am going to be angry. If it’s my own code I will take a lesson to be stodgy and boring and to eschew cleverness, as I have learned.

But if it’s someone else’s code I am going to be very angry. And I am going to lower my estimation of my coworker’s wisdom.

Optimize rarely. Only optimize code that actually is critical.

Thanks for reading.