Chaos and Order in Software Development
--
In this post, I want to share one of the most important ideas I heard in the last few years — that most companies are too well-run.
Chaos, Order, and Productivity
The original formulation of the idea comes from Jim Keller who explained it with a seemingly trivial chart of productivity vs. organizational order.
In the chaos, nothing can be done. Priorities change overnight, untested code gets deployed to production, databases fail with no backup, and there are five different languages and three libraries for any single task.
When there is a sensible order, productivity is at its peak. People can focus on the actual goal at hand, have time to do it properly, release it, and have it just work.
But when the order is perfect, nothing can be done again. Writing a line of code requires three planning meetings, four approvals, and a separate team to deploy it. Anything remotely novel needs a couple of Ph.D. theses to even try.
Now, here’s the key observation — once the organization starts to move towards order, it can’t be stopped. The move continues until nothing can be done.
Team Variance
The same idea can be formulated in terms of team variance. Say, a team takes on a number of different projects and addresses them using whatever technology and approach seems reasonable. This can work in three fashions:
- High variance — there are good projects, there are fantastic breakthroughs, and there are total failures.
- Medium variance — we take reasonable steps to eliminate the most common failures, and end up with some good projects, some breakthroughs, and very rare failures
- Excessively low variance — we’ve standardized everything. Every project is completed on time, on budget, and fails to impress anybody.
The axes in software development
Speaking about company organization is rather abstract. When we look at software development, there are a few axes…
