Engineering Against Chaos

If the prevailing model for SaaS startups were put into machine-like terms it would be something like this:

Product and engineering take inputs of money, equipment and time and return features, which enable the business side of a company to sell to more customers or keep and expand existing ones. In turn that money is reinvested into more of customer acquisition and more product and engineering and round and round it goes.

As long as sales can sell to more customers and account managers can serve the needs of existing customers (keeping existing customers happy with the product) then the company grows and can invest in new product development.

As long as engineers and product people can keep churning out features that create enough value for customers, then their part of the corporate machine is running in the green. In turn, a successful engineering and product team helps the sales team and account managers have an easier time selling and/or serving their (target) customers. And so these feedback loops can amplify each other.

The best measure I have found for a good engineer is how little entropy they introduce with every addition of code.

This model takes monitoring and maintenance to keep running. If customer acquisition costs are higher than net new annual recurring revenue then adjustments in strategy and approach have to be made. If engineering results aren’t valuable enough to justify their budgets then a change in strategy and approach is needed. There are a million ways to cross section data to monitor this and come up with strategies to keep this engine in tune.

But! There is an entire other system at work here and it is often entirely ignored, misunderstood or behind the scenes.

Anyone that has been responsible for an engineering team knows that the power of a talented group of engineers is unparalleled. It is also known that the ability for engineers to create chaos is equally unparalleled. Every time an engineer turns time, equipment and money into products and features there is a byproduct; the same byproduct of any machine, entropy (AKA chaos).

The best measure I have found for a good engineer is how little entropy they introduce with every addition of code. If they can create a valuable feature without breaking everyone else’s work, then that is a desirable engineer.

Inevitably though even the best engineers introduce some chaos into the systems that they work to improve. But eventually unchecked entropy will overwhelm a company’s ability to produce new features or even keep the ones they have, running.

Systems Administrators, DevOps Engineers, Site Reliability Engineers, whatever you call them at your organization, it is these peoples’ job to be able to take on the chaos introduced by other engineers and keep it from degrading the organization’s operation.

Often you will find these people as the firefighters in your engineering organization, but that is these engineers at their worst. Given the right environment these engineers can be your software fire-preventers. Instead of they themselves being the anti-chaos machines, they build tools that are always working to counteract entropy.

This pattern isn’t bound to engineering, on the business side there are the firefighter types that heroically save the day. If these people are just firefighting they are not being given the opportunity to be their best.

These anti-entropy engineers set up tools to help themselves and the other engineers in the fight against the degradation of systems; tools like:

  1. Unified frameworks of thinking around parts of the development process so that there is consistency and continuity in how an organization transfers features into production systems.
  2. Unified practices, to counteract the tendency for each engineer to build things their own way.
  3. Development tools, to enable engineers to build features faster and safer. Like sandbox environments (copies of the production environment for testing) or scripts to shortcut common task
  4. Documentation, to enable engineers to take on tasks and problems they would not otherwise have the ability to tackle.
  5. Metrics and monitoring, so that every engineer can have a pulse on the effect their code is having on an organization’s systems.
  6. Systems of redundancies and backups, so there is no single point of failure that can permanently damage a system.

Each of these (and there are many more) is a little anti-entropy machine built into the digital infrastructure and culture of a company continually empowering the fight against entropy in a digital network.

The goal of these anti-entropy engineers is to be stabilizers and enablers for other engineers.

If done right, the work of software engineers does not create enough entropy to overwhelm these tools and ultimately the organization itself. Thus the feature development process is more like being strapped to a rocket instead of moving through molasses.

The goal of these tools is to be able to handle not only the continuous chaos created by operating a digital business, but also meet the challenges of random spikes in chaos when systems start to fail.

In this way anti-entropy engineers are like a knee or ankle brace which do two things. They align all the energy so that all of the energy goes towards forward motion instead of some portion of it working to injure the joint. They also take the stress away from the joint so it can heal in the case where a joint is injured. By stabilizing crucial joints, these braces prevent or help heal injuries and also they allow the runner to go harder and longer with less fear of injury.

This pattern isn’t bound to engineering, on the business side there are the firefighter types that heroically save the day. If these people are just firefighting they are not being given the opportunity to be their best. When they can build the tools to prevent fires then the organization can really take off.

The goal of these anti-entropy engineers is to be stabilizers and enablers for others in the organization. This only works in a sustainable way if these anti-entropy engineers are able to undo entropy faster than other employees can produce it. By no means an easy task, but one that helps propel an organization towards innovation and success.

Anyone can sustain insane levels of innovation over a year, maybe even two. If you are starting to see innovation slow as you scale, it may not be the people you are bringing on or the systems you have in place. The cause may be the people you have not brought on and the practices you do not yet have in place.

Next time you wonder how the most innovative companies are able to sustain their incredible paces and forward momentum, look to their leadership and also their anti-entropy employees because those are the people that make an organization into an environment where every other employee is free to innovate and build with abandon.