Programmer’s discipline

What we can learn from those who scale systems for a living

I was reading Patty McCord’s book, “Powerful” when this line caught my eye (Patty is ex-CHRO, Netflix):

My whole career I have gotten along well with engineers, because engineers are very, very disciplined. When engineers start to whine about a process you’re trying to implement, you want to really dig into what’s bothering them, because they hate senseless bureaucracy and stupid process. But they don’t mind discipline at all.

I’ve spent much of my career learning to scale the people side of tech, first as a software consultant, then as a founder, and most recently at a hypergrowth startup. I’ve also programmed for fun since I was six. One could say I’ve seen both sides, tech and non tech. IMO, Patty is absolutely right, and it really helps to hear it from someone with her depth of expertise.

All competent programmers are experienced at designing, building and running large, complex systems. Most will eat your typical MBA for lunch when it comes to structured problem solving or execution discipline. There’s a reason product teams with several managers, dozens of staff and a multi million dollar budget are routinely outperformed by open source teams with no professional managers and a staff consisting entirely of volunteers and with zero budget.

So, what does it mean to say “Programmers are disciplined?”

Why is a programmer especially good at spotting senseless bureaucracy and stupid process?

How can non tech decision makers bounce off programmers’ skills to build orgs that are more effective?

Managing scale through tools

Building tech products is a Hard Problem™, and humans use tools to make Hard Problems tractable. Some of these tools are material, like hammers and nails, others exist only in our minds, like money and laws.

Teams depend on two meta-tools to manage the strategies, orgs and processes that are needed to solve Hard Problems.

We call them Systems and Disciplines.

From the wikipedia article on Systems

A system is a regularly interacting or interdependent group of units forming an integrated whole. Every system is delineated by its spatial and temporal boundaries, surrounded and influenced by its environment, described by its structure and purpose and expressed in its functioning.

and Disciplines

Discipline is action or inaction that is regulated to be in accordance (or to achieve accord) with a system of governance.

tl;dr

We break things down into Systems, and define how we engage with them through Disciplines.

For programmers, ‘disciplines’ are the processes and tools used to manage the complexity spawned by developing and maintaining large recursive systems.

Some (rather technical) examples of programming disciplines:

• Structuring files and directories well, and to a convention.
• Writing clear, expressive commit comments.
• Naming things correctly and expressively; more importantly, updating names across the codebase as the domain shifts.
• Moving from branching to continuous integration.
• Abstracting and encapsulating better.

Each one of these ‘disciplines’ operates at different levels of abstraction and in different sub-systems to protect against specific failure modes — and these are just the tip of the iceberg. Many of these disciplines are a perfect fit when dealing with the problems inherent in scaling businesses.

The consequences of systems

Programmers need systems and discipline because programming is difficult. This is well understood. But precisely how this difficulty manifests is crucial to understanding the programmer’s job.

Let’s break this down. I personally classify the challenges of programming into:

1. Ambiguitiy
2. Complexity
3. Integration
4. Paradox

Let me explain each of these labels.

Programming is the act of creating software. Software is a collection of rules combined with data that these rules act upon.

These rules need to be unambiguous, obviously — otherwise applying them is impossible for a computer. That’s one of the reasons special purpose programming languages exist: Human languages all use ambiguous grammars. Programming languages on the other hand use unambiguous grammars — a line of code can mean only one thing, and one thing alone.

When these unambiguous rules are applied in the context of a set of data, the software is said to be ‘running’. An example of such a rule in English is “When a User edits a Profile, ensure that it is their own Profile, otherwise reject the edit as invalid.”

So the programmer has to now eliminate the ambiguity in order to convert this rule to code, or it will not run correctly. Here’s a sampling of open questions:

• What is a User?
• What is a Profile?
• What is the relationship between a User and a Profile?
• What does “edit” mean? Who besides a User can edit? What besides a Profile can be edited?
• What does “ensure” mean? What do we do if we can’t ensure this?
• What does “reject” mean?
• What does “invalid” mean?
• Does this mean a “rejected edit” is by definition “invalid”?
• How is an “invalid edit” different from a “edit”?
• Is there such a thing as a “valid edit” and is it different from an “edit”?

I could go on, but my point is even a seemingly simple rule defined in an ambiguous grammar like English explodes into 10X–1000X more rules in an unambiguous one like Clojure. The programmer has to elicit and define all of them or risk causing severe bugs resulting in data corruption and crashes.

This is the first hard problem programmers have to solve — ambiguity.

As you can probably imagine, the rules for even simple programs trivially exceed even the smartest human’s cognitive capacity. Many programming paradigms and languages have been created in attempts to solve this problem over the years, all technically equivalent. Some have succeeded partially in their niches, most have failed, and none have proven themselves to be truly general purpose. This is why most full stack programmers are polyglot, with production experience in two or more programming languages.

Once the ambiguity is eliminated, the 10–1000X increase in detail becomes overwhelming because we are dealing with Complex Systems comparable in scale to macro economics or the weather, which manifest unpredictable emergent properties.

This is the second, hard, problem programmers have to solve — complexity.

Systems that are "complex" have distinct properties that arise from these relationships, such as nonlinearity, emergence, spontaneous order, adaptation, and feedback loops, among others.

And so, like mathematicians, soldiers, lawyers and bankers before them, programmers turned to systems and discipline to tackle the intractable nature of complex systems.

Programming is an exercise in Sytems Thinking in Complex (and I mean that in the mathematical sense) environments. Keep in mind, a key issue here is no human is smart enough to comprehend more than the tiniest piece of a complex system. As a whole, even a trivial program is as far beyond a human programmer’s ability to comprehend completely as the theory of relativity is to an ant.

Programmers overcome not being smart enough to hold every detail of a system in their head through the disciplines of abstraction and encapsulation. The code in a System is abstracted in a manner that aims to make each individual sub-system tractable to a (merely) human programmer. If a sub-system isn’t tractable, this process of abstraction is applied recursively, with each sub-system in turn being broken down into it’s own, smaller set of sub-systems until they become tractable to the intellect of a single human programmer. Eric S. Raymond wrote a superb email to Linus Torvalds on the topic back in 2000 that is well worth reading.

The code that allows you to read this line of text on a screen involves a few hundred to a few million such systems forming a massive, messy, recursive tree of interdependent systems. No single human on the planet understands the process well enough to walk you through every step involved in depth.

At this level of complexity, integrating sub-systems is now a non-trivial problem. We’ve replaced “I can’t grasp this” with “I grasp the pieces individually, but assembling the puzzle is still hard.” So the solution to complexity creates the third problem.

The third, hard, problem programmers have to solve — integration.

Integration of sub systems into a whole is attacked through execution discipline and strong encapsulation. Encapsulation mitigates the risk of sub systems not fitting together. This is done by modelling all sub-systems in a consistent way, drawing clear boundaries between systems and defining clear service agreements between them. But none of these guarantee success — most software projects fail at the integration stage. A simple partial solution exists, but is incredibly hard to execute: Continuous delivery, where we use automation to execute integrating, testing and releasing systems continuously to ensure that all the sub-systems integrate correctly to form the overall system.

Programmers create sub-systems in order to tame complexity. But this opens the door for different sub-systems to contain contrary, paradoxical rules that can result in everything from data corruption to full blown deadlocks. Sometimes these paradoxes can span several sub-systems, and manifest under a wide variety of conditions, from hardware issues to business teams with conflicting OKRs. So the solution to integration — abstraction and encapsulation — create the fourth hard problem.

The fourth, hard, problem programmers have to solve is — paradox.

Solving paradox is incredibly hard, as the chain of cause and effect forming a closed loop can be long and complex. Given that humans can’t hold systems at this scale in their heads, conflicting rules are only discovered at integration time.

What I’m labelling ‘Paradox’ manifests across all scales, starting with simple conflicts in lines of code in a file, to larger technical issues like deadlocks and race conditions, and on out to full blown inter departmental politics due to unforseen conflicts in OKRs.

Without solving for paradox, systems either fail to go to production, or once in production suffer from poor adoption and unintended consequences.

Here’s an example from India’s Aadhar system.

This is also why it’s important to value programmers with skin in the game over those who don’t. Otherwise you risk involving so called ‘ivory tower architects' who drive attribution bias to claim credit for successes while blaming failures on externalities.

Capable programmers mitigate risk

Most software projects fail. Anecdotally, more than 80%. Given this context, I believe there are two kind of programmers in the world: Ones with skin in the game, and ones without.

Those with skin in the game de-risk execution constantly, because they understand how risky building software is, and they also understand their rather large role in mitigating that risk. This de-risking will involve conversations about, among other things, scope and tech debt, and is a critical input into business strategy. Failing to de-risk is one of the most common reasons for systems to fail to keep up with scale.

In conclusion

Programmers are excellent sounding boards when developing non-tech systems. They can help spot inconsistencies and predict unintended side effects early.

This isn’t to say that programmers are right all the time about non tech (or even tech) systems. Far from it.

But nevertheless, capable programmers have a wealth of experience with systems development that makes them excellent sounding boards.