Platform health is the key to high performing teams

All stakeholders and team members want to be part of an efficient and effective process. But how does a business identify what it needs to do to create a culture of high performance?

After a complete outage in 1999, eBay went nearly 2 years without releasing a major feature so they could re-platform. In 2002 following the TWC memo from Bill Gates Microsoft went on a feature freeze for 1 year. The same happened with Amazon in 2004, Twitter in 2008, LinkedIn in 2009, and many more.

These businesses all nearly imploded because their teams were performing poorly. They needed to stop trying to build new features and address the health of their platform in order to survive and ultimately thrive.

According to Gene Kim (founder of Tripwire and author of The Phoenix Project), high performance in teams begins with DevOps. His definition of DevOps is a very useful one — because rather than say what it is, it describes the outcomes we should aspire to:

Devops is … “The architecture, technical practices, and cultural norms that increase our ability to deliver applications and services, enable rapid experimentation and innovation, and deliver value to customers quickly.”

The State of Devops Report 2019 used Cluster Analysis to identify trends in the deployment practices of teams who rate their performance as Elite, High, Medium and Low. It looked at things like deployment lead time and frequency — finding that these particular metrics are 100 and 200 times better in Elite vs Low performing teams. So what can be done to make sure your teams can achieve these better deployment practices that are so closely linked with team and business performance?

Technical debt = risk

Part of this at least can be attributed to the health of the architecture and code. Something engineers will often call ‘technical debt’. Technical debt leads to increased fear of deployments, increased complexity in testing, and increased pain of development. The trouble though with the term ‘tech debt’ is that it doesn’t transfer its meaning well from engineers to execs. Debt to an exec can be something quite acceptable — a strategic norm to be paid off later without consequence along the way, but this is not the case when it comes to platform health. Maybe instead we can use a shared concept that engineers and execs both understand: risk.

In a project full of technical debt, risk escalates the longer the technical and architectural problems are ignored. Fixes are deferred, which leads to more painful releases which slows a team’s ability to bring business value, leads to fear and dissatisfaction amongst engineers and eventually project meltdown. In order to avoid this risk from escalating we need to understand when to step in to fix the problems — and to do that we need a metric to measure against.

Risk can be measured in ‘pain’

Arty Starr (Founder of DreamScale & Author of Idea Flow) talks about one such metric to measure this risk in her book Idea Flow — the pain felt by the engineers developing the software. Working in a complex piece of software often requires debugging and troubleshooting to find out how it works — essential when implementing a new feature or fixing a defect. Your engineers are happily coding away when they find something unexpected — this leads to confusion which in turn impedes progress. This confusion is a normal part of the development process — but when it extends beyond a certain time, that causes us pain. Arty suggests that debugging something for more than 50 minutes is excessive and should always lead to a conversation to find out how to address the underlying issues. She suggests engineers should keep a record of their progress and note when that confusion moment begins and ends. If there’s no investigation and subsequent mitigation when an engineer feels pain, these problems will lead to ever increasing project risk.

Don’t be afraid to pull the andon cord

This reflection every time an engineer’s confusion extends beyond a certain tolerance is not dissimilar to the idea of the andon cord in a Toyota plant. The principal is that a single cord runs through the plant and any time someone on the production line finds something wrong they can pull the cord and stop the whole line. It could be that a component arrives at a worker’s station in an unsatisfactory condition — they pull the cord — the team find out how that happened — and put in place something that prevents it from happening again. Perhaps the assembly of a part took one minute instead of the expected 40 seconds — pull the cord, investigate, mitigate.

The idea beyond stopping everything any time something is wrong is simple — that 20 second delay is compounded over time and has knock on effects further down the chain for everyone — so to de-risk the whole business, immediate intervention is always preferred. There should be no shame in calling this out. Often. In a typical day at a Toyota manufacturing plant in Kentucky the andon cord is pulled, and the production line is completely stopped, 5000 times per day. Yes, five-THOUSAND times every day. This is how Toyota maintains its position as a high-performing business. Not by producing more features, but by refining their process to be more predictable.

Invest in platform health in order to de-risk

Looking at this data it’s clear businesses need to invest in platform health in order to de-risk. If engineers are suffering daily pain; if deployments are taking days instead of minutes; if engineers are scared of the codebase they are working in — it’s time to act. So many large businesses almost imploded because they didn’t address their technical debt and that put their whole business at risk.

So if you’re a devops engineer feeling the burden of deployments; a developer feeling the pain of feature development or debugging; or if your an exec wondering why you’re business isn’t innovating fast enough — maybe it’s time to start talking with one another about the risk that you have inadvertently accepted by neglecting the health of your platform.