5 Minute DevOps: Organizing for Failure

Bryan Finster
IT Revolution
Published in
3 min readJun 22, 2020

--

I recently saw a documentary on the history of the Ford Edsel. There were so many lessons on the UX research, design, manufacturing, and marketing of Ford’s new brand that apply to developing any product. I suggest everyone look into it. Since I only have 5 minutes, we will focus on Ford’s quality process for the Edsel.

In 1958, Ford started to manufacture the Edsel. Since this was an entirely new design, they didn’t yet have an assembly line for it. So, they shared the line with other Ford cars. The Edsel wasn’t just new bodywork. It had a new engine and more complicated assembly with many new innovative and complicated options. To accommodate the expected sales, they modified the line so that every eighth car was an Edsel. This, of course, led to constant context switching on the assembly line. Also, since the car was named after Henry Ford’s son, the previous chairman for whom the United Auto Workers had no love lost, there were occasional “assembly challenges.” However, Ford had a well-defined QA process that would keep the line moving.

Ford’s QA process ensured dealers got only the best Ford could deliver.

  • As each car rolled off the line, it went through road tests and quality inspection and was given a quality score.
  • All of the quality scores for the day’s production were averaged; if the average daily quality score met the minimum, the cars were shipped to the dealerships.

Since Edsel’s represented 1/8th of production and their quality scores were averaged in with the other 7/8ths, cars often shipped to dealerships with major issues to be resolved. This process worked fine for Ford until Toyota began disrupting the market.

The Toyota production system was created to reduce costs and increase quality so that they could compete. Core to the TPS is reducing waste by minimizing batch size, inspecting quality at every step, and preventing poor quality from moving to the next step. They made quality concurrent with manufacturing. This proved highly effective for increasing market share and allowing them to pivot to market needs rapidly. The Toyota system is something other manufacturers have worked to emulate for more than 50 years.

After watching the documentary, Ford’s quality process sounded very familiar to my experiences in the past delivering software.

  • Develop features, often with shifting priorities and pressure to meet deadlines for the scheduled release.
  • Throw it over the wall to the testing team who logs defects
  • A go/no-go decision is made on the large batch of changes based on the criticality of the defects.
  • If the minimum quality is reached to ship, then the release goes to the customers and an operations team fixes the defects based on customer complaints.

This process hasn’t been a viable business model for manufacturing for more than 50 years, yet it’s all too common in software development.

A friend of mine told me that his organization’s QA team had built a framework to make it easy for developers to fully test every change on their desktops. They were teaching teams TDD and BDD and were moving quality checks as close to the source of defect creation as possible. They were also focusing on the speed of feedback. A developer could get quality feedback from a fully integrated test in less than 3 minutes. They were taking Toyota’s lessons to heart and rapidly improving the teams’ delivery abilities.

My friend also told me that one of the development managers in his organization is pushing back on the whole idea. “Why are we wasting time on this? Why are we asking developers to test? We have a testing team.” Attitudes like this hurt the bottom line.

Is your organization executing a continuous delivery development process with small, tested changes and rapid quality feedback to the developers? If not, when will you finally catch up to where manufacturing disruptors were over 60 years ago? Software is eating the world, but only for those who can move from artisanal development to modern software delivery.

--

--

Bryan Finster
IT Revolution

Developer, Value Stream Architect, and DevOps insurgent working for Defense Unicorns who optimizes for sleep. All opinions are my own. https://bryanfinster.com