The Regression Death Spiral

Blake Norrish
Mar 3 · 11 min read

Here’s a story you might find familiar:

You’re a QA on a small development team; it’s Thursday morning in the second week of your two-week sprint. Your team still has several stories in development, and the stories need to be signed off by Friday morning to be ready for demo Friday afternoon. Your plans for a relaxing Thursday evening start to evaporate before your eyes.

In morning standup, the developers say they are wrapping up. You know they write unit tests and component tests as part of development, and you have already completed several feature walk-throughs with them, so you’re confident you’ll be getting high quality code. However, in addition to testing the stories you’ll still have to finish up the UI and API automation, validate some non-functional requirements, and do some ad-hoc sanity testing… and one day isn’t a lot of time. You can probably finish testing, but getting the automation wrapped up is going to be impossible. You give this bad, bad news to the scrum master.

“OK” says the scrum master, “how about we…” and you know exactly what is coming next. “ … how about we push automation to the next sprint?” Automation, explicitly called out in the definition of done, built into the story and always part of the estimate, is getting pushed into the future. Again. “I don’t like it either…” your scrum master says, seeing the expression on your face, “…but there’s a lot of visibility on velocity right now, as we’re coming up on that big milestone – we can’t not hit our committed points.”

“I understand” you reply, with a smile to show that you are a team player, and are also committed to ensuring the team looks like it’s succeeding.

End of story.

I’d wager this story is sadly familiar to many of you. It doesn’t matter the reason or context, we inevitably find ourselves in situations where we are pressured to delay, reduce, or just skip test automation. Unfortunately, the importance of automation within agile delivery and the implications of deferring it are lost on many stakeholders, planners, managers, etc. To them, test automation seems to be a requirement easily deemed nice to have, then cut from scope when deadlines loom.

I have been in this exact situation, and have found the phrase Regression Death Spiral to be a nice encapsulation of several testing concepts, and useful for communicating the risks of inadequate test automation to stakeholders. If you find yourself continuously defending the need for full test automation within your agile teams, add it to your arsenal and hopefully you’ll have as much success with it as I have.

Regression Testing

Before we get into the Regression Death Spiral, we need to understand the idea of regression testing – specifically, that testing a feature is more than just testing the feature.

Software is just lines of code, written at different times, by different people. Sometimes the code is written years apart, by people who will never meet. And all this code has to work together. This is no small challenge, and software engineers have developed strategies, approaches, patterns, and techniques to make building complex systems a little easier, less risky, and more predictable. These strategies help ensure that changes made *here* in software only have expected effects *here* – and not unexpected ones anywhere else.

Unfortunately, regardless of how careful we are when implementing new code, there’s always a chance that changes will introduce unanticipated side effects, that code for our new feature will in some way impact old features, in neighboring or even completely unrelated areas of the system. Given how intertwined and interconnected software code gets, as new features are layered over previous features by new people or new teams, the likelihood of unanticipated side effects can be anything from possible to expected.

Changes that break previously working code are called regressions, and there is an entire category of testing dedicated to detecting these types of defects: regression testing. Regression testing is nothing more than proving that things that used to work, still work, that new code did not have an unanticipated side effect in other areas of the application.

Thus, it’s naive to think of testing a feature as simply testing THE feature. Testing a feature requires a significant amount of testing outside and around the feature –in related or just high risk areas of the application – all because we must assume unintended side effects will occur.

The risk of introducing breaking changes outside of the feature in development and the need for regression testing is often overlooked by stakeholders or people without development experience. I’ve even heard it said that testing a feature is simply testing the enumerated acceptance criteria of that feature(!!!). Unfortunately, ignoring or underestimating the cost of regression is a key contributor to entering a Regression Death Spiral.

Agile Delivery and the Burden of Regression

Agile delivery introduces two concepts relevant to the Regression Death Spiral. First, agile delivery divides features into small, discrete user stories. Each user story should be atomic, so that it can be developed and tested as a single entity. Second, agile delivery encourages short development cycles with small, iterative releases. All flavors and hybrids of agile, Scrum, Kanban, SAFE, LeSS, ScrumBan, etc, in their own way, leverage these concepts. Let’s look at each individually.

Agile delivery shifts from large features defined by (even larger) specification documents to user stories that each represent some small amount of overall scope. Agile stories are independently developed, tested, reviewed, and signed off.

The use of small, discrete stories can create a false sense of isolation between code changes in different stories while undermining the case for regression testing. Yes, user stories are independent, but underneath these stories is the same interconnected code, and the same risk of introducing unintended changes. It does not matter how cleanly you divide your stories or how well you define your acceptance criteria, regression testing is as necessary in agile development as when developing a single large feature in a waterfall process.

This is a key point and is necessary to understand the Regression Death Spiral: changes in one story, or the addition of a new story, can still negatively impact the behavior of every other story. The move to small, discrete stories helps us define, understand, and parallelize work, but it in no way guarantees isolation or prevents one story from impacting another. The necessity of regression testing to identify these impacts has not changed.

Let’s move to the second concept – that agile delivery encourages small, incremental delivery. If it was only necessary to regress an application every six months, the cost of regression testing, even slow manual testing, would not be debilitating. In fact, this is how we used to deliver software before the transition to agile development – we would have a regression test plan, and every six months we’d run though this plan before a release. This would often take days or weeks, but it was not hugely burdensome within the scope of a six month release cycle.

The shift to small incremental delivery significantly increases the cost of regression by decreasing the interval between regression passes. Now regression might be required every two weeks, or even multiple times a week. This is a huge shift from the days of waterfall delivery. Imagine the futility of running a manual two week regression pass when your sprint length is exactly two weeks!

Let’s look at a simple hypothetical set of sprints to make it clear how quickly the cost of regression can burden a team’s velocity.

Assume we have a newly formed agile team just starting their first development iteration (sprint). In the first iteration (or some initial duration in a flow-based model) we might have three stories to test. The testing effort of these stories is fairly straightforward – we just test those three stories. In the second iteration, we pick up two additional stories. What is the testing effort of these stories? The naive answer is “somewhat less that the first sprint’s effort,” as we are only testing two stories. However, based on our understanding that code changes can cause regressions in any area of the application, we know that we must test both the two new stories, as well as regress the previously completed three stories.

Thus, our actual testing effort in the second iteration is actually greater than in the first, despite the fact that we picked up one fewer stories.

Obviously, this is an overly simplistic model, as it assumes that the effort required to manually regress a previously completed story is equivalent to testing a new story. Of course, not everything needs to be fully regressed, and testing something is always easier the second time. However, the cost of regression is non-zero, and for complex software it can approach the initial cost of testing. The exact ratio of initial testing to regression testing does not change the fact that some significant amount of time in iteration two will be spent regressing stories completed in iteration one, we can’t just assume they will still work.

If we extrapolate this pattern out for many sprints, we see that in an agile development model with manual regression, the cost of regressing previously completed stories quickly swamps the cost of testing new stories – the number of previously completed stories accumulates every sprint, and the entire set must be regressed every iteration.

This is the fundamental challenge of testing in an agile methodology – the incremental, iterative nature of the process necessitates a vast amount of regression testing every iteration, as new features & stories are added to the system much more quickly, at a cadence of days or weeks rather than months or years. Every time a story is completed, not only must it be tested, but all previously completed stories must, to some degree, be re-verified. Manual regression cannot scale with the increased delivery cadence of agile.

This is a Regression Death Spiral – every iteration it gets harder and harder to adequately regress the existing application. The team’s productivity is suffocated as the burden of testing previous work consumes a greater and greater portion of total overall capacity. Left unchecked, the initial velocity of the team will slow and then grind to a halt, as every new change requires a substantial and growing set of previous stories to also be tested.

Automation in Agile Delivery

In the previous section, we assumed that every test was manual, and the cost of rerunning a test was some fraction of the initial effort. Luckily, we know this is not the case – we know we can use software to automatically validate previously executed tests via, you guessed it, test automation!

Test automation exists in many forms – unit tests, API tests, component tests, integration tests, E2E tests, UI visual regression tests, etc. It doesn’t matter who writes them or how they interact with the application, all automated tests help to verify existing functionality and provide confidence previous work has not broken, reducing the manual regression burden.

It is important to understand that test automation does not accelerate initial testing. In fact, implementing test automation often slows initial velocity – automation is a development effort that requires thought and expertise just like anything else! Automation is not testing – it’s simply a programmed check of something that has already been tested. Fortunately, this is exactly what we need to alleviate a Regression Death Spiral!

If you consider the points of the two previous sections – that changes in software can have unintended impacts, and that agile delivery is just a continuous stream of small changes – it becomes obvious that agile delivery cannot function without test automation. Attempting to deliver iteratively without it is like going on a long road trip but refusing to stop for gas because the stop will slow you down. Gas is a fundamental part of car travel (for now), just as test automation is a fundamental part of agile delivery. Automation is the only thing that can mitigate the suffocating burden of regression and avoid the resulting velocity collapse.

It is rarely the case that a stakeholder or manager would argue that test automation should be ignored completely, passing 100% of testing efforts on to future sprints as manual regression. Much more commonly, teams are pressured to accept stories with partial or good enough automation. Doing this still passes some percentage of overall regression cost into future sprints as manual, non-automated regression. Unfortunately, any unfinished automation, regardless of how small, necessarily increases the regression burden in future sprints.

Leaving any amount of regression as time-consuming, manual testing creates a slippery slope that can quickly lead to a Regression Death Spiral. If you are testing and automating a story, what happens if, because of time pressure, we acquiesce to stakeholders and call the story done with incomplete automation? Some amount of regression burden is passed on to the next iteration, which then consumes even more testing time in the subsequent sprint. This gives even less time for automation in that iteration, causing even less testing to be automated, passing even more into the future, etc. This pattern repeats iteration over iteration, creating a positive feedback loop until the QA has zero time for automation, spends long hours and evenings barely keeping up with manual testing requirements, then quits to go look for a company with more mature agile practices.

It’s my strong opinion that there is no such thing as healthy agile delivery without full test automation. Fully automating stories, with an automation suite spanning all forms of tests (unit, component, integration, UI, etc.) is absolutely necessary to maintain delivery velocity over time. Any suggestion of removing this requirement from the definition of done for every story should be resisted and called out for what it is – a short term distraction to create the perception of success while jeopardizing long term development objectives. Not to mention QA sanity.

The Death Spiral

Let’s go back to our little story.

It’s Thursday morning before demo day and your team still has several stories in progress. Developers are wrapping up, and the code should be ready for testing very soon. However, there isn’t a lot of time remaining. A stakeholder, hearing this bad news, calls an emergency meeting and suggests deferring test automation to ensure the team hits its sprint velocity commitment.

You are about to protest, when your scrum master cuts in: “We can’t do that! It’s the first step of a Regression Death Spiral!”

“Automation is always part of the definition of done” adds a developer, “it’s part of the product, just like the customer-facing code.”

“Do you want us to succeed, or just appear to be succeeding?” This, coming from your business analyst, puts a smile on your face.

You decide to cancel that interview you have lined up for next week.

Acknowledgements

I did not coin the term Regression Death Spiral, but have never seen it explained to satisfaction in writing. The earliest use of the term I can find is in Testing Extreme Programming (2002) by Lisa Crispin and Tip House.

Several other references can be found with a quick Google Search:

http://ivory.idyll.org/blog/software-quality-death-spiral.html (2008)

https://xebia.com/blog/regression-testing-with-an-agile-mindset (2010)

Slalom Build

Thanks to Tim Knapp and Chuck Snavely

Blake Norrish

Written by

QA, Software Developer, Consultant, Pessimist.

Slalom Build

The Build Blog is a collection of perspectives and viewpoints on the craft of building digital products today, written by the technologists that build them. By builders, for builders.

More From Medium

More from Slalom Build

More from Slalom Build

More from Slalom Build

More from Slalom Build

Event Sourcing with AWS Lambda

259

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade