Regression Testing Wisdom from Agile Testing Days 2017

At my current job in a regulated industry, our software goes through a validation testing period before it goes live. A build is cut. It gets deployed to the validation environment, integrated with our other products. Depending on the size of the product or service, the testers have a few days to a few weeks to generate documentation that describe the features and the environment in case of an audit. Ideally, we’re regression testing during this time. While we continue to look for unexpected behavior, finding anything out of the ordinary means a headache for more people than I care to admit.

Sansoucci Park in Potsdam, Germany. Far away: Roman ruins. Closer up: 19th century ruins? After I read the plaque: reconstruction finished in 2014. Always read the plaque.

I left my team in the middle of a validation cycle to attend Agile Testing Days in Potsdam, Germany November, so I had regression testing on the brain. Here’s some of what stuck with me.

Jez Humble: Humans should not be regression testing.

I have this in RED in my notes. His talk about continuous delivery emphasized the speed of delivering customer value through continuous delivery. Enough said I think.

Lisi Hocke: Get developers to automate regression.

Lisi had an inspired idea to get this going: invite your developers to help you manually regression test. After a few hours of repetitive tasks, they’ll get motivated to write something that makes both your lives easier.

Angie Jones: People don’t want to collaborate with you when you have twelve spreadsheets.

I get very worried when I see a regression checklist that involves multiple sheets within the same spreadsheet. I start counting up how long it might take to get through, assuming we don’t have to circle back and investigate, ask questions, or report bugs. I wonder if things have changed since items were added. I wonder who was consulted about which items were high-priority, or necessary. I wonder about what might be assumed about the environment setup, or left off entirely. Rather than building confidence or feeling at peace, multiple spreadsheets of regression scenarios raise my blood pressure.

Poornima Vijayashanker: Give new employees a particular task.

Guided or paired regression testing can be one way to get to know a product when you’re new to a company or a team.

Rob Lambert: Be brave. Integrity is important.

In the year I’ve been in my current role, I’ve only found a bug that required a deploy in the midst of a validation period once. Raising it was terrifying. This would be the first release of the service, so it was hard for me to tie the bug to customer value. I had to ask a developer to work under extreme time pressure, only to stick myself with the testing of it. Less time to explore meant less confidence after the fix than before. But as a tester, it’s my responsibility to provide relevant information in a timely manner, even if that requires more than my average amount of bravery.

Mark Winteringham: The product is large. I cannot observe every change.

I’ve worked with people who insisted I caught everything. I didn’t. No one could. Be terrified, but also liberated.

Liz Keogh: Tell people failure is expected.

Everyone else goes into validation testing expecting success. I go in expecting failure. What if I told people we would fail? What if we planned for it? What if our deadlines accounted for failure?