Removing the Tester Safety Net

by Gareth Thomas

Moving to Continuous Delivery and a Quality Focused Process

We’re all familiar with the waterfall approach of software development. It keeps skill-sets in silos and, from a tester point of view, we were the ones squeezed for time when projects overran.

Adopting agile in the latest Membership Programme incarnation at the Financial Times many years ago started to make a change. The concept of starting to break work into smaller pieces and working much closer to one unit as a team removed the big bang approach of these problems. Ultimately they still existed. Like most development teams our testers were outnumbered by developers, but ultimately had as much if not more to do. The introduction of automated testing if anything made matters worse. When you’re new to agile you can struggle to work out where to build automated tests into the process. We agreed that they needed to be part of the sprint from day one, but this meant we still had split skill-sets — manual and automated testers. Both were needed to get the work done.

Over time you realise there are some inevitable improvements staring you in the face. Agile is very good at identifying bottlenecks in your processes. By the book of Agile the team is now expected to “swarm” to the affected area, but it really doesn’t always work that well. There is an inherent historical separation between developers and testers. If stereotypes are to be believed, then testers don’t believe developers can test, and developers don’t want to test. If you’re working in a world where testers are writing automated tests, then you’ve already broken this wall down. You have testers developing. What about unit tests? I’m sure you’ve got those, and I bet they’re written by the developers. So you have developers testing. You start to see overlaps between the roles.

But ultimately we have people with different backgrounds and different strengths which can use to our advantage. If the testers are the best people to suggest what should be tested, but the developers are best at writing code, why don’t we do that? The developers can write the automated tests that the team uses. In turn the testers could assist with better unit tests. This leaves the testers free to do the manual testing and minimises the work they need to do. It also places some responsibility on the developers for the testing process. So that works, right? Well, no. These are baby steps on the way to adopting a QA approach.

I don’t believe that ‘tester’ and ‘QA’ are terms which are interchangeable. A QA’s focus should be on quality throughout the process. A vast majority of that will be through testing, but may not be testing they are carrying out. They are making sure the team as a whole is doing the right thing. With that approach we can expect QAs to work closely with BAs, playing a significant role in any planning or scoping session. We need to ensure that everyone involved with any piece of work is fully aware of what needs to be done before a single line of code is written.

So, you’ve got QAs overseeing every stage of development and we’re automating all tests. This is surely an improvement and ultimately what you’re talking about here? No, again. We can go further.

At this point you still have someone acting as some form of gateway. I guarantee that you’ve worked on a team where someone will use the term “sign off”, or “can this go to production?” You can move into a world where this isn’t needed. Ultimately we want Continuous Delivery, but you can still get this to work without a fully automated process. Build your tests into a pipeline, even if every stage of the pipeline is manually triggered. You will need a team that understands what is in that pipeline. You need a QA that is confident that all of the tests and processes associated with that pipeline are as good as they can be, and provide all the information necessary for a deployment decision to be made.

Integrate as much automated testing as possible into your pipeline. Understand that when a build “goes green” that an adequate number of unit, regression, performance and security tests have been run. If you’re following Continuous Delivery this build will now be in production. If you have manual deployments you can now deploy it. No-one has physically looked at it. Machines have determined the overall quality and determined whether this can be deployed. You have removed the safety net.

I said “removed the tester safety net”, not the “safety net” as a whole. That’s because it would take a special kind of bravery to end this situation here and believe everything will continue as it is. No. Monitoring is now your friend. Or more importantly, automated monitoring. This is now your safety net. These monitors like everything else will need to be planned, developed and tested. Any findings from your production system, be they outages or “blips” can be fed back into the team process to make improvements to this system and any future developments. Monitoring in this manner also means you are continually testing a live system.

To make this work you simply must have collaboration and trust. It’s a cultural change. Developers can test and QAs can contribute to the development of those skills. QAs must be able to trust the automated systems used for deployment. They must also trust that developers can do the right things. The combination gives confidence. But you need developers that will embrace this approach and will seek out QA advice when they need it. We shouldn’t underestimate the importance of having the right developers in this process and this simply wouldn’t have worked had the development managers not recruited in the way they have. You’ll need to change your focus for your QA recruitment too. You’re now looking for people who can spot an opportunity to make improvements, and make them thinking about the customer and how to make things better for them. The whole team is responsible for the system in development and in production through support and monitoring. They are building confidence into the whole pipeline. Now you have removed the tester safety net.

So what do the QAs do now they’re no longer testing? Well, actually they still test. But ultimately this will not form part of every story. QAs can spend time exploratory testing the production systems and feed findings into the backlog. They identify metrics by which we can continue to provide quality. If you’re building a system to do something, we can count how many times it has worked. We should have an idea before we start even developing what the likely take up is. If no-one is using it, we’ve either built the wrong thing or it simply doesn’t work well enough (or at all!). You can apply this to defects, incidents, number of alerts, failed builds etc. They look for patterns in test failures to identify problem areas. Ultimately they will spend a great deal of their time with their respective team members helping to ensure that quality continues to be built into the process. Building that confidence into the delivery pipeline. The number of new developments and subsequent tests, code reviews, new systems to monitor etc won’t stop and ultimately we shouldn’t stop trying to make them better.. just without one person acting as a safety net.