Mastering QA every step of the way
Mariana Furini talks about a new process implemented for QA at SumUp Brazil.
Quality Assurance shouldn’t be just one step of the way to production but rather an attribute to all the steps of the way, including for as long as the product is alive and out in the world.
Not long ago, here at SumUp, I started a very open conversation with my squad, where all of us were supposed to give our very honest opinion and answer to some questions, such as:
- “What do you believe is our current state in terms of testing?”;
- “Why do each of us have a different understanding of the types of tests we do? e.g. the levels, executions, etc”;
We ended this forum with: “Based on what we have discussed, where do we want to go from here?” at this point, it was clear we had a lot to work on but at least we were in sync and knew what was our long term goal. That you will find out as well further on…
Setting the baseline and building quality together
I believe that a “baseline” (i.e., minimum required or a starting point) is necessary to keep consistency in a cross-functional team. In a group of people with different functional expertise working towards the same goal, it is common that the members of the team, each separately, will tend to do only their part of the puzzle, meaning, working on the pieces in which they are experts on and leaving the product part to the PM only, testing part to the QA, etc.
However, at this point when the QA gets something to test, it is often too late to reach the desired level of quality. To start testing at the end of the development cycle turns us (QAs) into bottlenecks, where we are overloaded and short on time, mostly related to the pressure to deliver. Also, it prevents the other team members from learning and improving their QA skills. How can this be any good in an “agile world”?
So, let’s talk about the old mores of a development cycle, which, by the book means “a process of planning, creating, testing, and deploying an information system”.
The development cycle is a habit, of sorts, organically developed by the people in the team and changing habits have been proven to be really hard by psychology itself. It’s like having a newbie in the team who starts asking the “why” questions about things we’ve been doing the same way for ages. Challenging. So, how can we trick these old habits in order to change the way we do QA nowadays as just being one step to go through to get the feature live? There sure is a lot of room for improvement, How could the whole development cycle be more effective in these terms?
When the testing is performed only at the end of this cycle, the level of urgency related to time sensitiveness is a lot higher. This is where QAs have to make a choice and often this choice is to optimize for speed, which generally means testing the feature manually or rushing through it, what’s worse.
“Too often quality is overshadowed by the pressure for speed.”
As it is said by the IT manager at ING Netherlands, Jannes Smit, in the “Accelerate” book.
Needless to say that finding a bug on this stage is also more expensive and I am not even considering here the context switching cost, which is hard to measure and often overlooked. So, how can we make sure while we try to change this, we won’t fall back to our old habits just because they are “faster”? The understanding of “slow down to speed up” is extremely required for this setup.
“If the quality isn’t there, don’t release. Put quality first, which in the long run improves speed, consistency and capacity while reducing cost, delays and rework. Best of all, this improves customer satisfaction and trust.” — Also, quoted from the “Accelerate” book.
To change old habits it is required inserting a new routine to the team and I strongly believe that tiny tweaks can lead to big changes. Simply noticing that you have fallen back into repeating past patterns is already a sign of progress and self-awareness comes before the great leap forward.
Here’s why setting a baseline helps us to remind ourselves daily of the path we decided to follow when we answered the question “What’s our long term goal?”. The biggest intention here is to show the possibility of unlinking QA from the testing phase and creating a new connection between QA and the actual assurance of quality to the whole development cycle. Meaning: let’s assure together the quality of the delivery of better software with fewer flaws, that will be loved and trusted by the end-user while we also optimize the company’s resources.
The current situation of a development cycle in a common agile sprint:
Where the release stage usually means: “It’s live and done”. But wait, what about supporting what you have built? The goal:
Where the support stage is also often done only by the QAs in general, taking care of all the issues reported from production, among several other requests, whereas the whole team should be also responsible for it.
Our baseline
How to come up with a baseline is totally up to the team. It is important, however, that everyone takes part in it, rather than being something created and imposed by “the QA” alone.
In our experience of building it together, we use somewhat the “work agreements format” where we have sessions to openly discuss specific topics, such as agreeing on the definitions of ready and done, among other topics like, how can we add more and more quality in each step we take, etc. And yes, it’s a looong journey. As it’s said, “You can only change culture one person at a time”.
We started with what we already had in place, drawing the development cycle mentioned above, dividing it into the major phases and creating checklists for each of the phases thinking: what can we add to each of these stages to improve as a whole? To split the responsibility of quality into small action points that used to be in just one of the phases, to the entire process.
Here I will share some of our “tiny tweaks” a.k.a. checklists that have been helping us build quality into our routines of delivery and also create new habits.
Refining
The first contact of the team with the idea of what’s next to be built. It’s where it’s most important to start asking questions about the reason why this feature is needed in the first place, it helps giving clarity and context to everyone. Before, it used to be just a discussion over the idea and the process of transforming this idea into one or more “User Stories”. The action points for this phase were placed inside our “Definition of Ready”, which included the checklist below:
- Design Prototype (if applicable)
- Non-blocking and mapped external dependencies
- User Story template completed
- Non-functional evaluation e.g.: disponibility, performance (just because how many times have we learned about unavailability on production the hard way, right?)
Planning
Time to start thinking about the “what” and “how” parts. The “what” meaning what’s going to be included in the next sprint? According to backlog priorities and some other variables, and the “how” meaning, how technically are we going to build it? Getting deeper into the solution itself. This phase is often divided and known as/called “planning and tech planning” as well.
It’s also an important phase for us, as QA engineers, to add to the team’s routine to ask questions about the testability of proposed implementation once getting to the how part, discuss the required effort of testing, etc. And another question I’ve started enjoying asking too, (thanks to @Ulli our VP of QA and Infrastructure here at SumUp) is: “what alert do we want to get if it stops working?” As you can imagine, with time, this can bring us to the next level of monitoring our systems.
Developing
Consider going full hands-on and not making the responsibility of this phase all about the developers, otherwise, it would be controversial going into this direction.
It’s possible to start identifying the scenarios and writing automated tests for the new features that are being developed. What kind/level of automation? again, it should be a decision of the team together with the QA’s experience and considerations.
Build the test automation right from this moment, so that it can be used over and over again each sprint, saving a great deal of time and energy in subsequent testing (that moment when you have to go back to an old context just to manually guarantee it is still working).
You can always do “pair programming” sessions, maybe even call them “pair testing”, it doesn’t really matter as long as the team members feel that they can all collaborate towards the same goal. This way it is possible to make sure the proper testing is being done and the QAs won’t be feeling “late” and be desperately trying to catch up with testing after the feature is already done.
Another good question to answer during this stage is “When is the feature ready to go to a pre-production environment?” We placed this checklist inside of what we called “Transition Contract”, to help us stick with the new routine:
- Continuous Integration passing (with all tests, lint and code coverage) and PR approved — we use either Jenkins or Github Actions to have it all automated;
- Feature’s acceptance criteria must be met;
- Documentation updated (if applicable) — in our case it is Postman — we use it to document the APIs part.
These checklists are not written in stone, you can always go back to any of the stages and add or remove things found necessary. Build → Learn → Adjust. I am sure that by the time I finally publish this, the checklists we created will have changed already (it wouldn’t be good if they didn’t, actually).
Releasing
Right to the most exciting part, releasing to the users. Let’s make sure the process will happen as smooth as it can be.
Considering that what is being released has been validated in a pre-production environment, there should be no extra secrets here. These validations can be automated as well, when aiming for a continuous delivery process you can have, for example, the “sanity checks” or regression tests at this stage.
In our case, we are not quite there yet but we are looking towards it, in the long term “optimal” goal. Also, having a “post-release checklist” to keep consistency:
- Keep a close eye on the data to see if everything is being stored as expected;
- Check out the monitoring tools for new error alerts, etc.
We are humans, so we all miss things sometimes even after good tons of testing. There will be some edge cases here and there that we don’t anticipate, which is OK too, it’s impossible to test everything considering all the combinations and possibilities. Even if it were possible, it would be too expensive, we do not want that. Talking about possible issues, which leads us to the final stage: supporting.
Supporting
Much of our work goes beyond building new features. Here, we do believe that when we build something we gotta support it. And, most importantly, everyone is responsible as well.
We care about being as close as possible to our end users, not only to gather feedback but also to provide sensational support when they need. It’s up to the squads to assume full ownership over assigned issues and take priority into account. We have set Slack alerts to notify the team’s channel when a new production issue pops in. At this time, whoever is on duty or sees it first is responsible for reassessing the priority — if critical, address asap; otherwise, add to squad backlog.
Once the initial assessment is done, we get back to the original reporter and other involved parties to update them on the possible resolution/next steps so they can get back to the user as well, with more information regarding the problem. There’s a lot more involved in the production support, but I won’t bore you with that many details for now.
Team self-assessment
How to know if all these changes are improving or regressing the quality of the product at the end of the day? Like the PhD, LMSW researcher Brené Brown says “If you can’t measure it, it doesn’t exist”. I am not sure we would be making the whole development cycle more effective, as I said in the beginning, and to which degree only by experimenting these changes without having some concrete outcomes to rely on.
There are some metrics that we can add in different stages of the development cycle to give us the answer to the question above:
- How is the code coverage doing? Increasing, decreasing?;
- Are we having more or fewer bugs found or reported on production, as well as types of bugs, it matters (e.g. interface, functionality, etc?) it can show where the team needs to focus more on adding extra efforts on tests;
- Run cycles of user eNPS from a product/usability perspective;
- Know how many downtimes/incidents have been happening.
- Time spent in prod support/fixing bugs in production, etc.
And so on, these are just some examples of tools that can provide metrics to help the team to make decisions on the work they are doing.
Build Quality Assurance into the culture of your teams and you will work in an environment where everyone is accountable for and genuinely cares about it, in a culture where everyone takes part in it, not only QAs. To feel like we own quality, it helps us to do the right things.
I am Mariana, Local QA Chapter Lead at SumUp. If you’re interested in empowering small merchants all over the world, check out our current job postings.
Thanks for reading!
