Quality Assurance and Software Delivery Processes in Frontend Engineering
How we improved SDLC of web apps at Azimo
This is the first in a series of blog posts in which we outline our Quality Assurance and software development processes for frontend engineering at Azimo.
The second blog post is already published here.
There is a big difference between Quality Control and Quality Assurance. QC focuses mainly on fulfilling quality requirements. It makes sure that the end product is working as expected and is free of bugs.
Quality Assurance is much more than that. By focusing on the entire process, it builds engineers’ confidence over quality, improves testing and development efficiency, and by that it makes us deliver faster in smaller batches. The end goals of QA are better SDLC processes, team scalability, and agility. Quality and bugs-free products are (only) derivatives of that.
As it was stated in the book “Accelerate: The Science of Lean Software and DevOps” and related “State of DevOps” reports:
“Speed and stability are outcomes that enable each other.”
With all that in mind, we invest a lot of time building QA culture in the Frontend Engineering team at Azimo. This series of blog posts sums up our achievements and key changes to the process which we made within last years.
The starting point
A few years ago, our web apps were delivered with these principles:
- Weekly release cycle — each of our applications was released once per week,
- Before a release, all features were merged into develop branch, which then we tested, fixed, and pushed to production,
- Each release was fully tested by our QA engineer, who gave the team the green light 🚦 to go live,
- We didn’t release on Fridays to sleep well over weekends.
Today this process seems to be far from perfect. But after a few years in a chaotic start-up-like environment, it put us in very good stead. We had a regular release cycle, which improved engineering work’s predictability. The weekly release cycle forced us to deliver things in smaller chunks which were easier to test and generated fewer bugs. And even though we faced some mistakes over time, we could identify and fix them much faster. A quick and stable release cycle also helped us parallel our work, so our frontend engineers could simultaneously work on a different piece of our product.
But this process had some severe constraints. With the rule ‘no releases on Friday,’ development had to be finished on Wednesday, with testing and releasing scheduled for Thursday. If we found any bugs, either fixes were done in a rush, or releases were delayed to the following week.
Test automations for released features were left in the backlog due to the lack of time. Our testing process was primarily concentrated on manual checks. Yes, this is quite contradictory — we couldn’t build automation that would increase our velocity. We didn’t have time to save time 🤷♀️. So why was that?
With a fixed release cycle and single development branch, our testing&release queue started growing. In peak times, we had many features merged into develop (not mentioning others, waiting to be merged). Even though the weekly release cycle was convenient for software engineers and resulted in small changes, things added up into big release candidates. Our QA engineering team very often needed to test multiple features not even related to each other.
The results?
- Our feature branches spent more than 40 hours in a testing phase (median for “in testing” status). In worst cases, this process could take hundreds of hours.
- Our testing&release queue had a few dozen features that sat in feature branches, sometimes even for a few weeks.
- The number of changes that omitted the “in testing” phase started growing in an uncontrolled way (even though many of them were “1-liner changes”, the risk of production bugs was growing)
Our QA and release process today
Over last years we improved our delivery processes and ended up with these principles:
- Development and testing are done on feature branches — our approach is close to trunk-based development, with short-living feature branches that are fully tested and then deployed directly to production.
- We can release features almost immediately after the development is done — no more one-week release cycle, no more a few dozen hours spent under “in testing.”
- Most releases don’t need to be checked by QA engineers — either manually or by running automated tests. Thanks to the QA culture approach, each engineer is equally responsible for the quality of our product.
Today, even though the team’s headcount didn’t change, we can release our app multiple times a day. No more testing&release queue. No more deploys to production without code being tested. Since all engineers took responsibility for the quality of our product, we experienced fewer bugs. And whenever we discover any issue during the testing phase, it doesn’t block other developers from releasing their changes (because we don’t have a single develop branch where all of the features are merged before the final testing&releasing phases).
According to our data, we release 2–3 times more frequently than two years ago.
Our web apps are also more stable, with a crash-free ratio above 99.9% for most of the time.
All of these achievements allow us to iterate over our product faster and unlock initiatives as the automatic dependencies updates described earlier on our blog.
What we’ve done
Changes in the delivery process and transformation from Quality Control to Quality Assurance don’t happen overnight. Here is the list of initiatives and implementations that helped us with shifting from scheduled, weekly releases to continuous delivery:
- Focus on the testing pyramid, improving unit tests coverage
- Better functional and end-to-end testing automation:
- — Cross-browser testing,
- — Visual regression testing,
- — End-to-end tests for edge case scenarios
- Focus on flakiness reduction
- Automation of complex testing flows (like email flows tested with Mailosaur)
In the next blog post, we will look closer at each of these.
Stay tuned!
Acknowledgments
All of our achievements described in this series wouldn’t be possible without the hard work of the entire Web engineering team. The biggest credits go to Agne Baranauskaite, QA Engineer, and Artur Kania, Software Development Manager. Together with the web team, they built the foundations for changes described in this series.
Towards financial services available to all
We’re working throughout the company to create faster, cheaper, and more available financial services all over the world, and here are some of the techniques that we’re utilizing. There’s still a long way ahead of us, and if you’d like to be part of that journey, check out our careers page.
