How to Find Balance in Software Testing
Finding balance between unit testing, integration testing, and end-to-end testing
In times of chaos, we all yearn to find some balance. What about balance in software development? We want reliably tested software to go to production. Errors and bugs need to be found before deployment, and tests need to be automated.
This is Tom’s story: His company got bigger and bigger clients. At the same time, the team of software developers got smaller and smaller. Super annoying. Tom’s team was frustrated with faster product releases and couldn’t keep up with the quality of the product.
The company then decided to cut expenses and cut off the automation guys. They replaced them with a cheap offshore team. The new team wanted to run automation tests unattended overnight on a live system.
Tom’s team politely said no. The offshore team proceeded and started testing overnight. To make things even worse, this was happening during the coronavirus outbreak.
Imagine what Tom, who had just successfully transitioned his team to work remotely, will now wake up too. The system was running unattended overnight. He did not wake up when alerts came. He was just so tired from his kids running around all day. Now the hell broke loose.
He needs to repair the damage, revert to stability, and calm down the kids at the same time. You don’t want to be Tom right now.
No, no, no … good testing practice and and an experienced testing team aren’t to be cut off.
We need to get control over coronavirus. But we also need to get control over the quality of our software.
Instead of having millions of manual software testers, development teams have moved toward automating most of their testing efforts.
Automated tests go hand in hand with agile development practices, continuous delivery, and DevOps culture. With effective software testing in place, developers can move fast and with confidence.
But what is the most optimum balance between unit, integration, and e2e tests? Can we find one unified answer to the question?
Before answering this burning question, let’s remind ourselves what effective testing is:
- In an effective testing practice, the goal is to find errors — not to write tests
- A successful test case is one that detects a yet undiscovered error
So what would be the best answer to our question?
How to Find a Balance Between Unit, Integration, and End-to-End Tests
Finding the right balance to how many unit, integration, and end-to-end (e2e) tests to write is not easy. Every team and application is different and requires a different ratio.
Most projects benefit from having a balanced mix of various automated tests to capture different types of errors. The exact composition depends on the nature of the project.
The testing pyramid was introduced over a decade ago, and it’s a great and relevant model.
The pyramid says tests on the lower levels are cheaper to write and maintain — and quicker to run. It suggests three testing types: unit test, service tests, and UI tests. Tests on the upper levels are more expensive to write and maintain — and slower to run. Therefore, you should have lots of unit tests, some service tests, and very few UI tests.
In the last 10 years, we’ve seen many changes — from microservices to cloud to serverless to Kubernetes. Would it be stupid to say the testing pyramid model isn’t the best for all types of applications? Would it be crazy to say we should embrace new testing techniques?
The testing pyramid, TDD, and unit tests are still powerful techniques and are probably the best match for many applications.
But like in everything, the exception proves the rule.
Let’s take a look at an IoT application that fires millions of events into a message bus like Kafka/MQTT. Events are then stored into some data warehouse and queried by some analytics UI. This app integrates services and has almost no logic. Do you think it’d still be wise to insist on writing the majority of our unit tests?
What about in apps like chatbots, Alexa Skills, or crypto apps? In all of these apps, the testing pyramid isn’t the best match.
With front-end frameworks like React, Angular, and Vue.js, it becomes apparent that UI tests don’t have to be on the highest level of your pyramid. You can unit test your UI in all of these frameworks.
If you want to keep pace, you’ll have to look into ways to deliver your software faster without sacrificing its quality.
The testing pyramid is super simple and can be a guide for many projects. Two things are worth remembering from Cohn’s original test pyramid:
- Write tests with different granularity.
- The more high level you get, the fewer tests you should have.
And don’t become too attached to the names of the individual layers in Cohn’s test pyramid.
Cohn also talks about the observation that a lot of developers completely ignore the service layer. That observation is still true.
So what’s the answer to our question: How do you find a balance between unit, integration, and e2e tests? It depends. Let’s discuss this further.
Three Steps for Better Software Testing on Your Team
- Learn more about software testing, continuous delivery, and DevOps. Here’s where you can get started:
2. A good way to identify the best strategy with your team is to analyze and reflect on questions like:
- Which tests within your codebase have made a big impact?
- Which ones saved you from having your application crash in production?
Take a moment and answer these two questions with your team.
3. Google for:
- Test automation
- Agile and DevOps
- Codeless automation testing
- Test automation
- Agile testing
- DevOps testing
- Artificial intelligence and machine learning in testing
Have a great, balanced day.