One guideline to test them all — Part 4
So, what are the test types we need?
To be able to answer this question, we need to ask another question first; “What are the promises that our product makes?” which brings even more questions:
- Which platforms are officially supported?
- What are the minimum system requirements?
- Which level of WCAG will be supported?
- Does it guarantee to meet any performance expectations? (Eg: startup time, uptime, handling a certain number of requests, or completing a task under certain time, etc.)
- Does it guarantee to backup/restore any kind of state?
- Does it guarantee to secure/protect any type of data?
- Does it guarantee backward compatibility?
- Does it guarantee a certain level of security?
- Can it be installed/uninstalled/updated?
- Does it need certain permissions to work?
- Does it support i18n and l10n?
These are just some of the questions I could come up with at the moment. You can increase them.
Each type of promise requires a different testing strategy. So, as you can easily see, it’s not straightforward to answer what kind of test types we need. On the other hand, we don’t have to answer all those questions or don’t need to meet any expectations that we didn’t promise. The only thing that matters is what we officially promised to deliver.
Remember:
Even though it’s not straightforward to determine which test types we need, there are 4 types of testing that are common almost in every project.
- Static analysis
- Unit tests
- Integration tests
- Functional tests
Static analysis allows us to maintain code quality of the product. Unit, integration, functional tests prevent releasing unplanned/unintentional breaking changes and everyone would like to avoid that at all costs. That’s why these 4 types are happened to be the most common testing types.
Static analysis
I’m currently also working on another article series about static analysis. So, I won’t be going into too much detail here. For now, I’d only like to share a story with you.
Once upon a time, there was a great doctor who can cure almost any disease. He was so good at what he was doing, his name reached beyond his country.
One day, a traveler hears his name and wants to meet him. When he arrives at the doctor’s village, he learns that the doctor has two older brothers who were also doctors but the traveler never heard of them.
He gets even more curious, and asks the doctor, “You are the youngest of your siblings, yet you are the best of them. How did you surpass your brothers like this?”.
The young doctor smiles calmly, “Surpass them? No, both of them are actually better doctors than me. I only cure the common problems that everybody can face in their daily life. So, I’m just the most visited and most well-known. My older brother saves patients who has the rarest diseases. Therefore, very handful of people know his skills. And my oldest brother prevents people to get sick in the first place. Thus, nobody knows his name.
I think static analysis tools resemble the oldest brother in this story. They are actually solving problems before they happen by simply assisting you to follow best coding practices and keep a consistent coding conventions across your products, yet we don’t invest enough time in static analysis. That’s why I’d suggest you give a bit more importance to this step if you were not doing it until now. You can start from here: Awesome Eslint, SonarQube and Code Climate.
Functional Tests
I’d like to follow an unusual order and start talking about functional tests before integration and unit tests. Because I think they are the only test type that can give you the ultimate confidence to release your product.
Before going any further let’s question why we need functional tests and assume that you only wrote unit and integration tests to test your product. Let’s ask the magic question: Now, what could go wrong?
Usually, unit and integration tests are executed on a single platform. Thus, we don’t know what could happen with other supported platforms. For instance, something that works on Chrome may not work on Firefox or a certain NodeJS API that we use with NodeJS v8.x might be deprecated with NodeJS v12.x.
So in other words, functional tests are integration tests to check if the product is behaving on target platforms as expected.
Ideally, with functional tests, each product must be tested on a production-like environment on all officially supported platforms. That means if the product is a web application we need to test it on all supported browsers and if it’s a mobile application, maybe on all devices we support. If it’s a backend application or I/O connector, we need to test it with all supported node versions and operating systems.
Also when writing functional tests, the product should be used as it would be used in production. That means the product should not have monkey-patching, stubbing or mocking. Any test related behavior could be and should be set by environment variables and ideally, that should only change minor things like the logging level.
Functional tests must cover all acceptance criteria of the product. That means everything that we promised to deliver must be tested with functional tests.
Functional tests must be written by only using the product’s public interface. If a functional test relies on a certain global or internal state, ideally product should have a public API that allows us to configure the product to reach the desired state.
However, if that’s not the case, then we can have utility functions to create the desired global state (creating database records, config or data files, setting date, etc.). This way, when we run the product, the desired internal state can be created automatically by using the global state and the inputs coming from the public interface.
Functional tests are (not always but often) scenario-based. That means, if your product has a certain action that can only be invoked after another action, you may need to test each step separately.
So, to sum up, let’s assume we followed all the advice above and wrote functional tests that cover all possible scenarios that can happen with our product on all supported platforms. Now, what could go wrong?
You would wait a lot and it would cost a lot. In terms of publishing a bug-free product, nothing could go wrong with this approach. You would always have a 100% confidence since you have covered literally everything.
However, in order to run these tests, you would need several servers and dependencies (database, mail servers, web services etc) which will be costly and since tests are stateful, you’ll need to wait until all relevant steps are executed. That time also has a cost.
During the development phase, we need a fast and cheap feedback mechanism in order to find out potential problems as fast as possible.
Yes, now the circle is complete. I’m talking about unit and integration tests.
However, these are loosey-goosey terms and nobody has a standard approach to write them. With this article series, my main goal is to solve this ambiguity.
I have a dream, a dream where we can separate unit tests and integration tests so distinctly and clearly, even a junior developer or a new developer in the team could tell what needs to be tested and how it should be tested. A dream where unit and integration tests, together, can replace functional tests.
