Testing Automation — Making It Right #2: Test Types and Integration Tests
This is the second part of the series.
#1 — Theory
#3 — Test Machine Learning Based Applications
There are lots of terms for testing types: e2e (end to end), unit, integration, sanity, smoke, component, contract…
In order to keep it simple, and avoid fighting on syntax, I would list three of them, and sketch their boundries:
- Unit test — testing an algorithm, a function. It doesn’t test a flow, integration with any 3rd technology (like database) or any other services.
- Integration test — testing a flow: http request, processing of item, complete single cli command.
- End to end (e2e) test — testing the whole system: all components up and running, UI and server, full sequence of cli commands that represent a real world scenario, etc.
Unit test:
Pros: Fast, code branches (if… else…) can be easily tested
Cons: In web applications, you need to work hard to mock services. Therefore, writing tests become hard and slow task, with lots of maintenance. You are testing the internal of the codes, not integration points. By nature, integration points have less changes than method signatures.
The most problematic part with unit tests is the confidence in the code you deliver. You don’t know whether you can ship your code even if tests pass successfully. For example, an algorithm can relay on a DB query. The algorithm is perfect, but the query is incorrect. In such case, you have a bug in the code you just wrote, but you cannot discover it using the unit tests.
Integration test:
Before analyzing this type of test, which is my favorite, we need to better define it: What is “integration”? What does it include?
Different engineers will tell you different things. Here is my definition for integration test:
Testing a flow. We test a single component (usually, an OS process). If needed, other components / processes can run as well, assuming they meet the following preconditions:
a. We can assume they work.
b. They return consistent results.
For example:
- DB can be run as part of the integration test, as we can assume it works, and for a certain input — it returns consistent output.
- Integration with Google search API must be mocked, as it doesn’t return a consistent output for a certain input.
- If your company develops two microservices, and one calls another — the other microservice must be mocked, as we cannot assume it works.
The idea behind these rules is to get green light whenever you didn’t break the code, and red light when you are the one who broke the code. Dealing with endless “why it doesn’t work” questions makes the processing of writing and running tests very cluttered.
That’s why Google search API must be mocked: we cannot guarantee the test will pass. If we test several services we develop at once — one service can break the test, and it will be hard to find why it is broken, or who broke it. It won’t always be related to the code you just wrote.
Now, the pros of integration tests are — you have pretty good confidence in the flow; you are capable of developing each component separately, which makes the coding more efficient; integration between teams become easier.
Cons: Inability to test nuances in the code on the one hand, and you don’t know that your system as a whole works on the other hand.
E2E test:
Sounds like the dream of every manager: In end-to-end tests, you are spinning up the whole system, and start running complete scenarios. Happy happy joy joy!
In practice, its a VERY expensive solution for complex systems: —
(a) It takes time to run all tests. That means you cannot run all of them before deploying critical fix.
(b) Since the developer didn’t develop the whole system, it is not the developer who write the tests. Therefore, there are test engineers that are responsible for the tests, and that makes the feedback process on the code quality much longer. See details about the feedback loop in the first blog post.
(c) Much more people are involved in testing, reviewing and investigating bugs.
(d) The resources to run the tests are often very expensive.
(e) Environments where the tests run on become buttlenecks, and from time to time hard to maintain.
Therefore, my recommendations for testing are:
- Use unittest for complex algorithm. Not for web request flows.
- Use integration tests as much as possible, especially for web application. Make sure every developer can run them at any time, from their IDE; Make sure the “green light” for shipping doesn’t depend on other services to work.
- Use end to end testing only for basic sanity and extremely important flows (e.g. signup)
You can read this great blog post, describing google testing approach. I completely agree about the e2e tests part. As oppose to them, I would take unit/integration test approach according to the nature of the tested element, and not according to the pyramid rule of thumb.
