CI/CD: Catch Errors As Early as Possible in Various Test Stages in Our Deployment Pipelines
Integrated test stages are important for every Deployment Pipeline
Automated tests should be used by every developer in our daily work. Integrating them in your work creates a fast feedback loop, finding problems or bugs early on and fixing them as soon as possible if there are no fix constraints (release-date, end of iteration, resources, etc).
Deployment Pipelines are good places for running automated tests every time a new change is checked into our version control systems. Deployment Pipelines build and test our applications automatically in production-like environments — so we can find any build, test or integration error as early as possible during the deployment process.
The goal of the deployment pipeline is to provide everyone in the value stream, especially developers, the fastest possible feedback that a change has taken us out of a deployable state. This could be a change to our code, to any if our environments, to our automated tests, or even to the deployment pipeline infrastructure (e.g., Jenkins configuration setting). [DevOps Handbook]
Different Test Stages in Our Deployment Pipeline
In Deployment Pipelines we can integrate different test stages, like fast-running automated tests, or resource-heavy automated tests for different times, for example, instant-running automated tests after code checking or nightly tests.
First Stage (Unit Tests)
The first stage is the one after a developer checks in new code changes. This is also called Commit Stage which is triggered automatically. Here, the application is built, an artifact is created, static code analysis is running, duplication and test coverage get checked. Also, local Unit Tests are running — these are the fastest tests in a development environment.
Unit Tests test single methods, classes, or function in isolation. These tests “stub out” databases or other external dependencies with mocks, static code, or classes created with Dependency Inversion.
This kind of testing is a fast-running automated test and should be completed in less than ten minutes to enable a fast feedback loop for developers.
Second Stage (Acceptance Tests / Integration Tests)
After the first Pipeline finished successfully the second stage gets triggered automatically. This stage is also called Acceptance Stage. Here the artifact from the first stage gets deployed in a production-like environment and runs the automated acceptance and/or integration tests — this reduces variance that can avoid downstream errors which are difficult to diagnose (e.g., different compilers, flags, library versions, configuration, operation systems).
Acceptance Tests test the application as a whole to verify the criteria for a user story, the correctness of an API, or other cases.
Integration Tests test the correctness of interaction with other applications or services that pass the previous stages, too.
Both test strategies are slow, resource-intense and costly. This stage is completed in a couple of hours. So, we want to minimize the amount of these tests and find as many defects as possible during unit tests. To make this possible we try to cover as many bugs found in this phase with unit tests in the First Stage.
Another option would be to split both test strategies into different Pipeline Stages.
Other Stages and Manual Tests
If the second stage is successful we can add other stages as well, e.g., automated usability tests, performance tests, exploratory tests, capacity tests.
After all these stages have also successfully completed, the artifact from stage one will deployed into a test environment automatically and a developer or tester can run their manual tests. Found bugs, as in stage two, should be fixed and covered as fast unit tests.
If the manual tests are successful, our application is in a releasable state and we have a releasable artifact.
At the end of each development interval, we must have integrated, tested working and potentially shippable code, demonstrated in a production-like environment, created from trunk using a on-click process, and validated with automated tests. [DevOps Handbook]
Impossibility To Integrate Tests Into the Application
Sometimes it is almost impossible to integrate tests into an application, for example it could be too difficult or too expensive to write or to maintain. In this case our application architecture or system architecture is too tightly coupled and we need to add strong module boundaries. With a loosely coupled system modules can be tested independently and isolated.
Working Effectively with Legacy Code by Michael C. Feathers is a great collection of principles to write testable code and architecture:
Another good resource to create testable code is Refactoring by Martin Fowler:
Conclusion
It is important to have a comprehensive and reliable set of automated tests in our Deployment Pipelines. This way, we validate that the application is in a deployable state. And if this is not the case, the Pipeline guarantees a fast feedback loop for us, the developers.
Thanks for reading! Follow me on Medium, or Twitter, or subscribe here on Medium to read more about DevOps, Agile & Development Principles, Angular, and other useful stuff. Happy Coding! :)
