Faster feedback with precompiled smoke tests in CI
Background
In my team at John Lewis and Partners, we have many microservice applications deployed in the public cloud. When we commit changes to our applications there are certain types of automated, lightweight tests that we want to be run in a deployed environment to give extra confidence that our changes are safe for production.
We like writing our tests in Kotlin as they are expressive, easy to change and can can be as simple or complex as required. This also means that we don’t have to introduce another language into our projects. The downside has been that we have traditionally run these tests from Gradle, which introduces overhead and means that the tests have to be rebuilt for each pipeline run.
For us to be able to achieve something approaching continuous delivery, our pipelines need to be fast and reliable.
This article describes how we have written our tests and CI pipelines in a way that allows us to write them in Kotlin and also benefit from very fast test execution.
Our CI tool is Gitlab, but the principles will apply to other CI tools.
Project and pipeline setup
We’ll set up two projects in Gitlab (so each can have a separate pipeline). The first will have the application we are testing and the second contain the tests we run against the deployed environment. Example repositories are linked at the bottom of the article.
Project containing tests
This can be set up as a standard Gradle project, with JUnit tests living in a package under the src/test/kotlin directory.
We’re going to build a JAR of our test classes (and their dependencies), so will need to configure Gradle to do this. We also set the JAR manifest to run our test-running code when it is executed:
We then need to include code to run the tests. This makes use of the junit-platform-launcher, which can discover tests in a package and run them. The code snippet below supports defining separate test suites in different packages and being able to select which to run by passing in an argument. This code will write test output using java.util.logging and will write a test summary at the end. If any tests have failed the failures will be printed out and the process will end with an exit code of 1. The test output is functional but not beautiful and with extra effort this could be improved.
The tests can be run locally by building the JAR, then running it:
./gradlew shadowJar
java -jar build/libs/standalone-tests.jarThe Gitlab pipeline will have two stages, one to build the JAR and store it in the project Gitlab package registry, the other to run the tests:
Here we save the JAR we have just built in the project’s generic Gitlab Package Registory. We do this by using curl to access the Gitlab API. Ideally we would version this using ${CI_COMMIT_SHORT_SHA}, but there is currently no housekeeping available on generic packages within Gitlab, so until this feature becomes available we will just overwrite the file in the existing version of the package.
We have also been able to specify a rule so that the build stage is not triggered when called from another pipeline (which is the normal case when we want to run the tests.)
Project that will run the tests
Usually for our scenario, we would want to trigger these tests from the project which contains the application that is being tested. This can easily be achieved by configuring the Gitlab CI pipeline to trigger the project containing the tests.
Conclusion
As you can see with this simple example, the overheads and time of starting tests can be reduced significantly by not having to execute a build tool or download dependencies when the tests have not been changed.
At the John Lewis Partnership we value the creativity of our engineers to discover innovative solutions. We craft the future of two of Britain’s best loved brands (John Lewis & Waitrose).
We are currently recruiting across a range of software engineering specialisms. If you like what you have read and want to learn how to join us, take the first steps here.
