Testing complex business flows: From cones to pyramids

By Nikos Sideris, Senior Software Engineer

About 2 years ago, we began a new project at Signal that would fairly distribute and report earnings among vessels in our Pool. From the start of this project, testing had a significant role in the software development circle. In this blog post, I talk about the challenges which we faced while testing complex business flows along with the old and the new testing pipeline.

Old Testing Pipeline

Our testing pipeline consisted of unit tests and integration tests. These integration tests made sure that all the dependencies of the service would be validated by running earning distributions and validating the end results. The target was to write as many integration tests as we could to cover many cases. We built an integration test project where external dependencies like databases were put on a docker container. We also created a script that would update the docker container databases with the latest schema/data in order to be in sync. Restful web services were mocked using WireMock, an API mocking tool for fast, robust and comprehensive testing.

We configured our continuous integration pipeline so that every pull request would run automatically the unit tests without giving the option to the issuer of this request to complete it until all unit tests passed. But the integration tests were slow and were put to run after the pull request had been merged on the main develop branch. We had managed to have continuous integration but without continuous delivery. Following a successful run of integration suite, we had to manually deploy to production and perform any necessary manual testing.

After a while, we noticed that there were some problems with the productivity of the team using the above approach including:

• The integration suite for around 20 tests was taking about 30 minutes to complete.
• In the case of failing tests, it was difficult to identify what merged pull request caused the failure as there were cases where multiple pull requests merged to the main branch prior to the initialization of the suite.
• No early feedback.
• Slow development of new integration tests. For example, when we had a change on the table schema, we had to run docker on our local machine and then run some scripts that would synchronize the docker container and finally upload the container image to the cloud provider.
• Suite could not scale.

The new Testing Pipeline

Goals

Following are notes taken from our architecture decision records (ADR). Along with the redesign of the new testing pipeline, we decided to start keeping an ADR for our software design choices that are architecturally significant.

We need to create a new testing pipeline that satisfies the below criteria:

• It should follow industry test best practices: Test Pyramid (70%-20%-10%). Write tests with different granularities. The more high-level you get, the fewer tests you should have.
• It should be scalable and maintainable.
• Max. time of build, should not exceed 10 minutes.
• Tests should be able to run in parallel.
• It should run different levels of tests at different stages. Lower level tests should be run at earlier stages. Some slow end-to-end tests can be run on a nightly build.
• Ensure fast feedback to increase development productivity
• Think Smaller, not larger. Create focused functional tests.
• If a higher-level test spots an error and there’s no lower-level test failing, you need to write a lower-level test. Push the tests as far down the test pyramid as you can.
• Write as few end-to-end tests as possible

Our Test Pyramid

Level 1

On this level, we have the fast running tests that will produce very fast feedback to the developer. Developers should be able to run these tests in the local environment to increase productivity.

Unit Tests

They consist of testing individual methods and functions of the classes, components or modules used by our software.

Smoke Tests

Basic tests that check basic functionality of the application. They are meant to be quick to execute, and their goal is to give you the assurance that the major features of your system are working as expected. Like Configuration is correct, DI is correctly setup, Automapper is correctly configured.

Level 2

Here we have tests that will require more time than Level 1 tests. However, time on this level should not be significantly slower than Level 1. Developers should be able to run these tests in the local environment. These tests should be created to run in parallel.

Functional Tests

These tests focus on the business requirements of an application. They only verify the output of an action and do not check the intermediate states of the system when performing that action. The purpose of these tests is to validate business logic as a whole without being affected by external dependencies. Thus, all of its dependencies should be mocked. From the start of this project, we had decided to use onion architecture together with domain driven design that helped us to mock our hard dependencies like APIs and databases from our domain layer and focus on our tests on specific business requirements.

Integration Tests

We verify that different modules or services used by our application work well together. No business logic should be tested, only interaction with external dependencies. By saying interaction with external dependencies, we did not mean to call an actual dependency but rather to mock it. What we actually test is that our mappings and query/command logic between these external dependencies work properly.

A logical question now is what is the difference between Functional and Integration tests. In functional tests we are testing the business logic. On the other hand, on the integration tests we don’t because business logic does not exist on our data access layer. What we actually mock is dependencies like SQL or Mongo in contrast to mocking interfaces that happen on functional tests. In order to mock SQL databases, we use EF Core SQLite. We deliberately avoided using in-memory provider by EF Core as we wanted something that will simulate as closely the behavior of a SQL database and in-memory database is not capable of that. For the Mongo database, we used Mongo2Go that is a wrapper around MongoDb libraries and very easy to set up. For APIs, we used WireMock.

Contract Tests

Contract tests are tests at the boundary of an external service verifying that it meets the contract expected by a consuming service. We are using Pact for that.

Level 3

This is the slowest running tests that will provide feedback on the developer late. For this reason, these tests should not be run on the pre-merge build phase. All application critical logic should already be tested on the below levels (Level 1 — Level 2). Level 3 is under development at the time that I write this blog post.

End-to-End Tests

End-to-end testing replicates a user’s behavior with the software in a complete application production like environment. We are thinking of having canary releases here deploying the latest binaries to the production environment and to apply the end-to-end tests in a continuous integration way. If all are successful, then, we will start redirecting the traffic.

Performance Tests

Performance testing checks the behaviors of the system when it is under significant load. It can help you understand if new changes are going to degrade your system. We have used JMeter for doing some load testing and checking the performance of the application under pressure and we plan to use it here as well.

CI Pipeline

Test Coverage Reports

For every build on the development branch, we are also generating test coverage reports using coverlet to generate the test coverage and then ReportGenerator to generate the report. All integrated with our build management and continuous integration server.

Key achievements

The new test pipeline we believe is more effective than the old one. We still have work to do in order to also have continuous delivery finalizing Level 3 of the pyramid. We are using the new design for around 5 months now and below are some of the key takeaways.

Developer Early Feedback and better productivity

Level 1 and Level 2 can run on the private development of each developer without the need to set up anything else before. This is very important for us as we get early feedback on our changes without the need to trigger a test suite on another machine. It also gives us the ability to have the results even before using the pull request. As we saw before, 90% of our test suite is built on top of these levels.

It scales

Parallelization and abstracting the hard dependencies made a big impact on the time we needed for a complete run of the testing suite and made sure that adding new tests will not heavily impact this.

It is more maintainable

Removing the hard dependencies helped us not to have flaky tests. Also, we combined object mother with fluent builders in order to reduce code duplication and increase the development time.

###

For more tech related articles, click here to follow Signal’s Tech Blog on Medium.

Want to join the Signal team? View our Job Openings here.