Testing your Spring Boot service with Couchbase more efficiently

Written by Dominik Weidemann, Johannes Jasper, Daniel Moritz — July 3rd, 2018

Here at neXenio, we write tests for different layers of integration following the test pyramid (as explained by Martin Fowler, for example). In this article, we describe our additional testing layer that ensures the correct behavior of our backend services at every level from an incoming REST request down to the database. We also explain the technical issues we faced while automating these integration tests in our pipeline and how we solved them.

Technical context

Our Java based services (Spring Boot) use Couchbase (a NoSQL database) to store data. Docker is used to deploy and scale all our services independently. We chose to run Couchbase in a docker container during our tests. Required seed data for our tests are defined in plain JSON files. These files can be imported into the running database container directly via Couchbase default tools.

One aspect that differentiates Couchbase from other databases is its use of indices. As with other data sets, an index in Couchbase is a data structure that accelerates queries which require scanning the data set for any given constraint, such as checking fields against a WHERE clause. Unlike other databases, Couchbase requires these indices for any scan. If a query contains fields that do not match any index, it fails. By default, Couchbase creates a primary index that allows searching across all fields in any document. As a result, such primary indices are larger and slower than specific secondary indices and the Couchbase community explicitly recommends not using primary indices in production environments (see GSI for details).

Harmonizing indices on different stages

To prevent missing indices and long running queries, we decided to maintain the index definitions in the same repository as the service using it. This allows our engineers to adapt them immediately if changes are necessary and also allows easier testing to check if the indices fit with the data model. Furthermore, we enabled our services to directly create all required indices on their own at boot time.

Couchbase cleanup is too slow

To ensure a clean and consistent data set at any time, the testing database needs to be cleared after each test run.

There are two major approaches to clear a Couchbase database:

1. Flushing. Flushing a bucket (the Couchbase equivalent to a multi-table database) removes all documents. It is known to be an expensive and slow operation. Even for small or empty databases, flushing requires multiple seconds which makes it too costly as an operation to run after each test.
2. DELETE all elements. By running DELETE FROM myTestBucketwe could easily remove all elements, but that would require a primary index since all elements must be accessible.

Obviously the second approach is more feasible for rapid automated testing which is why we used a primary index for a long time on our test infrastructure.

While this approach worked well in general, it had a drawback: the testing infrastructure and the production environment ran different index configurations, the former using a primary index for fast deletion of all documents and the latter explicitly avoiding it. As mentioned before, queries that do not match any index will fail in Couchbase. If an engineer forgets to update an index after adapting a query or data model, it could result in a bug. Having a primary index in testing however acted as a fallback for missing secondary indices. This mismatch led to services failing in production, even though they were tested extensively and passed all quality gates. We encountered this problem several times and therefore abandoned the primary index in our testing environment and replaced it with more concise secondary indices only.

This, on the other hand, put us back to square one with the issue of clearing the database after each integration test.

Solution: Tracking Database Access using Aspects

We solved this problem using aspect-oriented programming (AOP) which allows us to track method calls without altering production code. We now use AOP to track writes to the database and store the used document IDs. After each test, all documents written during that test are deleted explicitly by their IDs. It is noteworthy that access by ID in Couchbase does not require a database scan and thus does not require an index. This allows a very fast cleaning of the database and also enables us to test our services using only the indices they themselves defined.

The following section demonstrates our approach using a very simple code example that can be found on Github. In order to run the tests, we provide a docker image with a pre-configured Couchbase cluster.

docker run -d --name cb_demo -p 8091-8094:8091-8094 -p 11210-11211:11210-11211 nexenio/couchbase_demo

Our demo use case has a single model

The @Id and @Document annotations stem from spring-data-couchbase, i.e. the respective spring boot starter. The corresponding user repository simply extends CouchbaseRepository, thus providing access to simple methods such as findAll, findById, save, count, etc. Additionally, it defines a method findByName which relies on a secondary index on the field name. This index is already provided in the docker image.

The goal of this demo is to test basic functionalities of the UserRepository.

Each test generates a new User with a constant ID and stores it. The first test asserts that it can be retrieved by ID. The second test tries to retrieve the user by name which in turn tests if all necessary indices are created.

The crucial parts here are the insert operations during the setup of each test. If we run both tests together or run any test again, this will fail since a user with the same ID already exists.

Note, however, that the test extends IntegrationTestSuite which auto-wires our DocumentDeletionAspect. This aspect maintains a set of document IDs and keeps track of every document inserted, upserted, or replaced on a bucket.

After each test, the IntegrationTestSuite clears all recorded documents

This approach fulfils our requirements and allows fast integration testing only by using the same indices as in production environment.

Side note: since the DocumentDeletionAspect only records and clears documents that are created during the tests themselves, some documents might not be removed correctly if a test fails or throws an unexpected exception. We added an additional InitialPurgeRunner to ensure a clean database at the initial beginning of a test. The InitialPurgeRunner disregards any document that was stored in the database beforehand. It is executed once the spring context is available, creates a temporary primary index to delete all remaining documents and immediately drops the index again once done.

Conclusion

Testing an application with real-world configurations i.e. using production-grade indices helped us find database-related problems earlier in our development cycle. This posed new challenges especially in maintaining fast test feedback. The presented solution demonstrates how aspect-oriented programming can help to keep track of your application’s state and alter it, independent of the used technology stack.