Parallel database integration test on Go application
Integration test is one of the test layer in testing pyramid. Usually it will take a longer time compare to unit test because we don’t mock anything. In order to optimize the time spend on the test, one of the approach is to run the test parallelly. In this blog post, I will specifically share about database integration test using postgresql as the database.
Ideally, every test function has to be independent, so they won’t affect each other. In other words, every test function has their own state. This is a good sign for parallel test. In order to have separate data between each test function, I created a function to create a schema and load the test data everytime a test function start and drop the schema after completing the test. Every schema created contains hash in their name to prevent collision.
Helper function
Let’s start with helper function for printing error in testing. Checkout Ben Johnson’s helper functions, it saved me some lines of code and made my error message looks better and more verbose.
Test data
In order to run database integration test, dummy data must be provided. Go test has good support for loading test data from files. Firstly, go build ignores directory named “testdata”. Secondly, when we run “go test”, it sets current directory as the package directory. This allows us to use relative path to “testdata” directory to load the dummy data.
Creating a test database connection
Calling “CreateTestDatabase” will create a connection to the test database and create a new schema for testing. This function returns the database connection, schema name, and cleanup function for dropping the schema. For testing purpose, it is better to fail the test if error occurred instead of returning the error to the caller. (Note: Returning cleanup function pattern is inspired from the Mitchell Hashimoto — Advanced Testing with Go talk).
Loading test data
I used “.sql” files for providing the test data. 1 sql file contains data for 1 table. The sql script includes table creation and data insertion command. All sql files were kept in a folder named “testdata”. Here is the sample of the sql file.
Here is the tricky part. Because each function runs on their own unique schema, we can’t directly execute the query inside those sql files. We have to add the schema before every table name in order to create table and insert data to the right schema. For example, “CREATE TABLE book …” should be changed into “CREATE TABLE uniqueschema.book …” and “INSERT INTO book …” should be changed into “INSERT INTO uniqueschema.book …”. I used regexp for modifying the query. Here is the code for loading the test data.
Creating the test
Before running every test, it will create a test database with unique schema and defer the cleanup function that will drop the schema. Query has be injected with unique schema for testing purpose. The most important part of the code implementation is the database connection has to be configurable in order to change the real database connection to a test database connection. Don’t forget to add “t.Parallel()” at the beginning of the function to mark the test to run in parallel. Here is the complete code.
Note: For “TestGetBooks”, I expect that there are 2 books returns by the query because I provide 2 sample data in “testdata/book.sql” although there is an insertion test above it. If we don’t separate the schema between both test, the “TestGetBooks” will fail because now there are 3 data inside the table, 2 from sample data, 1 from the insertion test above. This is the advantage of having separated schema between each test because they won’t affect each other.
I have created a sample project in my github. You can clone it and run the test to see the result.
Conclusion
For my project, this approach reduces 40–50% of the time spend after running integration test parallelly. Another advantage of running the test parallelly is that we can eliminate some bugs that might be happened when the app serves several concurrent action.
Happy testing 🎉 🎉 🎉
