Image by Erin Rhodes

Testing Rails Multiple Database Connections with CircleCI

If you have yet to read my colleague April’s excellent post about Pairing Heroku with AWS RDS, go ahead and do that then carry on with this post. 😎

As April mentioned, we are actively deprecating legacy code here at Guild and breaking applications and logic into more manageable and scalable microservices. Our team has taken on the responsibility of user data in what we call the User Profile Service. We wanted this data migration to be as seamless and non-disruptive as possible for end users as well as for other applications using our data. The approach we chose was to maintain dual copies of the data until we could be very sure that moving over to UPS would not break anyone else’s code. This meant that we kept the same user data in the old source of truth (the Guild monolith database), as well as in the new UPS database. We achieved this awkward state by taking advantage of multi-DB connections, which are made easier in Rails 6.

Setting up the connection for multiple databases in a Rails 6 application is well documented and we were able to get it up and running with minimal tweaks. However, one piece we had a hard time getting to work were our tests in our CI/CD pipeline utilizing CircleCI. So, let’s dive in and talk about how we figured that out.

Imagine an rspec test:

In the above example we are calling an UpdaterService and asserting that records in both databases are updated and saved. We don’t want our records to get out of sync, so we need each update to be reflected in both databases! Since we have successfully created the link between the databases, this should theoretically work. We defined a database base model class that links to our secondary database:

Then we linked our “old” model to it:

We have all of our external databases set up, so when we run our spec:

$ rspec spec/services/updater_spec.rb

Finished in 0.19529 seconds (files took 3.41 seconds to load)

1 example, 0 failures

Success! Our database is set up correctly and our tests pass locally. OK, let’s push to GitHub and create a PR.

CircleCI is not happy with this approach:

Failure/Error: establish_connection YAML.load(ERB.new(File.read(Rails.root.join(“config”, “secondary_database.yml”))).result)[Rails.env.to_s]
ActiveRecord::NoDatabaseError:
FATAL: database “secondary_database_test” does not exist

It doesn’t have any idea of the secondary database. So, what to do? We decided that the path of least resistance would be:

1. Add PostgreSQL Client to our CircleCI Build

2. Generate a mock database table that matched the attributes we cared about

3. Test against that

Step One

Let’s install PostgreSQL client on CircleCI so we can easily generate a fake database table:

Step Two

We can just generate a fake table that has the correct attributes we need to test against with some simple SQL.

$ touch test_db.sql
$ vi test_db.sql

In this file we added a simple SQL snippet that creates a table that satisfies our testing needs!

Step Three

Putting it all together:

And that’s it! Now when CircleCI runs our tests, it can use this mock database in order to assert changes are reflected in the main database as well as the secondary database.

By utilizing Rails 6 Multi Database connections, we were able to unravel a complicated database migration into something more manageable, while creating limited disruption to our end users or to the other engineers within Guild. Since testing is an important facet of Guild’s engineering culture, it would have left a bit of a pit in our stomachs if we couldn’t test that our code was modifying both sources of data effectively in our CI/CD pipeline. We hope this article can help someone else too!