A few weeks ago the news broke that the company behind RethinkDB is shutting down. Not exactly what I wanted to hear a month after making the tough decision for my product squad to migrate our main service from MySQL to RethinkDB. I hope that Rethink has a long, happy life ahead of itself as an open source project, because it’s a great database for a lot of the work we do at Social Tables. In fact, we’re hopeful that transferring governance of RethinkDB to a foundation could lead to greater transparency and more responsiveness to community needs.
When I took over as technical lead for Guest, a guestlist management app, I inherited a backend architecture that was becoming increasingly difficult to maintain. Our primary data store was MySQL, where we saved all of our guestlist data and the relationships between guests, tags, meals, seats, and several other resources. Alongside this, we were running Elasticsearch to store composed guest documents for faster read availability. Originally we were using Elasticsearch for, you know, search, but it stopped serving this purpose once we built search on the client-side of our application. We were left with two very different data solutions that we struggled to keep in sync, and data integrity was a growing concern.
It was the proverbial mattress in a swimming pool that tech leads are supposed to avoid at all costs. As our business logic grew more complex, writing handlers and database accessors to operate on the same data in two places became ever more cumbersome. It was slowing down the team’s velocity and I could see a near future where certain critical features would be nearly impossible to implement.
Usually it would be difficult to find the time to pull off a refactor of this magnitude. Luckily, I had the opportunity to work on it with one other teammate when the engineering team at ST dedicated two weeks to an Innovation Sprint, an internal hackathon where autonomous teams of engineers used bleeding-edge technologies to build the next generation of events software.
So why Rethink? Rethink is incredibly powerful as a JSON database designed to support realtime apps, and above all, I realized what Guest really needed was a document store. The relationships between guestlists, guests, and collections could be represented differently and didn’t inherently demand a relational database. And after struggling to keep data consistent across two data stores, atomicity was top of mind and I liked that writes on a single document in Rethink are guaranteed to be atomic. And although we haven’t yet made use of Rethink’s realtime capabilities, I look forward to using those features to support collaboration across the web and mobile apps that represent Guest.
A few days into the refactor, as we started rewriting the database accessors in ReQL, Rethink’s query language, it became clear that this was the right decision. Our complicated MySQL queries to join data across multiple tables and then update the same data in Elasticsearch could be boiled down to one ReQL query for each endpoint. My pull requests from this work tended to involve a few hundred lines of code added and thousands of lines deleted.
It was a huge relief to be able to start writing our code this way, and when the rest of the engineers on our squad started working with Rethink they felt the same way. Since the refactor, it’s much easier for all squad members to contribute to the backend and our velocity writing that code is way up, leading to an increased ability to deliver features.
I won’t say that RethinkDB is perfect. ReQL syntax can be a little difficult to wrap your brain around at first, and the queries aren’t the easiest to debug, but the docs are solid and the Data Explorer makes testing queries straightforward. Another issue for us: unlike MongoDB, Rethink does not support unique secondary indexes. If we want to enforce uniqueness amongst collections stored in subdocuments, we’re writing that logic ourselves. And a small gotcha: Rethink doesn’t manage keeping connections alive and closes by default after 60 seconds. We worked around this by creating a connection for each request and closing it afterwards.
None of these things are deal-breakers and several product squads at Social Tables have adopted Rethink with great results. Rather than be wary of Rethink without a corporate steward, we’re optimistic about its future. As an open source project, there will be more freedom to prioritize community interests, which means the technology will continue to grow and maybe even accelerate improvements. I’m looking forward to participating in that community and leveraging its realtime features to keep improving the Social Tables platform.
Thanks to Dan MacTough and Erica Geiser for contributions and feedback on this post.