Picture credits: https://www.flickr.com/photos/ochre_jelly/40678816071

The Hitchhiker’s Guide to MongoDB Transactions with Mongoose

Atomicity and Transactions

Before we can discuss MongoDB transactions, we need to know a little bit about atomicity and how it relates to Database and MongoDB in particular. Anything is atomic in nature if it is an indivisible and irreducible series of database operations such that either all occur, or nothing occurs.

So in simple terms, what Transactions allow us to do is group multiple database operations in a way, that either all of them succeed or none of them do. Suppose, we have two users John and Jane. John transfers $10 to Jane’s account. Let’s examine a simplified way of how this action can be stored in the database -

Step 1 — We subtract $10 to John’s account

Step 2 — We add $10 to Jane’s account

What if Step 1 completes successfully but Step 2 fails. Now, John has $10 less but Jane doesn’t have $10 added to her account. This causes an inconsistent state. This is where Atomicity and Transactions come to rescue. Using Transactions, we can make the two steps part of the same atomic operation. So, the changes are persisted in the database only when all the steps succeed otherwise any intermediate changes are rollbacked.

MongoDB Transactions

MongoDB has had atomic write operations on the level of a single document for a long time. But, MongoDB did not support such atomicity in case of multi-document operations until v4.0. Multi-document operations are now atomic in nature thanks to the release of MongoDB Transactions.

Before MongoDB v4, the two-phase commit system was used to emulate transactions. The npm module fawn was widely used for that and Emmanuel Olaojo wrote an article explaining the whole mechanism in great details. But, this system of transactions had its own drawbacks as it was just an emulation of the transaction semantics and inconsistencies during commit or rollback could still happen.

In MongoDB, the new transactions are based on sessions, we can start a session then start a transaction on that and after doing some database operation using that session we can either commit (save the changes) or abort (rollback the changes) the transaction. Finally, we can end the session. The transactions have the following properties —

1. When all operations in a transaction are successful, the session is committed and all changes made using that session is persisted in the database.
2. When an operation using the transaction session fails, the transaction session is aborted and no intermediate changes are persisted in the database.
3. Until a transaction session is committed no write operations in the transactions are visible outside the session.

One caveat to using MongoDB Transactions is that currently it is supported only on replica sets and not standalone servers or clusters (support for clusters coming soon). For development purposes, you can use the run-rs npm package to run MongoDB replica sets.

We have been using MongoDB transactions with Mongoose in production for quite some time now. Mongoose provides a nice Promise based API to work the transactions and sessions. This post is about sharing the experiences we have had working with it and the things that you might need to consider when working with Transactions.

Note: You need Mongoose v5.2 and above to work with Transactions.

Getting Started

Let’s get a minimal setup going for using MongoDB transaction working with Mongoose. Assuming we are using run-rs for the development environment, we first install run-rs globally —

$ npm i -g run-rs

Then, we run run-rs command to get our replica sets running —

// to run-rs with its own separate mongo installation
$ run-rs --version 4.0.0
// to use the mongo installation you've on your system
$ run-rs --mongod
// run-rs usually purges all old data, append --keep to skip purging
$ run-rs --mongod --keep

Now, that the replica sets are running, you can start a init a new project and install dependencies including Mongoose (v5.2 or greater).

Mongoose Transaction example

In the code snippet above, we try to implement the same money transfer scenario we discussed earlier using MongoDB transactions with Mongoose. You can clearly see that it can get really verbose. Therefore, we created a small package that takes care of most of the boilerplate code and provides a cleaner API to work with Mongoose transactions.

Useful Tips and Tricks

Here are some useful tips and tricks for working with Mongoose transactions API -

1. Helper function for managing Queries and Writes within a transaction

In the example above, we had to start a session, start a transaction, wrap all the queries and writes that needed the session in a try/catch block, commit/abort the transaction and finally end the session. This can get very repetitive when you are working with sessions in a huge codebase and can also cause errors and bugs if not taken care of correctly.

We have created a small package named mongoose-transact-utils that provides a helper function exactly for this use case —

runInTransaction util function usage example

The runInTransaction function takes care of the creating, starting, committing/aborting and ending of the transaction session. It accepts a callback that is passed the session. You can then use this session object for any queries or writes you want to happen within the transaction. If any error occurs within the callback, the transaction is aborted and the error is rethrown for you to handle (like reporting or logging the error).

2. Explicitly create each Collection

When using transactions, all the collections that need to be operated on must already be created. If not, it will throw an error like — Cannot create namespace xxx.xxx in multi-document transaction.

The MongoDB documentation says —

Operations that affect the database catalog, such as creating or dropping a collection or an index, are not allowed in multi-document transactions. For example, a multi-document transaction cannot include an insert operation that would result in the creation of a new collection.

We suggest using a structure like where you export all the database models from a single unified place —

Explicit createCollection commands for each Mongoose model

3. Avoiding write conflicts and related errors

MongoDB will throw a WriteConflict error if two separate transaction sessions attempt to modify or write the same document. This situation must be avoided. If you suspect this kind of situations can occur in your codebase, you may want to use some kind of locking mechanism to prevent this. Redlock is an excellent package for implementing such a solution using Redis.

Write Conflict Errors and Transient Transaction Errors

4. Accessing associated session in Mongoose

Mongoose attaches a $session on the return values of queries that are linked with a session.

Mongoose $session getter/setter example

This neat trick can be used to seamlessly chain related queries and writes sharing the same session or switching sessions when necessary.

5. Examples of using transaction session with Mongoose query and write APIs

An example demonstrating the usage of a session with common write and query methods available via Mongoose for quick reference purposes.

usage of a session with Mongoose API

Conclusion

Multi-document Transactions are still quite new to MongoDB and there are not many best practices of working with them out there yet. I hope the tips, tricks, and gotchas I have learned about while working with Transactions at scale alongside Nitish Mehta and Sharad Chand will help others adopt this new feature. Let us know about your experiences with the same.

At CashPositive, we strive to best utilize the cutting edge technologies to better solve the problems that businesses currently face in terms of credit management. If this interests you, drop us a mail at hiring@cashpositive.in

If you want to dive further in -
MongoDB — Atomicity and Transactions
Mongoose Transactions
GitHub — mongoose-transact-utils