MongoDB integer primary key (nodejs example) using server side functions

Piotr Karpala
Apr 1, 2018 · 5 min read

About 1 year ago I developed a nodejs REST API that was supposed to replace another application. Basically, my team didn’t have access to some enterprise API on our testing environments, so we had to develop something quickly that fulfilled same requirements.

One funny thing about the app that we were “mocking” was that it used a relational database (Oracle) and it was generating old school integer primary keys for each inserted object.

I wanted to go with MongoDB for few reasons:

  1. Easy to test — there are DB mocks available
  2. Super easy to integrate with nodejs (everything is JSON already)
  3. Easy to setup with docker

About the primary key thing I just thought — “how hard can it be”? Turns out it wasn’t that hard but it was definitely interesting, hence this post :)

MongoDB and keys

MongoDB (by default) uses something called “ObjectId” for the data keys.

You can search for data this way:

But in fact, any type can be the key, although MongoDB recommends using ObjectId due to its qualities for handling big collections, sharding etc.

That’s all great, but what happens if we need an integer key that goes in sequence? Turns out MongoDB doesn’t have it out of the box, so it’s up for developer to implement that.

First, let’s look at MongoDB transactional model — in MongoDB, a write operation is atomic on the level of a single document, even if the operation modifies multiple embedded documents within a single document.

Theoretically, nothing is stopping us from putting that key generating logic in our source code, but it might not be that easy. Things to consider here are:

  1. How to ensure we’re getting a sequence? Is some global id++ enough?
  2. How to avoid key collisions when multiple requests are asking for a new key?
  3. Is it going to be easy to test?
  4. Is it going to be easy to scale (more than one node process)

Thread synchronization and atomic operations can be tricky, some languages handle that better than others. But maybe there’s another way? Usually the thing in software that does atomic operations very well is the database, so maybe there’s a feature in MongoDB that can be used here?

Turns out there is.

FindAndModify to the rescue

FindAndModify guarantees that nothing going to happen to the data between finding and updating, which is the transaction support we needed.

In this call, we look for object with _id equal to name, if it doesn’t exist it’s going to be created (that’s the upsert:true — update or insert). For the object that’s found/inserted, it performs an update — it increments seq by one. The new:true tells mongo to return object after update (by default it returns the object before update).

There are actually 3 behaviors that may happen when this is called by multiple clients.

  1. Exactly one findAndModify() successfully inserts a new document.
  2. Zero or more findAndModify() methods update the newly inserted document.
  3. Zero or more findAndModify() methods fail when they attempt to insert a duplicate. If the method fails due to a unique index constraint violation, you can retry the method. Absent a delete of the document, the retry should not fail.

Let’s walk through this and explain what happens in each case.

  • Case 1 is easy — document with _id equal to name doesn’t exist so it’s inserted.
  • Case 2 is what we wanted all along — multiple calls and each updates the newly inserted document.
  • Case 3 is interesting. Because we’re querying by _id and MongoDB guarantees that _id is unique, we can get an insert failed exception due to key constraints. This may happen when two clients are calling this method, object with _id: name doesn’t exist and both of them are trying to perform an insert.

I’ve decided to use a stored procedure to the job, and before you close this browser window shouting “This is B#! Stored procedures suck!!!”, hear me out.

Yes, they do suck in most cases :) BUT — I think this use case justifies them.

Yes, MongoDB has something like stored procedures

Btw. I’m using hapijs in this example, but this is going to work anywhere. This code is executed in server.js file, just after HTTP server starts.

const db = server.mongo.db;

Logger.log(`Started the API on port ${server.info.port} using environment: ${process.env.NODE_ENV}`);

As you can see, I’m saving (upserting) a MongoDB code object in the system.js collection. The code object is a function that can be executed on the server in any query.

Because it’s done every time server starts — I know it’s always up to date and ready to do what code is expecting.

At this point you might be thinking

“Ok. I’m cool with the procedure, but this has been a complete rip off of MongoDB documentation”.

Ok, that’s fair. The thing is that at this point MongoDB and my post diverge. They are saying to do this:

Which I couldn’t figure out how to do from nodejs API.

What I had to do is to evaluate the result of the stored procedure in code before performing the insert.

This method is called before each insert. There’s an obvious performance cost there, but what can you do if that’s the requirement?

findFreeId method will be called only in case when we hit a key collision, or another exception happens. It’s even more costly to run, because it tries to find a random free id (in a very high range, in case of this application it’s highly unlikely that id ever gets to that point).

For production ready systems, probably a better solution is required.

Maybe it would even make sense to use PostgreSQL or some other relational database to solve this particular problem of key generation.

In my case I wanted to stick to MongoDB to enable quick development and testability.

Summary

If you need auto incrementing id field in MongoDB — I recommend following https://docs.MongoDB.com/v3.0/tutorial/create-an-auto-incrementing-field/. In case of nodejs MongoDB API, you may also need to use eval method.

As far as MongoDB stored procedures (functions) are concerned, it’s a good tool to have in your toolbelt.

Piotr Karpala

Written by

NodeJS enthusiast

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade