How to add a Redis cache layer to Mongoose in Node.js
Introduction
If you have a Node web server that uses MongoDB via `mongoose` and you would like to add a cache layer to it, I hope this article will help you reach your goal. We are going to build together, step by step, a cache layer using Redis and add it to an existing Node application.
If you are more comfortable to look directly at the repository here is the link to it. To run this application locally you need to install docker on your machine. If you are not familiar with docker and docker-compose you should visit docker official website.
Application Overview
The existing application we are going to work on is a simple Node app with Express, MongoDB and mongoose.
The application has two endpoints:
- GET /api/books — get a list of books from the DB.
- POST /api/books — add a new book.
Example of simple express routes
All the transactions executed over MongoDB through an ORM package called mongoose.
Example of a Mongoose model
Everything works now, right? Now let’s imagine we have a tremendous amount of books in our DB. Every time we try to query a list of books, we struggle with performance issues. We can tackle this problem from several different directions, but we are going to focus on a cache layer solution.
What does it mean a cache layer?
Cache is a high-speed data storage layer which stores a subset of data, so that future requests for that data are served up faster than is possible by accessing the data’s primary storage location. Caching allows you to efficiently reuse previously retrieved data.
Long story short, you can think it’s a tiny database that runs in the memory of your machine and allows you to read and write data quickly and efficiently.
There are a couple of products that give you a cache layer but we gonna focus on Redis. Redis is an in-memory data structure project implementing a distributed, in-memory key-value database with optional durability.
Let’s see how the cache layer fits in our application. Now instead of querying our MongoDB for every request, we will query the Redis first and only if we didn’t find what we are looking for, we will query MongoDB.
App Implementation
First, we will add a cache configuration and connect our application to the Redis server, for this purpose we will add the Redis package from npm.
Second, we need to add the logic that queries Redis, and if there is no answer it will query our MongoDB. After we got new data from MongoDB we’ll store it in Redis. Because we want to reuse this logic for different queries, we will have to add a hook in Mongoose’s query generation and execution process.
Cache file when exec function its the function we override
Every query in mongoose uses a Query class to build the query and executes it via exec function. We use the javascript prototype ability to add our reuse cache logic inside exec function.
To create the unique key in our cache, we use the query itself via getQuery() function that Query class exports (this function includes “where clause” ). Redis key has to be a string and getQuery() function returns an object, so we have to convert it to a string with JSON.stringify() function.
Also, you can see in line 7 that we add Promisify function to wrap Redis get function with a promise (it works with callback).
When we get the response back from Redis, the response type is a string. The expected result is the mongoose model, and we have to be consistent with this expectation. We use the constructor of the model we are querying to convert the response.
In case we didn’t get data from Redis cache, we will call the regular exec function that queries MongoDB, store the result in Redis and return it.
Is everything perfect?
In this solution, we have a few more issues to solve.
- Do we want to store the results in Redis forever?
No. Of course not.
Imagine that you query MongoDB for all the books of a specific author, and store the results in Redis. The next time you’re going to query for the same author, you’ll go straight to Redis. But what will happen if a new book was added to the DB for this author? You’ll never gonna get it!
So we have two ground rules in this case. First, add a TTL for each key in Redis. Second, clear the relevant keys in Redis each time the DB gets updated. - Is the query key stored in Redis unique enough?
No.
For example, we have another collection of authors in our MongoDB. we query mongo with the id of 1 in the books collection, so the key in Redis will be 1. After that, we query the authors’ collection with the same id of 1. Because we already have the id 1 stored in Redis we will get a response of books collection with the same id. To handle this problem we need to store each query for a specific collection. For this, we can use Redis hashes. - Do we want our cache mechanism to work for each query?
I don’t think so.
We need to pass an argument that represents our willingness to query Redis or query directly MongoDB. This argument will be used as a flag.
A full example of our cache service
After the above changes, we need to fix our routes file as so:
A relevant change in our routes function with a different example of cache use
Conclusion
We added a cache layer by expending mongoose Query exec function via a javascript prototype. Now we are able to query in an efficient way our DB and cache for every query we want without to change our code.
Thank you for reading, hope you enjoyed it.
Originally published at https://epsagon.com on January 9, 2020.
