Replication in Databases with Redis: Architecture for horizontally scaling databases

Wait, what is this? And why do I need this?

All these fancy new words, replication, horizontal scaling seem …complex. And your application is probably already complex enough, I won’t be surprised if right now you are thinking why do you even need this. Why can’t you close this tab right now and go back to your simple Postgres running on an EC2 instance?

Before you close this tab, allow me to try to explain why these fancy new words are somewhat important by pointing out problems in the idea of the “simple DB running on an EC2 instance” architecture.

So, let’s say you have a database(it doesn’t really matter which one) that runs in a single virtual private server on some cloud hosting provider.

Okay, this is simple, so I simply have Postgres(again, it can really be any database, I just chose Postgres for now) running on some server. But does it have any problems? Is it perfect?

Availability

Let’s assume we are running this on an EC2 instance. AWS promises 99.5% availability of a single EC2 instance. 99.5% seems like a very large number, and it won’t be farfetched to kind of think of it as always available. But it is not always available, it won’t be available for 0.5% of the time. That roughly translates to 44 hours per year. This means that your database(and probably the rest of the application) would be down for ~2 days per year.

Apart from this, you may need to update your database, backup data, maybe perform OS updates, etc. All this adds to more downtime in your database.

If you are building something simple, like a blog website that gets a few hundred requests per day then you can manage that downtime. However, if you are building a large production system handling anything above hundreds of requests per minute, then the product team may not be too happy when you try to explain why your application is unavailable randomly for hours every month.

Scale

A single database running on a single server, at the end of the day, has access to limited hardware. This means a limited amount of RAM, a limited amount of CPU, and a limited amount of network bandwidth. This essentially means that as you add more users and features to your application, your database needs more of these physical resources. Sure, you can increase the size of your instance which would add more CPU and memory, maybe move from a medium to a large server, then to an xlarge server, and so on, but vertically scaling like this has limits. Eventually, your really really large server will also have issues handling a large number of users(and no, you can’t switch to a really really REALLY large server). Not only this, vertical scaling like this might also be costly.

The reason why vertical scaling can be costly is due to elasticity. Elasticity is simply a fancy word that defines how quickly can you scale your system up and how quickly can you scale it down. If you buy the really large server but that server is running at full capacity in peak hours but sitting idle at night, then you are wasting money by running this large of a server at night.

your really really REALLY large server at night probably

So, what should we do?

Preface

Before we dive into the architecture, I wanted to talk about a few things,

Since I want to write a blog post and not a book, I’d explore these problems and their solutions with Redis. Similar solutions exist for other databases as well though.

I’d also like to get a bit of terminology sorted out.

Horizontal scaling is when you add more homogeneous nodes into your system. So instead of running a single database instance, you run two instances or three or … so on.

Vertical scaling is when you run a single database instance but you add more hardware capacity to it(by adding more RAM and CPU for example) to it. This comes with some downsides. More on that here.

Let’s talk Architecture!

We know that our single database on a single server has some problems. Maybe we can improve that. Let’s see how!

Horizontal scaling with replicas

A very intuitive way to solve this problem is just to add more databases(that we will call replicas). The idea is simple, we say that our original database is master and we add more databases, calling each of these databases replicas. The names are pretty descriptive and should tell you what each of these databases does. The master database holds the ground truth of the data and the replicas simply replicate the data from the master database.

All the write operations go to the master database, and the read operations can go to the replicas.

We can have a lot of flexibility here on how communication between master and the replicas work, whether these replicas communicate to the master, or maybe the master communicates to the replica when new data is added, what kind of consistency we are targeting, etc. but for now, let’s assume that data is written to the master database and the master database sends a stream of new data written to it to its replicas(this is how it would work in Redis). Let’s also assume this communication is asynchronous. What this means is that the master database will not wait for the replicas to add or update data.

When the master database receives a request to write new data or update older data, it performs those operations and sends the stream of commands to the replicas. But once it has sent the commands to the replicas, it doesn’t wait for the replicas to actually perform that operation. Instead, it returns to the client that data has been added. Eventually, the replicas also insert or update the data and become consistent with the master database.

This means that sometimes(when your master has inserted data but your replicas haven’t yet) your database can be in an inconsistent state and when users are reading data from replicas, they may end up reading stale data. Depending on what you are building, this may or may not be an issue for you. For example, if you are building a website like Youtube, and you use Redis to store like count for videos, then if you were to use replicas when someone would like a certain video on Youtube, it is possible that it may take a second or so for the new like count to be visible to other users. This shouldn’t really matter for that specific application but for some other applications, that inconsistency can be an issue.

Adding these replicas to our architecture helps us a lot. For most applications, read traffic is much higher than write traffic. Think Twitter or Youtube for example. How many tweets do you read every month and how many times do you tweet something. This ratio could be one new tweet you write to a hundred tweets you read. This should be similar to most people on Twitter in fact. The traffic that you’d get on an application like this is very read-heavy. The good thing is that our replica nodes are made precisely for reading, and we can add more replica nodes if required.

We could even run these on separate virtual private servers.

Now we can distribute our read traffic to these separate servers and we can send write traffic to our master database. This way we can improve the performance of our database very significantly. Now we don’t need to rely on a single machine(and its limited hardware) to serve all requests and can instead utilize multiple machines for this.

We also improve the availability of our system when reading. Instead of a single EC2 instance, we have multiple instances that can serve read requests. This means that even if a single instance goes down, we still have others that can be used to handle the load.

Apart from all this, we can have more complex architectures as well, maybe we can add more replica nodes during peak traffic and scale in during off-hours, such as at night. This way we don’t waste server resources and save money as well. This also fixes the elasticity problem we were discussing earlier.

Furthermore, we can also improve the availability of our system when writing as well. Let me explain how. If the master node were to fail,

We can promote one of the replicas to become the master(after all they are also Redis instances).

This obviously is not very simple, but good for us there are projects(like Redis Sentinel) that already tackle this problem and we can use them without doing a lot of work!

Software engineers IRL

So, the configuration to do all this must be really complex, right?

No, not at all!

Implementation

I used docker to run the Redis replicas and the Redis master. This is what my directory structure looks like

.
├── docker-compose.yml
├── master
│   ├── Dockerfile
│   └── redis.conf
└── replica
    ├── Dockerfile
    └── redis.conf
2 directories, 5 files

You can check out all the source code on Github.

I created two docker images, one for Redis master and the other for Redis replica. For both of the images, the Dockerfile is identical-

It simply downloads the default configuration for redis:6.2, copies a custom configuration located in the context directory, and runs Redis with the new custom configuration. The reason why we need the default configuration is that our new custom configuration overrides the default configuration.

This is the configuration file for the master

And this is the configuration file for the replica -

The configuration files are pretty small, they just turn off protected mode and bind to 0.0.0.0 instead of 127.0.0.1, which allows us to use it from outside docker. Turning off protected mode is not a good approach for production environments but for our simple example, it should work fine. They also include the default configuration file to import all the default values for configuration that we haven’t defined in our files.

Finally, the replica configuration file contains this line

replicaof master 6379

This line denotes that the Redis database will serve as a replica of the database running on port 6379 on the host master.

Once we have all this setup, we can create a docker-compose file to run a master and two replica nodes.

And that is all!

We can simply build and run these containers using

docker-compose build; docker-compose up -d;

Once you have the containers up and running, you can open a Redis CLI for the master and one for one of the replicas to see the replication happen. The master is connected to your local port 6379, and the replicas are running on ports 6380 and 6381.

Here is a simple demo of what it looks like

Conclusion

Replicas are an awesome feature in most databases that helps in scaling up the database massively without a lot of effort. A lot of databases support creating these master-slave replications really well.

A good counterargument may be that in these architectures we are only scaling read capacity instead of write capacity. That is true, however, most applications are read-heavy. Think of the application you use, like Medium for example. How many blog posts do you read on Medium vs. how many do you write. Most of the traffic you are generating is read traffic. Not only that, for some databases, like relational databases, reading is heavier than writing. So, a lot of the problems related to handling scale in databases are with reading from the databases, instead of writing into them.

Replicas can also add to the availability of the database, and they can be useful for a lot of other use cases as well. For example, when you want to back up a database, your database may incur downtime or reduced performance. If you have a replica, then you can simply use the replica for backing up while the primary database would be running smoothly.

But what if your data solution requires write-heavy traffic? For example, a chatting application. In a chatting application where most of the communication is happening between two people, both of them would probably be reading older messages only once when they start the app, and then reading new messages as they come. They would also be writing the same amount of messages. So your read and write traffic is pretty similar. For this, we can look into master-master architecture or sharding. These are pretty large topics on their own so I will have to write about them separately.

Now, before closing this off, I would also point out a few issues.

The first one is docker compose is definitely not the right container orchestration for this problem. We can definitely add some complexity and make it better by using some other container orchestration, for example, ECS. Since I did not want to start with using ECS for database replication, I talked about docker compose to reduce some complexity in this post.

With read replicas or distributed data systems in general, you have to think about the CAP theorem. If you liked what I wrote and want to explore this further, I will definitely encourage you to read more into the CAP theorem.

Finally, as with everything, all of this comes at a cost. The cost for this can be more $$ on your AWS bill as you add more databases or developer time as devs build this architecture. Whether this is useful for you or not depends on what you are trying to do really.

That’s pretty much it, really. Thanks for reading if you made it this far.