How to Create REDIS Sharded Cluster(with Replicas) in AWS/Docker
In this article I will discuss couple of advanced distributed system features that can improve robustness and performance, especially for large systems. The first topic we will explore is adding redundancy to a database through Replication. We will then build upon that to look at Sharding, a scalable partitioning strategy used to improve performance of a database that has to handle a large number of operations from different clients.
Both of these features are supported by REDIS.
REPLICATION
Replication refers to a database setup in which several copies of the same dataset are hosted on separate machines. The main reason to have replication is redundancy. If a single database host machine goes down, recovery is quick since one of the other machines hosting a replica of the same database can take over. A quick fail-over to a secondary machine minimizes downtime, and keeping an active copy of the database acts as a backup to minimize loss of data.
SHARDING
For larger render farms, scaling becomes a key performance issue. Having a large number of clients performing high-throughput operations can really test the limits of a single database instance. Sharding is a strategy that can mitigate this by distributing the database data across multiple machines. It is essentially a way to perform load balancing by routing operations to different database servers.
Install & Configure Master and Slave Nodes
Please refer my other articles on MongoDB and RabbitMQ, to see the steps on how to setup Linux EC2 instance and install Docker.
Install DockerCompose as well.
I am going to create a cluster using 6 nodes, which is sharded across 3 nodes, and each node has one replica node. We can even create 2 replicas of each master, as I did in Mongo cluster.
Create docker-compose.yml file and copy following script to it. I also defined subnet in docker-compose to assign private IPs to each container.
There is script to start commander as well, it is a client to access REDIS.
version: '2.2'
services:
redis01-master:
image: redis
user: root
sysctls:
net.core.somaxconn: 20000
ulimits:
nofile:
soft: 64000
hard: 64000
ports:
- 6379:6379
- 16379:16379
volumes:
- ./redis01-master/conf/redis.conf:/usr/local/etc/redis/redis.conf
- /var/log/redis/redis01-master/redis.log:/var/log/redis/redis.log
command: redis-server /usr/local/etc/redis/redis.conf
networks:
app_net:
ipv4_address: 173.17.0.10 redis02-slave:
image: redis
user: root
sysctls:
net.core.somaxconn: 20000
ulimits:
nofile:
soft: 64000
hard: 64000
ports:
- 6382:6379
- 16382:16379
volumes:
- ./redis02-slave/conf/redis.conf:/usr/local/etc/redis/redis.conf
- /var/log/redis/redis02-slave/redis.log:/var/log/redis/redis.log
command: redis-server /usr/local/etc/redis/redis.conf
networks:
app_net:
ipv4_address: 173.17.0.50 redis02-master:
image: redis
user: root
sysctls:
net.core.somaxconn: 20000
ulimits:
nofile:
soft: 64000
hard: 64000
ports:
- 6380:6379
- 16380:16379
volumes:
- ./redis02-master/conf/redis.conf:/usr/local/etc/redis/redis.conf
- /var/log/redis/redis02-master/redis.log:/var/log/redis/redis.log
command: redis-server /usr/local/etc/redis/redis.conf
networks:
app_net:
ipv4_address: 173.17.0.20 redis03-slave:
image: redis
user: root
sysctls:
net.core.somaxconn: 20000
ulimits:
nofile:
soft: 64000
hard: 64000
ports:
- 6383:6379
- 16383:16379
volumes:
- ./redis03-slave/conf/redis.conf:/usr/local/etc/redis/redis.conf
- /var/log/redis/redis03-slave/redis.log:/var/log/redis/redis.log
command: redis-server /usr/local/etc/redis/redis.conf
networks:
app_net:
ipv4_address: 173.17.0.60 redis03-master:
image: redis
user: root
sysctls:
net.core.somaxconn: 20000
ulimits:
nofile:
soft: 64000
hard: 64000
ports:
- 6381:6379
- 16381:16379
volumes:
- ./redis03-master/conf/redis.conf:/usr/local/etc/redis/redis.conf
- /var/log/redis/redis03-master/redis.log:/var/log/redis/redis.log
command: redis-server /usr/local/etc/redis/redis.conf
networks:
app_net:
ipv4_address: 173.17.0.30 redis01-slave:
image: redis
user: root
sysctls:
net.core.somaxconn: 20000
ulimits:
nofile:
soft: 64000
hard: 64000
ports:
- 6384:6379
- 16384:16379
volumes:
- ./redis01-slave/conf/redis.conf:/usr/local/etc/redis/redis.conf
- /var/log/redis/redis01-slave/redis.log:/var/log/redis/redis.log
command: redis-server /usr/local/etc/redis/redis.conf
networks:
app_net:
ipv4_address: 173.17.0.40 redis-commander:
image: rediscommander/redis-commander:latest
environment:
- REDIS_HOSTS=redis01-master:173.17.0.10:6379,redis02-master:173.17.0.20:6380,redis03-master:173.17.0.30:6381,redis01-slave:173.17.0.40:6384,redis02-slave:173.17.0.50:6382,redis03-slave:173.17.0.60:6383
ports:
- 8081:8081networks:
app_net:
driver: bridge
ipam:
driver: default
config:
- subnet: 173.17.0.0/16
- gateway: 173.17.0.1
I provided the following configuration for REDIS nodes in redis.conf, for more details please check REDIS official documentation
bind 173.17.0.10
protected-mode no
port 6379
pidfile /var/run/redis_6379.pid
cluster-enabled yes
cluster-config-file nodes-6379.conf
cluster-node-timeout 5000
appendonly no
save ""
tcp-backlog 20000
maxclients 10000
logfile "/var/log/redis/redis.log"
#logfile "/var/log/redis/redis01-master/redis.log"
repl-diskless-sync yes
repl-diskless-sync-delay 3Initialize the Cluster
Once all containers are up and running, connect to any one of the REDIS nodes and execute following command to initialize the cluster.
redis-cli --cluster create 173.17.0.10:6379 173.17.0.20:6380 173.17.0.30:6381 173.17.0.40:6384 173.17.0.50:6382 173.17.0.60:6383 --cluster-replicas 1Failover
Let’s assume one of the master nodes crashed/down. In that case the corresponding replica node will be promoted to new master.
Once you restart the node C or create a new node C, the cluster will again go back to old configuration of 3 master and 3 slaves. But C` will remain master and restarted node will join as slave to C`.
Articles referred:
https://hub.docker.com/r/bitnami/redis-cluster
