Open in app

Sign in

Write

Sign in

Deep dive into Redis Clustering

Pubudu Boteju
8 min readMar 21, 2019

--

In this article, I will be talking about how Redis manages its storage in a distributed storage concept, how it handles fail-over and utilize performance and to setup Redis Cluster in local machine.

What is Redis?

Redis (Remote Dictionary Service) is,

  • An open source(BSD licensed)
  • NoSQL database server
  • In-memory data structure store (Keeps data in cache)
  • An advanced key-value store (Redis keeps data as key-value pairs)
  • Supports data structures such as strings, hashes, lists, sets and sorted sets

Redis Clustering

Redis Clustering provides a consistent and resilient data service where data is automatically sharded (Partitions data) across multiple Redis nodes (Automatically split your dataset among multiple nodes). And it provides a master/slave setup for enhance availability in case of a failure.

How Redis manages its storage in a distributed storage concept

  1. Redis Cluster Topology -

Minimal cluster that works as expected requires to contain at least 3 master nodes in the cluster and Redis recommendation is to have at least one slave for each master.

  • Minimum 3 Redis master nodes on separate 3 machines for each
  • Minimum 3 Redis slaves (One replica for each master node), 1 slave per master (to allow minimal fail-over mechanism)

2. Redis Cluster TCP ports -

Every Redis Cluster node requires two TCP connections open. The normal Redis TCP port used to serve clients, for instance let’s take 7000, plus the port obtained by adding 10000 to the data port, so 17000.

This second high port (In here, 17000) is used for the Cluster bus, that is a node-to-node communication channel using a binary protocol. The Cluster bus is used by nodes for failure detection, configuration update, failover authorization and so forth. If you don’t open both TCP ports, your cluster will not work as expected. So make sure that you open both ports in your firewall.

Note that for a Redis Cluster to work properly you need, for each node:

  • The normal client communication port (in here 7000) used to communicate with clients to be opened to all the clients that need to reach the cluster, plus all the other cluster nodes (that use the client port for keys migrations)
  • The cluster bus port (the client port + 10000) must be reachable from all the other cluster nodes.

3. Redis Cluster data sharding -

Redis Cluster does not use consistent hashing, but a different form of sharding where every key is conceptually part of what we call an hash slot.

There are 16384 hash slots in Redis Cluster (Redis clustering follows a distributed slot map approach which distribute 16384 slots among Cluster nodes), and to compute what is the hash slot of a given key, we simply take the CRC16 of the key modulo 16384.

Every node in a Redis Cluster is responsible for a subset of the hash slots, so for example you may have a cluster with 3 nodes, where:

  • Node A contains hash slots from 0 to 5500.
  • Node B contains hash slots from 5501 to 11000.
  • Node C contains hash slots from 11001 to 16383.

This allows to add and remove nodes (scale) in the cluster easily and does not require any downtime.

Redis Fail-over Procedure (Redis Cluster master-slave model)

Master-Slave concept in Redis increases data availability by preventing the single point of failure. Every master node in a Redis cluster has at least one slave node (A replica of the Master node). When the Master node fails to operate or becomes unreachable, the cluster will automatically choose its slave node/one of the slave nodes and make that one as the new Master. Therefore, failure in one node will not stop the entire system from working.

For instance let’s say we have cluster with nodes Master A, Master B, Master C, if Master B fails the cluster is not able to continue, since we no longer have a way to serve hash slots in the range 5501–11000.

However when the cluster is created (or at a later time) we add a slave node to every master, so that the final cluster is composed of Master A, Master B, Master C that are masters nodes, and Slave A, Slave B, Slave C that are slaves nodes, the system is able to continue if Master B fails.

Slave B replicates Master B, and when Master B goes down, then Redis Cluster will perform automatic fail over process that will force the Slave B as the new master and will continue to operate correctly.

  1. 3 Servers, 3 Redis Master nodes on each separate Server and 3 Redis Slaves, one Slave per each Master. This figure illustrates Master- Slave connection when every node is healthy

2. Server 2 unavailable

When a Master node fails to operate,the Server of that master node fails functioning. Lets consider our 2nd Server goes down and Master B does no longer functioning.

3. Fail over — Slave B becomes as Master B

Then Redis Cluster will perform automatic fail over process and a the relevant Slave becomes and works as the Master. In here Slave becomes and work as the Master B.

4. Check offsets and update the role ( Master B -> Slave B)

When the 2nd server available after some time, it checks its updates off set values and updates respective roles and makes connections again.

Attention: In this case 4, it can be a dangerous situation since two master nodes are there on the same server (Server 3). Therefore database administrator should rearrange the Cluster structure in such way that on each server will be one single master and a slave of some other master.

However note that if Master B and Slave B fail at the same time Redis Cluster is not able to continue to operate.

Inter node communication in a Redis Cluster: PING / PONG

All nodes continuously ping other nodes in the cluster. A node recognizes and marks another node as possibly failing when there is a timeout longer than N seconds.

Every PING and PONG packet contain a gossip section which contains the information about other nodes idle times, from the point of view of the sending node.

Setup a Redis cluster in you local machine

How to install Redis on Ubuntu 16.04

In here I will be talking about how to create a cluster using three Linodes to demonstrate sharding.

Prior to starting, I recommend you to familiarize yourself with the following:

Install Redis on Each Linode

Step 1- Install the dependencies:

sudo apt-get update && sudo apt-get upgrade
sudo apt install make gcc libc6-dev tcl

Step 2- Ensure the installation is successful by running:

make test

Step 3- Repeat installation for each and every server that will be part of the cluster.

Configure Master and Slave Nodes

This guide manually connects each master and slave across three Linodes

This setup uses three Linodes running two instances of Redis server per Linode. You must ensure each host is independent, and then consider using additional nodes if there is a need to maintain uptime requirements.

Step 1- SSH into server 1. Navigate to redis-stable/ then copy redis.conf. Configuration files in this guide are named consistent with the figure above:

cp redis.conf c_slave.conf
mv redis.conf a_master.conf

Step 2-In a_master.conf, comment the bind directive and enable cluster mode. The ports in this example will range from 6379 to 6381.

/redis-stable/a_master.conf

1 # bind 127.0.0.1
2 protected-mode no
3 port 6379
4 pidfile /var/run/redis_6379.pid
5 cluster-enabled yes
6 cluster-config-file nodes-6379.conf
7 cluster-node-timeout 15000

Step 3-In c_slave.conf, the configuration will be similar except for an update of the port number. redis-trib.rb will be used later to configure this into a slave for the appropriate master, rather than the slaveof directive.

/redis-stable/c_slave.conf

1 # bind 127.00.1
2 protected-mode no
3 port 6381
4 pidfile /var/run/redis_6381.pid
5 cluster-enabled yes
6 cluster-config-file nodes-6381.conf
7 cluster-node-timeout 15000

Step 4-Repeat this process across the remaining two Linodes, taking care to specify the port numbers for all master slave pairs.

Server………… Master……………… Slave
1……………..…6379………………...6381
2………………. 6380…………………6379
3………………. 6381…………………6380

Connect Master and Slave

Master/slave replication can be achieved across three nodes by running two instances of a Redis server on each node.

Step 1- SSH into server 1 and start the two Redis instances.

redis-server redis-stable/a_master.conf
redis-server redis-stable/c_slave.conf

Step 2- Substitute a_master.conf and c_slave.conf with the appropriate configuration file for the remaining two servers. All the master nodes should be starting in cluster mode.

Server 1

_._
                _.-``__ ''-._
           _.-``    `.  `_.  ''-._           Redis 4.0.1 (00000000/0) 64 bit
       .-`` .-```.  ```\/    _.,_ ''-._
      (    '      ,       .-`  | `,    )     Running in cluster mode
      |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
      |    `-._   `._    /     _.-'    |     PID: 10352
      `-._    `-._  `-./  _.-'    _.-'
      |`-._`-._    `-.__.-'    _.-'_.-'|
      |    `-._`-._        _.-'_.-'    |           http://redis.io
       `-._    `-._`-.__.-'_.-'    _.-'
      |`-._`-._    `-.__.-'    _.-'_.-'|
      |    `-._`-._        _.-'_.-'    |
      `-._    `-._`-.__.-'_.-'    _.-'
          `-._    `-.__.-'    _.-'
              `-._        _.-'
                  `-.__.-'

Create Cluster Using Built-In Ruby Script

Step 1- If Ruby is not already installed, it can be found in your package manager:

sudo apt install ruby

Step 2- Install the Redis gem:

gem install redis

Step 3- Navigate to the source directory to run the script.

redis-stable/src/redis-trib.rb create ip.of.server1:6379 ip.of.server2:6380 ip.of.server3:6381

Step 4 - Accept the configuration with three masters. Successful set up of the cluster will return the following message:

>>>Creating cluster
>>>Performing hash slots allocation on 3 nodes...
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join.
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

Step 5- See all the current nodes connected to the cluster by using the redis-cli tool. The -c flag specifies connection to the cluster.

redis-cli -c -h ip.of.server1 -p 6379
ip.of.server1>CLUSTER NODES

References :

Cloud Computing
Redis
Cluster

--

--

Pubudu Boteju

Written by Pubudu Boteju

21 Followers

Trainee Software Engineer at mvv Infromation Technology | Software Engineering Undergraduate |University of Kelaniya

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams