Installing a High Available Redis Cluster
Architectures are developing and evolving, so is caching.
In memory caching has been generally implemented. However since the number of servers increases, in memory caching is getting heavy burden.
Then distributed caching concept has emerged. The applications do not hold the cached data on them anymore, it is located on the separated servers.
Redis is the most widely known and used distributed caching application. Actually it is a key value nosql datastore.
The problem begins at this point.
As Redis is a nosql database, it supports CAP.
Redis uses sharding in order to divide data into chunks. It is called in Redis terminology is Clustering.
And availability is made up with Sentinels.
Clustering
Redis sharded data automatically into the servers.
Redis has a concept hash slot in order to split data. All the data are divided into slots.
There are 16384 slots. These slots are divided by the number of servers.
If there are 3 servers; A, B and C then
Node A contains hash slots from 0 to 5500.
Node B contains hash slots from 5501 to 11000.
Node C contains hash slots from 11001 to 16383.
Redis manages the data on the servers by using master-slave model.
In order to remain available when a subset of master nodes are failing or are not able to communicate with the majority of nodes, Redis Cluster uses a master-slave model where every hash slot has from 1 (the master itself) to N replicas (N-1 additional slaves nodes).
In our example cluster with nodes A, B, C, if node B fails the cluster is not able to continue, since we no longer have a way to serve hash slots in the range 5501–11000.
However when the cluster is created (or at a later time) we add a slave node to every master, so that the final cluster is composed of A, B, C that are masters nodes, and A1, B1, C1 that are slaves nodes, the system is able to continue if node B fails.
Node B1 replicates B, and B fails, the cluster will promote node B1 as the new master and will continue to operate correctly.
Let’s create a 3 nodes redis cluster which has 1 master and 2 slaves.
In order to create 3 servers, Vagrant should be facilitated.
Vagranfile below, creates 3 Ubuntu 16.04 servers.
+-----------+--------------------+
| IP | Hostname |
+-----------+--------------------+
| 10.0.0.11 | redis-master-1 |
| 10.0.0.12 | redis-slave-1 |
| 10.0.0.13 | redis-slave-1 |
+-----------+--------------------+By using vagrant ssh command respectively, we can install redis server at first hand.
$ vagrant ssh redis-master-1
$ vagrant ssh redis-slave-1
$ vagrant ssh redis-slave-2The commands below are;
1. First which redis version is installed, I preferred 4.0.11 in this article.
2. Then configure OS specific language settings.
3. Install build-essential and tcl packages. Redis is build manually then make command needs these libraries.
4. Download the redis file and extract it into a folder.
5. Then with make command, build redis binary. It you wish by make test command, tests should be run.
6. By providing the file locations and redis port to the ./utils/install_server.sh, necessary system services are created.
Both redis-server and redis-sentinel services are created.
7. So as to configure master and slaves, the redis services are stopped.
Now we have 3 redis servers with default configurations, it is time to configure them as master slave model.
Redis holds config file at location /etc/redis/6379.conf
There are lots of settings such as
# OS runs the process as daemon
daemonize yes# process file location
pidfile /var/run/redis_6379.pid# exposed port
port 6379# take all the requests
bind 0.0.0.0# redis auth mechanism only takes password no username is provided. # In this context this_is_a_very_secret_password is the password.This setting is not necessary in slave.
requirepass this_is_a_very_secret_password
A server with these configurations runs as MASTER.
For slave,
# the password of master server, in this context this_is_a_very_secret_password is provided
masterauth this_is_a_very_secret_password
# this is the most critical setting. the ip or hostname of the master and the port.
slaveof 10.0.0.11 6379Now we have 1 master and 2 slave redis cluster.
With command redis-cli -a this_is_a_very_secret_password info on every node,
- At master node, it listed its role as master and there are two slaves and their ips and ports info.
- At slave nodes, their role as slave and master’s ip.
Shorten for the sake of brevity,—-@10.0.0.11-
redis_version:4.0.11
…
role:master
connected_slaves:2
slave0:ip=10.0.0.12,port=6379,state=online,offset=1862618,lag=0
slave1:ip=10.0.0.13,port=6379,state=online,offset=1862618,lag=0-@10.0.0.12-
redis_version:4.0.11
…
# Replication
role:slave
master_host:10.0.0.11
master_port:6379
slave_read_only:1-@10.0.0.13-
redis_version:4.0.11
…
# Replication
role:slave
master_host:10.0.0.11
master_port:6379
slave_read_only:1
So let’s test the nodes,
First we are setting a dummy key value pair at master node and the result is OK.
$ redis-cli -a this_is_a_very_secret_password -h 10.0.0.11 set key-01 value-01
OKHowever on slave nodes the responses are error on both.
$ redis-cli -a this_is_a_very_secret_password -h 10.0.0.12 set key-01 value-02
(error) READONLY You can’t write against a read only slave.$ redis-cli -a this_is_a_very_secret_password -h 10.0.0.13 set key-01 value-03
(error) READONLY You can’t write against a read only slave.
While getting the value of a the key given, at all nodes we can reach the value.
$ redis-cli -a this_is_a_very_secret_password -h 10.0.0.11 get key-01
“value-01”
$ redis-cli -a this_is_a_very_secret_password -h 10.0.0.12 get key-01
“value-01”
$ redis-cli -a this_is_a_very_secret_password -h 10.0.0.13 get key-01
“value-01”High Availability
In the happy path scenarios, the clients can connect to this cluster and make any read, write operations.
The real problem is when the master is down. Reads still work, but no write is in progress. None of the slaves should be elected as master, in order to cluster work successfully.
Here is the solution: Sentinel.
Sentinel runs as separate process and listens to master. Regarding to the master and slave info at master sentinel configure its settings.
This is the full list of Sentinel capabilities at a macroscopical level (i.e. the big picture):
Monitoring. Sentinel constantly checks if your master and slave instances are working as expected.
Notification. Sentinel can notify the system administrator, another computer programs, via an API, that something is wrong with one of the monitored Redis instances.
Automatic failover. If a master is not working as expected, Sentinel can start a failover process where a slave is promoted to master, the other additional slaves are reconfigured to use the new master, and the applications using the Redis server informed about the new address to use when connecting.
Configuration provider. Sentinel acts as a source of authority for clients service discovery: clients connect to Sentinels in order to ask for the address of the current Redis master responsible for a given service. If a failover occurs, Sentinels will report the new address.
Sentinel is designed as working distributed. Many sentinels can run cooperatively.
In case of failover scenarios, best option is setting up at least 3 nodes. 5 and 7 are options according to the scale of the system.
Sentinel is no more than a redis-server process. Only differences are process parameter and conf file.
redis-server /etc/redis/26379.conf — sentinel
Sentinel runs by default listening to 26379 tcp port.
Sentinel configuration is simpler than redis server.
1. Host and port we will listen for requests on
2. Master group name, master ip, port and quorum
3. Password of the master
When starting a sentinel process with the conf above, sentinel adds additional settings to its conf.
Every time master and slaves are switched these configs are changed as well on redis server nodes.
Under the section # Generated by CONFIG REWRITE
1. ip and port of the known slaves
1. ip and port of the known sentinelsLet’s create a sentinel cluster. For using less resource we should create at existing 3 nodes.
$ sudo REDIS_PORT=26379 \
REDIS_CONFIG_FILE=/etc/redis/26379.conf \
REDIS_LOG_FILE=/var/log/redis_26379.log \
REDIS_DATA_DIR=/var/lib/redis/26379 \
REDIS_EXECUTABLE=`command -v redis-sentinel` ./utils/install_server.shLet’s get some information about high availability.
- Ping the node
$ redis-cli -p 26379 -h 10.0.0.11 ping
PONG- Get the master of group redis-cluster.
When running the command for all nodes, the response is the same.
$ redis-cli -p 26379 -h 10.0.0.11 -a this_is_a_very_secret_password sentinel get-master-addr-by-name redis-cluster
1) “10.0.0.11”
2) “6379”- Get all the masters.
IP, port, role and number of slaves info are listed.
$ redis-cli -p 26379 -h 10.0.0.11 sentinel mastersShort for brevity.
1) 1) "name"
2) "redis-cluster"
3) "ip"
4) "10.0.0.11"
5) "port"
6) "6379"
9) "flags"
10) "master"
25) "role-reported"
26) "master"
31) "num-slaves"
32) "2"
33) "num-other-sentinels"
34) "2"
35) "quorum"
36) "2"
37) "failover-timeout"
38) "10000"
39) "parallel-syncs"
40) "1"
- Get all the slaves.
IP, port, role and master info are listed.
$ redis-cli -p 26379 -h 10.0.0.11 sentinel slaves redis-clusterShort for brevity.
1) 1) “name”
2) “10.0.0.13:6379”
3) “ip”
4) “10.0.0.13”
5) “port”
6) “6379”
9) “flags”
10) “slave”
25) “role-reported”
26) “slave”
33) “master-host”
34) “10.0.0.11”
35) “master-port”
36) “6379”
2) 1) “name”
2) “10.0.0.12:6379”
3) “ip”
4) “10.0.0.12”
5) “port”
6) “6379”
9) “flags”
10) “slave”
25) “role-reported”
26) “slave”
33) “master-host”
34) “10.0.0.11”
35) “master-port”
36) “6379”
- Let’s do a Failover. Then get master and slaves.
$ redis-cli -p 26379 -h 10.0.0.11 sentinel failover redis-clusterHowever, nothing is changed. This is because the sentinel updates redis server conf files. When converting a slave to a master, sentinel does NOT add password.
In order to resolve this problem, requirepass setting is also set in slaves.
After setting the configuration, rerun the failover command.
As seen the responses below, 2nd node becomes master and 1st node becomes slave.
If we try to set a value at 1st node, an error will be taken.
$ redis-cli -p 26379 -h 10.0.0.11 sentinel failover redis-cluster
$ redis-cli -p 26379 -h 10.0.0.11 sentinel masters
$ redis-cli -p 26379 -h 10.0.0.11 sentinel slaves redis-cluster1) 1) “name”
2) “redis-cluster”
3) “ip”
4) “10.0.0.12”
5) “port”
6) “6379”
9) “flags”
10) “master”
25) “role-reported”
26) “master”
31) “num-slaves”
32) “2”
33) “num-other-sentinels”
34) “2”
35) “quorum”
36) “2”1) 1) “name”
2) “10.0.0.11:6379”
3) “ip”
4) “10.0.0.11”
5) “port”
6) “6379”
9) “flags”
10) “slave”
25) “role-reported”
26) “slave”
33) “master-host”
34) “10.0.0.12”
35) “master-port”
36) “6379”
2) 1) “name”
2) “10.0.0.13:6379”
3) “ip”
4) “10.0.0.13”
5) “port”
6) “6379”
9) “flags”
10) “slave”
25) “role-reported”
26) “slave”
33) “master-host”
34) “10.0.0.12”
35) “master-port”
36) “6379”
- Another option to test failover is shutting down a server.
$ redis-cli -p 6379 -h 10.0.0.11 -a this_is_a_very_secret_password shutdownAs a conclusion, redis sentinel is a vital and powerful tool to achieving high availability in redis. There are more options in official document.
Sample codes are at https://github.com/nerobianchi/redis-ha
References:
