Horizontal expansion of Redis high availability clusters
Although the Redis3.0 and later versions have clustering capabilities, which provide higher performance and availability than the previous version of the sentinel mode, but the horizontal expansion of the cluster is more troublesome. Let’s take a look at how the Redis high-availability cluster is doing today. Expansion, the original cluster (see the following figure) consists of 6 nodes, and 6 nodes are on one machine (ip is 192.168.0.61), using pseudo-distributed three-master three-slave mode.
1. Start the cluster
# Start the entire cluster
/usr/local/redis/bin/redis-server /usr/local/redis-cluster/8001/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis-cluster/8002/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis-cluster/8003/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis-cluster/8004/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis-cluster/8005/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis-cluster/8006/redis.conf# The client connects to the redis instance of port 8001
/usr/local/redis/bin/redis-cli -c -h 192.168.0.61 -p 8001# View cluster
/usr/local/redis/bin/redis-cli -c -h 192.168.0.61 -p 8001
192.168.0.61:8001> cluster nodes
As can be seen from the above figure, the entire cluster runs normally, with three master nodes and three slave nodes. The instance nodes of port 8001 store 0–5460 of these hash slots, and the instance nodes of port 8002 store 5461–10922 of these hash slots, port 8003. The instance node stores 10923–16383 these hash slots. All the hash slots stored in the three master nodes constitute the storage slots of the redis cluster. The slave points are the backup slave nodes of each master node, and the storage slots are not displayed.
2. Cluster operation
We add another master (8007) to a slave (8008) on the basis of the original cluster. See the following figure for the cluster after adding nodes. The new nodes are indicated by dashed boxes.
1) Add redis instances 8007 and 8008
# Create 8007 and 8008 folders under /usr/local/redis-cluster, and copy the redis.conf file under the 8001 folder to the 8007 and 8008 folders.
mkdir 8007
mkdir 8008
cd 8001
cp redis.conf /usr/local/redis-cluster/8007/
cp redis.conf /usr/local/redis-cluster/8008/
# Modify the redis.conf configuration file in the 8007 folder.
vim /usr/local/redis-cluster/8007/redis.conf
# Modify the following:
port:8007
dir /usr/local/redis-cluster/8007/
cluster-config-file nodes-8007.conf
# Modify the redis.conf configuration file in the 8008 folder.
vim /usr/local/redis-cluster/8008/redis.conf
# Modify the content as follows:
port:8008
dir /usr/local/redis-cluster/8008/
cluster-config-file nodes8008.conf
#Start 8007 and 8008 services and view service status
/usr/local/redis/bin/redis-server /usr/local/redis-cluster/8007/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis-cluster/8008/redis.conf
ps -el | grep redis2) View the redis-trib command help
cd /usr/local/redis-3.0.0/src
redis-trib.rb1.create: Create a cluster environment host1:port1 … hostN:portN
2.call: can execute the redis command
3.add-node: Add a node to the cluster. The first parameter is the ip:port of the new node. The second parameter is the ip:port of any existing node in the cluster.
4.del-node: remove a node
5.reshard: resharding
6.check: check the cluster status
3) Configure 8007 as the cluster master node
# Use the add-node command to add a primary node 8007 (master), green for the new node, red for the known node, and see the last "[OK] New node added correctly" prompt for the new node to join successfully.
/usr/local/redis-3.0.0/src/redis-trib.rb add-node 192.168.0.61:8007 192.168.0.61:8001# View cluster status
Note: When the node is added successfully, the new node will not have any data, because it has not allocated any slot (hash slot), we need to manually assign the hash slot to the new node.
# Use the redis-trib command to assign a hash slot to the 8007, find any master node in the cluster (the red location represents any one of the master nodes in the cluster), and re-shard it.
/usr/local/redis-3.0.0/src/redis-trib.rb reshard 192.168.0.61:8001The output is as follows:
…
How many slots do you want to move (from 1 to 16384)? 600
(ps: How many slots need to be moved to the new node, set by yourself, such as 600 hash slots)
What is the receiving node ID? eb57a5700ee6f9ff099b3ce0d03b1a50ff247c3c
(ps: Which node to move the 600 hash slots to, you need to specify the node id)
Please enter all the source node IDs.
Type all to use all the nodes as source nodes for the hash slots.
Type done once you entered all the source nodes IDs.
Source node 1: all
(ps: Enter all to extract the corresponding number of slots from all the master nodes (8001, 8002, 8003) to the new node, and the total number of slots extracted is 600)
…
Do you want to proceed with the proposed reshard plan (yes/no)? yes
(ps: Enter yes to confirm the start of the sharding task)
… ……
#View the latest cluster status
As shown in the above figure, now our 8007 has a hash slot, which means that you can read and write data on the 8007! So far our 8007 has been added to the cluster and is the master node (Master)
4) Configure 8008 as a slave node of 8007
# Add from node 8008 to the cluster and view the cluster status
/usr/local/redis-3.0.0/src/redis-trib.rb add-node 192.168.0.61:8008 192.168.0.61:8001As shown in the figure, it is still a master node, and no hash is assigned.
We need to execute the replicate command to specify which node id of the current node (slave node), first need to connect to the newly added client of 8008 node, then use the cluster command to operate, assign the current 8008 (slave) node to a Under the master node (here the 8007 master node created before, red indicates the node id )
/usr/local/redis/bin/redis-cli -c -h 192.168.0.61 -p 8008192.168.0.61:8008> cluster replicate eb57a5700ee6f9ff099b3ce0d03b1a50ff247c3c
# View cluster status, 8008 node has been successfully added as a slave node of 8007
5) Delete 8008 slave node
# Delete the slave node 8008 with the del-node command , specify the delete node ip and port, and the node id ( red is 8008 node id )
/usr/local/redis-3.0.0/src/redis-trib.rb del-node 192.168.0.61:8008 1805b6339d91b0e051f46845eebacb9bc43baefe# Review the cluster status again, as shown in the following figure, the slave node of 8008 has been removed, and the redis service of this node has also been stopped.
6) Delete the 8007 master node
Finally, we try to delete the master node 8007 that was added before. This step is relatively more troublesome, because there is a hash slot allocated in the master node, so we must first put the hash slot in 8007 into other available hosts. Go to the node, and then remove the node operation, otherwise there will be data loss problem (currently the master data can only be migrated to a node, temporarily can not do the average allocation function), execute the command
/usr/local/redis-3.0.0/src/redis-trib.rb reshard 192.168.0.61:8007The output is as follows:
…
How many slots do you want to move (from 1 to 16384)? 599
(ps: not exactly 600 slots here)
What is the receiving node ID? deedad3c34e8437baa6ff013fd3d1461a0c2e761
(ps: Here is where you need to move the data? 8001’s primary node id)
Please enter all the source node IDs.
Type all to use all the nodes as source nodes for the hash slots.
Type done once you entered all the source nodes IDs.
Source node 1: eb57a5700ee6f9ff099b3ce0d03b1a50ff247c3c
(ps: here is the need for a data source, which is our 8007 node id)
Source node 2: done
(ps: Enter done directly here to start the migration plan)
…
Do you want to proceed with the proposed reshard plan (yes/no)? Yes
(ps: Enter yes here to start the migration)
At this point, we have successfully migrated the data of the 8007 master node to 8001. We can look at the current cluster status as shown below. You will find that there is no hash slot under 8007, which proves that the migration is successful!
# Finally, we can use the del-node command to delete the 8007 master node ( red indicates the node id of 8007 ).
/usr/local/redis-3.0.0/src/redis-trib.rb del-node 192.168.0.61:8007 eb57a5700ee6f9ff099b3ce0d03b1a50ff247c3c
# View the cluster status, everything is restored to the initial state of the cluster! You’re done!
