[Redis] Multi-key command in cluster mode (feat. CROSS-SLOT)

Problem

We have trouble executing mutli-key commands in redis cluster mode.

Have you ever seen “CROSSSLOT Keys in request don’t hash to the same slot” error?

This article will give you hints on how to resolve it.

Example of CROSS-SLOT issue

# Enter redis-cli with cluster mode
$ redis-cli -c
# Try to delete multiple keys
127.0.0.1:6379>  DEL key1 key2 key3 ...
-- (error) CROSSSLOT Keys in request don't hash to the same slot

Why do CROSSSLOT error occur?

Before see cross-slot issue, we have to understand how the Redis cluster executes each command.

Redis Cluster mode

Suppose we configure 3 master nodes with 3 shards and 1 replica node per each.

Docker-compose.yaml

version: '3.8'

services:
  redis-master-0:
    image: 'redis:6.0.5-alpine'
    container_name: redis-master-0
    command: redis-server /usr/local/etc/redis/redis.conf
    volumes: 
      - ./config/redis/redis-master-0.conf:/usr/local/etc/redis/redis.conf
      - redis-data-master-0:/data
    ports:
      - '6379:6379'
      - '6380:6380'
      - '6381:6381'
      - '6382:6382'
      - '6383:6383'
      - '6384:6384'
    environment:
      - REDIS_CLUSTER_ENABLED=true
      - REDIS_CLUSTER_REPLICAS=1
      - REDIS_CLUSTER_ANNOUNCE_PORT=6379

  redis-master-1:
    image: 'redis:6.0.5-alpine'
    container_name: redis-master-1
    network_mode: "service:redis-master-0"
    command: redis-server /usr/local/etc/redis/redis.conf
    volumes:
      - ./config/redis/redis-master-1.conf:/usr/local/etc/redis/redis.conf
      - redis-data-master-1:/data
    environment:
      - REDIS_CLUSTER_REPLICAS=1
      - REDIS_CLUSTER_ANNOUNCE_PORT=6380

  redis-master-2:
    image: 'redis:6.0.5-alpine'
    container_name: redis-master-2
    network_mode: "service:redis-master-0"
    command: redis-server /usr/local/etc/redis/redis.conf
    volumes:
      - ./config/redis/redis-master-2.conf:/usr/local/etc/redis/redis.conf
      - redis-data-master-2:/data
    environment:
      - REDIS_CLUSTER_REPLICAS=1
      - REDIS_CLUSTER_ANNOUNCE_PORT=6381

  redis-slave-0:
    image: 'redis:6.0.5-alpine'
    container_name: redis-slave-0
    network_mode: "service:redis-master-0"
    command: redis-server /usr/local/etc/redis/redis.conf
    volumes:
      - ./config/redis/redis-slave-0.conf:/usr/local/etc/redis/redis.conf
      - redis-data-slave-0:/data
    environment:
      - REDIS_CLUSTER_ANNOUNCE_PORT=6382
    depends_on:
      - redis-master-0

  redis-slave-1:
    image: 'redis:6.0.5-alpine'
    container_name: redis-slave-1
    network_mode: "service:redis-master-0"
    command: redis-server /usr/local/etc/redis/redis.conf
    volumes:
      - ./config/redis/redis-slave-1.conf:/usr/local/etc/redis/redis.conf
      - redis-data-slave-1:/data
    environment:
      - REDIS_CLUSTER_ANNOUNCE_PORT=6383
    depends_on:
      - redis-master-1

  redis-slave-2:
    image: 'redis:6.0.5-alpine'
    container_name: redis-slave-2
    network_mode: "service:redis-master-0"
    command: redis-server /usr/local/etc/redis/redis.conf
    volumes:
      - ./config/redis/redis-slave-2.conf:/usr/local/etc/redis/redis.conf
      - redis-data-slave-2:/data
    environment:
      - REDIS_CLUSTER_ANNOUNCE_PORT=6384
    depends_on:
      - redis-master-2

# this is used for enable redis cluster
  redis-cluster-master-entry:
    image: 'redis:6.0.5-alpine'
    container_name: redis-cluster-master-entry
    network_mode: "service:redis-master-0"
    volumes:
      - ./config/redis/redis-cli.sh:/usr/local/src/redis-cluster.sh
    command: /usr/local/src/redis-cluster.sh
    depends_on:
      - redis-master-0
      - redis-master-1
      - redis-master-2
      - redis-slave-0
      - redis-slave-1
      - redis-slave-2

volumes:
  redis-data-master-0:
    driver: local
  redis-data-master-1:
    driver: local
  redis-data-master-2:
    driver: local
  redis-data-slave-0:
    driver: local
  redis-data-slave-1:
    driver: local
  redis-data-slave-2:
    driver: local

At first, when a client requests a command to redis the request will be forwarded to the node randomly.
The node having the slot of the key, the node would respond to the client directly.

127.0.0.1:6379> DEL key11
(integer) 0

master-0 node has a slot of ‘key11’ so, responds to the client directly.

However, if not exist in the node?

127.0.0.1:6381> DEL key11
-> Redirected to slot [1787] located at 127.0.0.1:6379
(integer) 0
127.0.0.1:6379> DEL key11
(integer) 0

Let’s see a diagram

As you saw above, every single command will succeed either directly or after redirection.

Multi-key operation problem

If you try to execute multi-key commands like

• MGET/MSET
• DEL/UNLINK

Let’s see the multi-key cross-slot issue using ‘DEL’ command.

127.0.0.1:6379> DEL key1 key2 key3 key4 key5
(error) CROSSSLOT Keys in request don't hash to the same slot

CROSS-SLOT issue occurs when you try to perform an operation that involves multiple keys in a Redis Cluster, but those keys do not belong to the same hash slot.

SLOT how to be decided in Redis?

Refer to redis official docs

HASH_SLOT = CRC16(key) mod 16384

Solution

Grouping & Mapping

We already know redis calculates slot using CRC-16.
So it’s possible to calculate slot of a key in advance on the application this library

$ npm install cluster-key-slot

Application Logic

1. Send keys being deleted in batch to the Redis server
2. Calculate slot using CRC-16
3. Grouping keys by slot creating a slot-key map
4. Iterate all slot-key map
5. Each iterating, delegating the multi-key command on node which has hash slot provided by slot-key map.

Make slot-key hash map in application

Execute multi-key command per node which has slot.

Example code (Typescript)

async deleteAll(keys: ReidsKey[]) {
    // (1) Grouping keys by slot
    const slotKeysMap: Map<number, RedisKey[]> = new Map()

    for (const key of keys) {
        // CRC16(key) mod 16384
        const slot = this._redisNodeManager.calculateSlotByKey(key)

        if (!slotKeysMap.has(slot)) slotKeysMap.set(slot, [key])
        else slotKeysMap.get(slot).push(key)
    }

    const batchCommands = []
    for (const [slot, keys] of slotKeysMap) {
        // find master node which has slot
        const targetMasterNode = await this._redisNodeManager.getTargetMasterNodeBySlot(slot)
        batchCommands.push(targetMasterNode.del(keys))
    }

    await Promise.allSettled(batchCommands)
}

We get the benefit of reducing network round trip time (RTT) between client and redis-server by multi-key commands.
The number of commands has a range [1, 16384] which is decided by the number of slots.

P.S. Hash Tag solution

Many articles or blogs, introduce a ‘hash-tag’ solution to resolve CROSS-SLOT issue because redis key belongs to the same slot when having same hashtag {…}

Use the {...} key tags

However, this solution is not able to apply to keys that already exist

So I developed and shared an application-level solution.

References

• https://www.dragonflydb.io/error-solutions/crossslot-keys-in-request-dont-hash-to-the-same-slot