Redis 7 有什麼新更新呢?

Redis 在 2022–04–27 推出新版本 7.0 功能更新,而 ElastiCache Redis 也在 2022–11–08 也推出該版本的更新,而這個版本有幾個重要的更新,例如 “ Redis Functions”’、”ACL improvements — ACLv2"、”Sharded Pub/Sub”、”AOF improvement — Support Multi-Part AOF” 等新功能。

Jerry’s Notes
What’s next?
15 min readNov 9, 2022

--

#1 Redis Functions

  • Redis Functions 是一個新 script 功能,就跟使用 lua 是一樣的。會跟數據一樣,存放在 Redis server 端,就如果一般的鍵值。支援持久化,可複製,並且在節點重啓之後,可以直接從節點上讀取。
  • Redis functions written in Lua 5.1
  • Redis function 中不能去使用其他的 function (. (No support like LUA)
  • Can’t block write → atomicity and can’t rollback → big overhead. (No support like LUA)
  • Timeout setting default is 5 seconds (The same as LUA)
  • Redis Functions:

ElastiCache for Redis 7 includes Redis Functions, enabling developers to execute LUA scripts with application logic stored on the server. With Redis Functions, ElastiCache for Redis stores the functions alongside the data thus making the scripts just as durable as the data in Redis, and does not require re-sending the scripts to the server with every connection. ElastiCache for Redis 7 will automatically manage reloading your functions in case of node failures or replacements, and addition of shards when scaling out.

  • Function Life Cycle

A function needs to be created and named in Redis before it can be used. To do this, the FUNCTION CREATE command is used. Function code is loaded into the specified engine that compiles and stores it. After the function is created, it can be invoked using FUNCTION CALL command that executes the named function. Created functions are also propagated to replicas and AOF, and are saved as part of the RDB file.

  • 支援匯出/備分 functions

[+] Redis functions | Redis:
https://redis.io/docs/manual/programmability/functions-intro/

Function 功能測試記錄

#2 ACL improvements — ACLv2

ElastiCache for Redis 7 includes support for the next version of Redis Access Control Lists (ACLs). ElastiCache for Redis 6 introduced ACLs, enabling customers to limit the commands or group of commands that a specific user could execute on a set of keys. With ElastiCache for Redis 7, clients can now specify permissions on specific keys or keyspaces in Redis using selectors, and specify multiple sets of permissions on keys or keyspaces for the same user.

Multiple selectors

A selector is a set of allowed commands + first args + a set of key patterns that match against Redis commands. Each user can now have 1 or more selectors, and as long as 1 selector matches against the user, the command will be allowed. We are introducing the concept of the “root permissions” which is the selector that is applied to users when they are created. This selector is mutable in order to be maximally backwards compatible.

Selectors

Starting with Redis 7.0, Redis supports adding multiple sets of rules that are evaluated independently of each other. These secondary sets of permissions are called selectors and added by wrapping a set of rules within parentheses. In order to execute a command, either the root permissions (rules defined outside of parenthesis) or any of the selectors (rules defined inside parenthesis) must match the given command. Internally, the root permissions are checked first followed by selectors in the order they were added.

For example, consider a user with the ACL rules +GET ~key1 (+SET ~key2). This user is able to execute GET key1 and SET key2 hello, but not GET key2 or SET key1 world.

Key based permissions

We will introduce three sub-permissions on keys, read + write. The goal here is to make it easy to restrict specific types of operations on key values and to allow defining permissions that work with modules. This functionality also lets you reason about “future proofing operations”. Key based permissions do not replace command permissions.

ACLv1 vs ACLv2 比較

#3 Sharded Pub/Sub

Redis has supported the publish-subscribe mechanism since 2.0. Users using the pubsub command family can establish a message subscription system. However, Redis pubsub has some problems in the cluster mode; the most significant of which is the broadcast storm brought by large-scale clusters.

Redis pubsub is published and subscribed by channel. However, channels are not treated as data processing in cluster mode. They do not participate in hash value calculation and cannot be distributed by slot. Therefore, Redis broadcasts messages to users in cluster mode.

The problem is clear. If a cluster has 100 nodes and users publish messages to a channel at node 1, the node needs to broadcast the messages to the other 99. If only a few of the other nodes subscribe to the channel, most of the messages are invalid, which causes waste to the network, CPU, and other resources.

Sharded-pubsub is used to solve this problem. It distributes channels by shards. A shard node is only responsible for processing its channels rather than broadcasting them, which simply avoids the waste of resources.

Sharded Pub/Sub 重要改善

可以跨分片組通道(shard channels),來傳遞訊息,分片組通道會同分配鍵值的方式一樣,將特定”分片組通道”,交由特定的分片組(shard/slot)來處理,所以客戶端(Redis Client)就只會到該分片組上的節點,來取得訂閱的內容。

Shard channels are assigned to slots by the same algorithm used to assign keys to slots.

A shard message must be sent to a node that own the slot the shard channel is hashed to.

  • 優點是,當發佈者(Pub)去 SPUBLISH 傳送內容到 “分片組通道(shard channels)” 時,就只會在特定的 “分片組(shard/slot)” 內來處理,而訂閱者(Sub)也必需到該分片組(shard/slot)節點,來訂閱的內容,這樣可以避免發佈者(Pub),傳遞內容到所有的分片組(shard/slot)上。
  • Redis7 開始提供三個新的命令 SSUBSCRIBE, SUNSUBSCRIBE, SPUBLISH 來達到 Sharded Pub/Sub 目的。

SSUBSCRIBE, SUNSUBSCRIBE and SPUBLISH are used to implement sharded Pub/Sub.

Reids Pub/Sub and Sharded Pub/Sub 的差別,及詳細測試記錄,請參考以下文檔。

#4 Snapshot improvements

There are some memory we can release in the fork child process:

Serialized key-values the fork child process never access serialized key-values, so we try to free them. Because we only can release big bulk memory, and it is time consumed to iterate all items/members/fields/entries of complex data type. So we decide to iterate them and try to release them only when their average size of item/member/field/entry is more than page size of OS.

Replication backlog Because replication backlog is a cycle buffer, it will be changed quickly if redis has heavy write traffic, but in fork child process, we don’t need to access that.

Client buffers If clients have requests during having the fork child process, clients’ buffer also be changed frequently. The memory includes client query buffer, output buffer, and client struct used memory.

Redis snapshot backup 快照如何執行,請參考下面文檔。

AOF improvement — Support Multi-Part AOF

The main issues with the the original AOFRW mechanism are:

buffering of commands that are processed during rewrite (consuming a lot of RAM)

freezes of the main process when the AOFRW completes to drain the remaining part of the buffer and fsync it.

double disk IO for the data that arrives during AOFRW (had to be written to both the old and new AOF files)

  • 在AOF文件中增加了數據更新時間點的標識,使得用戶可以恢復某一時間點的數據。
  • Add timestamp annotations in AOF for point-in-time recovery: aof-timestamp-enabled
  • ps://github.com/redis/redis/pull/9326

Enabled with the new aof-timestamp-enabled config option.

Timestamp annotation format is “#TS:${timestamp}\r\n”.” TS” is short of timestamp and this method could save extra bytes in AOF.

We can use timestamp annotation for some special functions.

know the executing time of commands

restore data to a specific point-in-time (by using redis-check-aof to truncate the file)

  • redis.confg

!!! 請注意 ElastiCache Redis 並不支援 AOF 功能,以下是不支援的原因。

Q: Why ElastiCache didn’t support AOF after engine version 2.8.22?

AOF is disabled by default. To enable AOF for a cluster running Redis, you must create a parameter group with the appendonly parameter set to yes. You then assign that parameter group to your cluster. You can also modify the appendfsync parameter to control how often Redis writes to the AOF file.

■ 主要理由是因為點節點發生故障時,EalsatiCache會直接更換該節點,所以AOF無法避免資料遺失,故使用多從節點來同步(replication)數據,才是比較佳的做法。另外AOF在還原數據時,所花費的時間也較使用RDB檔來得時間長,另外AOF 開啟 always 寫入時,也會因為磁盤寫入問題,而造成 Redis Engine stuck。

#5 Improved management of memory consumed

  • Improved management of memory consumed by network buffers, and an option to drop clients when total memory exceeds a limit
  • 多了一個參數 maxmemory-clients: 可以限定所有客戶端使用的內存總和的最大值。

A mechanism for disconnecting clients when the sum of all connected clients is above a configured limit. This prevents eviction or OOM caused by accumulated used memory between all clients. It’s a complimentary mechanism to the client-output-buffer-limit mechanism which takes into account not only a single client and not only output buffers but rather all memory used by all clients.

maxmemory-clients max memory all clients are allowed to consume, above this threshold we disconnect clients.

This config can either be set to 0 (meaning no limit), a size in bytes (possibly with MB/GB suffix),

or as a percentage of maxmemory by using the % suffix (e.g. setting it to 10% would mean 10% of maxmemory).

舊的參數:

  • client-output-buffer-limit normal 0 0 0
  • client-output-buffer-limit replica 256mb 64mb 60
  • client-output-buffer-limit pubsub 32mb 8mb 60
  • Client-output-buffer-limit-slave-hard-limit (Default by node type)
  • Client-output-buffer-limit-slave-soft-limit * Default by node type)
  • client-query-buffer-limit: default 1GB (added in 4.0.10)

#6 RDB version up to 10

  • 向下相容舊版的 RDB 檔,但一樣,只能向下相容,舊版本的 Redis 不能使用新版本 Redis 所匯出的 RDB 檔。

#7 Protected-mode default value from no to yes

  • 在redis.conf配置文件中,protected-mode 默認更改為yes。
  • 只有當你希望你的客戶端在沒有授權的情況下可以連接到Redis server的時候可以將protected-mode設置為no。

#8 Support for hostnames, instead of IP addresses only on cluster mode

  1. New config “cluster-announce-hostname” which is a hostname that an externally facing client can use to connect to this node. Using the new mechanism we will send an hostname extension to all nodes, so that eventually all nodes in the cluster will know our hostname.
  2. Nodes do not talk to each other with the hostname.
  3. This hostname will be added as the 4th field to the CLUSTER SLOTS output which is the primary way clients will discover it.
  4. Another new config “cluster-preferred-endpoint-type” option to configure what type of endpoint is shown by default.

#9 Others new parameters

#10 redis-cli new added parameters

怎麼使用 redis command line tool : redis-cli 請參考以下連結。

--

--

Jerry’s Notes
What’s next?

An cloud support engineer focus on troubleshooting with customer reported issue ,and cloud solution architecture.