Redis Transactions and LUA script

Redis 支援操作一致性(Transactions)(原子性操作)的要求,您可以使用 MULTI 的命令,再包含您要連續執行的命令,最後再使用 EXEC 命令,來一次執行。而這個過程中不會有任何命令可以在中間執行,包含 Eviction 及 Reclaim。另外LUA script 也支持操作一致性(Transactions)的要求。

Jerry’s Notes
What’s next?
6 min readMar 21, 2022

--

Transactions

Redis — transactions: https://redis.io/topics/transactions

Q. Does Redis support Transactions? 原子性操作?

Redis transactions allow the execution of a group of commands in a single step.

This semantic is similar to the one of MULTI / EXEC. From the point of view of all the other clients the effects of a script are either still not visible or already completed.

All commands in a transaction are sequentially executed as a single isolated operation.

It is not possible that a request issued by another client is served in the middle of the execution of a Redis transaction. No other client requests are served when Redis is executing a transaction.

Transaction is Atomic, either all of the commands or none are processed.

MULTI, EXEC, DISCARD and WATCH are the foundation of transactions in Redis.

> MULTI 開始,中間的命令不會立即執行。

> EXEC 執行中間所有命令。

命令是 FIFO 的方式來執行。

> DISCARD 取消所有準備操作的命令。

WATCH: Marks the given keys to be watched for conditional execution of a transaction.

> WATCH XXXX 針對特定 key 進行監控,如果被監控的 key 被異動過,則執行 EXEC 時會失敗.。

WATCH is used to provide a check-and-set (CAS) behavior to Redis transactions.

WATCHed keys are monitored in order to detect changes against them. If at least one watched key is modified before the EXEC command, the whole transaction aborts and EXEC returns a Null reply to notify that the transaction failed.

Q: Why Redis does not support roll backs?

If you have a relational databases background, the fact that Redis commands can fail during a transaction, but still Redis will execute the rest of the transaction instead of rolling back, may look odd to you.

However, there are good opinions for this behavior:

■ Redis commands can fail only if called with a wrong syntax (and the problem is not detectable during the command queueing), or against keys holding the wrong data type: this means that in practical terms a failing command is the result of a programming errors, and a kind of error that is very likely to be detected during development, and not in production.

■ Redis is internally simplified and faster because it does not need the ability to roll back.

An argument against Redis point of view is that bugs happen, however it should be noted that in general the roll back does not save you from programming errors. For instance, if a query increments a key by 2 instead of 1, or increments the wrong key, there is no way for a rollback mechanism to help. Given that no one can save the programmer from his or her errors, and that the kind of errors required for a Redis command to fail are unlikely to enter in production, we selected the simpler and faster approach of not supporting roll backs on errors.

LUA script

LUA script 是一個可程式化的功能,讓使用者可以將業務邏輯用程式化的方式寫成 script,然後將這個 LUA script 匯入 Redis 中重複使用,除了這個目的外,LUA script 也具備操作一致性(Transactions)(原子性操作)的特性,當 LUA script 在執行中,若有寫入的操作,是無法被中斷的。 LUA scripting is used to extend the functionality of a Redis Server by allowing more complex logic and operations to embedded into a running Redis instance.

■ Replica node 從節點也是可以使用LUA腳本,但只能做讀取的操作。
■ 使用Lua腳本可以減少網絡的開銷。當網絡傳輸慢或者響應要求高的場景中尤為關鍵。
■ Lua腳本可以將多個請求通過腳本形式一次進行處理,減少網絡的相關時延。Redis還提供了Pipeline來解決這個問題。

■Lua腳本可以常駐在redis內存中,所以在使用的時候,可以直接拿來復用.也減少了代碼量
■ Atomicity: 原子操作。在Lua腳本中會將整個腳本作為一個整體來執行,中間不會被其他指令而打斷,因此保證了原子性。
■ Lua腳本本身也可以看作為一種事務,而且使用腳本起來更簡單,並且可控制的流程更靈活。
■ Lua腳本操作的數據,不能跨分片組(shard),要使用hashes (hset/hget) 來達到這個目標。

■ LUA scripting is used to extend the functionality of a Redis Server by allowing more complex logic and operations to embedded into a running Redis instance.

■ when Redis instance is running a LUA script, it will not run anything else.

It starts accepting commands again from other clients, but will reply with a BUSY error to all the clients sending normal commands. The only allowed commands in this status are SCRIPT KILL and SHUTDOWN NOSAVE.

■ With Redis, LUA scripts run inside the event loop of the Redis Server. These scripts are run after being loaded from the client and are either evaluated directly with “EVAL” or invoked later through a SHA1 digest of the LUA script that run with EVALSHA command. EVAL and EVALSHA are used to evaluate scripts using the Lua interpreter built into Redis.

■ When a script reaches the timeout — maximum execution time (five seconds by default), it is not automatically terminated by Redis since this violates the contract Redis has with the scripting engine to ensure that scripts are atomic. Interrupting a script means potentially leaving the dataset with half-written data. For these reasons when a script executes for more than the specified time the following happens

Here is a list of command when working with LUA script:

SCRIPT LOAD: 載入 LUA script 到 Redis 伺服器中。載腳本到腳本的緩存,但不執行它。

■ 主節點載入時,從節點也可以使用。

■ 從節點載入時,只有該節點可以使用。

> SCRIPT LOAD "return KEYS[1]"
" 4a2267357833227dd98abdedb8cf24b15a986445"

EVAL: 載入時,同時執行。

let’s you evaluates a script directly. 不用 script load,直接使用 eval 來執行與指令碼內容。

> eval "return KEYS[1]" 1 world
"world"

EVALSHA: 執行已載入的LUA腳本。

evaluates a script by its SHA1 digest that are cached on the server side using the SCRIPT LOAD command. EVALSHA command is otherwise identical to EVAL. SHA-1是一種密碼雜湊函式。

■ 從節點也會有一樣的資料,主節點跟從節點都可以執行。

> evalsha 4a2267357833227dd98abdedb8cf24b15a986445 1 hello
"hello"

■ SCRIPT DEBUG

■ SCRIPT EXISTS: 檢查特定腳本是否存在。

SCRIPT FLUSH: 清除所有的腳本緩存 (重啟Redis也是一樣的效果)。

SCRIPT KILL: 停到現在執行中的腳本。

ElastiCache Parameter: lua-time-limit (default: 5 seconds)

■ The SCRIPT KILL command terminates the running LUA script if the script has performed no write operations.

■ If the script has performed any write operations, the SCRIPT KILL command will not be able to terminate it; in that case, the SHUTDOWN NOSAVE command must execute.

!!! 當這個 LUA script 有包含”寫入” 的操作時,該 LUA script無法透過命令”SCRIPT KILL”來中斷,只能等該 LUA script 執行完成,或是執行時間超過 “ElastiCache Parameter: lua-time-limit” 5秒的限制,因為這5秒鐘內,是連 health check 都無法回應(Redis是單執行序),所以 LUA script 超時後,該節點會直接被 ElastiCache 直接汰換掉。

ElastiCache Parameter: lua-replicate-commands

Always enable Lua effect replication or not in Lua scripts (Default: yes)

When script effects replication is enabled, the restrictions on non-deterministic functions are removed. You can, for example, use the TIME or SRANDMEMBER commands inside your scripts freely at any place. 當關閉時,從節點只會更新”最後”在主節點執行完成的結果,預設是每一筆執行的過程,都會同步到從節點上。

!!! 這個參數只有在 ElastiCache 5 的版本可以被關閉。但這個功能在 Redis 官方文檔也提及,這個功能是沒必要的,所以在 Redis 7 也已經程除了。

https://redis.io/topics/eval-intro#script-replication
it is a shame to re-compute the script on the replicas or when reloading the AOF. In this case, it is much better to replicate just the effects of the script.

Note: Starting with Redis 5.0, script replication is by default effect-based rather than verbatim. In Redis 7.0, verbatim script replication had been removed entirely. The following section only applies to versions lower than Redis 7.0 when not using effect-based script replication.

LUA script and script kill 命令測試

測試1: 若 lua 只有讀取的行為,可以使用script kill 來中止

> eval 'while 1==1 do end' 0
(error) ERR Error running script (call to f_c045d3ae3b3eca855e00c772db40aa560b3a1fc8): @user_script:1: Script killed by user with SCRIPT KILL…
59.20s)

> get hello
(error) BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.
> script kill
Ok

### Redis server ###
24489:M 07 Jan 2022 04:43:19.469 # Lua slow script detected: still in execution after 5000 milliseconds. You can try killing the script using the SCRIPT KILL command. Script SHA1 is: c045d3ae3b3eca855e00c772db40aa560b3a1fc8
24489:M 07 Jan 2022 04:44:13.669 # Lua script killed by user with SCRIPT KILL.

測試2: 若 lua 有執行寫入的行為,無法使用script kill 來中止

127.0.0.1:6379> eval 'while 1==1 do redis.call("set","k","v") end' 0

> get key
(error) BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.

> script kill(error) UNKILLABLE Sorry the script already executed write commands against the dataset. You can either wait the script termination or kill the server in a hard way using the SHUTDOWN NOSAVE command.
> SHUTDOWN NOSAVE
not connected>

Q: Transaction 跟LUA script 有什麼不同?

■ Transaction: 將一串的命令連續執行,中間不充許執行其他命令。

■ LUA script: 充許客戶將命令 function 化來編程,並且可以載入 Redis中來重複使用,而該 LUA script 的過程中,也不充許執行其他命令。

兩者可以達到 transaction operation 原子性操作的目地,但用途是有些不同的。

Q: 為什麼執行 LUA script 時,收到 “(error) CROSSSLOST Keys” 這樣的錯誤訊息?

$ redis-cli -h xxx -c mset key1 value1 key2 value2
(error) CROSSSLOST Keys in request don't hash to the same slot

A: 因為可以時操作多個鍵值 key ,但只能hash slots在同一個node,所以造成在該節點上,而要操作在不同 slot/shard 上的鍵值,可以使用 hash tags 來將鍵值存取在同一個 slot/shard 上。

$ redis-cli -h xxx -c mset {key}1 value1 {key}2 value2
OK

--

--

Jerry’s Notes
What’s next?

An cloud support engineer focus on troubleshooting with customer reported issue ,and cloud solution architecture.