Redis Buffer Types
這邊說明 Redis Buffer 種類、及在 ElastiCache 中的參數值，並且說明各種 Buffer 的用途、及可能遇到的問題。
Redis needs to handle a variable-length output buffer for every client, since a command can produce a big amount of data that needs to be transferred to the client. However, it is possible that a client sends more commands producing more output to serve at a faster rate at which Redis can send the existing output to the client. This is especially true with Pub/Sub clients in case a client is not able to process new messages fast enough. Both the conditions will cause the client output buffer to grow and consume more and more memory. For this reason, by default Redis sets limits to the output buffer size for different kind of clients. When the limit is reached the client connection is closed and the event logged in the Redis log file.
There are three types of Client Output Buffer:
1) Normal Client Output Buffer (Normal COB)
Normal clients have a default limit of 0, that means, no limit at all..
■ client-output-buffer-limit-normal-hard-limit: 0 (Modifiable: Yes)
■ client-output-buffer-limit-normal-soft-limit: 0 (Modifiable: Yes)
■ client-output-buffer-limit-normal-soft-seconds: 0 (Modifiable: Yes)
Re-visit the feature to kill customer clients when Redis is over maxmemory (導致replica node連線中斷)
2) Pub/Sub Client Output Buffer (Pub/Sub COB)
Pub/Sub clients have a default hard limit of 32 megabytes and a soft limit of 8 megabytes per 60 seconds.
■ client-output-buffer-limit-pubsub-hard-limit: Default 32MB (Modifiable: Yes)
■ client-output-buffer-limit-pubsub-soft-limit: Default 8MB (Modifiable: Yes)
■ client-output-buffer-limit-pubsub-soft-seconds: Default 60s (Modifiable: Yes)
3) Replica Client Output Buffer (Replica COB)
Record in primary node: Default hard limit and soft limit depend on cache node type. And it can be found in public documentation.
■ client-output-buffer-limit-slave-hard-limit: Default depending on node type (Modifiable: No)
■ client-output-buffer-limit-slave-soft-limit: Default depending on node type (Modifiable: No)
■ client-output-buffer-limit-slave-soft-seconds: 60s (Modifiable: No)
Client query buffers: Input buffer
Max size of a single client query buffer and default is 1GB. Every client is also subject to a query buffer limit. That is the buffer we use to accumulate commands from the client, and is actually only an extreme limit to avoid a server crash in case of client or server software bugs.
■ 每個客戶端query buffer自動動態調整使用記憶體大小的，範圍在0~1GB之間；當某個客戶端的query buffer使用超過1GB — Parameter: client-query-buffer-limit: default 1GB (Modifiable: Yes), server會立即關閉它，為避免過度使用記憶體，觸發oom killer。
■ query buffer不受maxmeory限制: 模擬100個客戶端，連續寫入大小為500MB(生產建議小於1KB)的Key; redis server設定maxmemory為4gb,但redis實際已用記憶體43gb(見used_memory)。
Redis.conf
# client-query-buffer-limit 1gb
backlog buffer = only for replication
The backlog is used for recording updates to data at the primary node. When a read replica connects to the primary, it attempts to perform a partial sync (psync), where it applies data from the backlog to catch up with the primary node. If the psync fails, then a full sync is required.
The size, in bytes, of the primary node backlog buffer. The backlog is used for recording updates to data at the primary node. When a read replica connects to the primary, it attempts to perform a partial sync (psync), where it applies data from the backlog to catch up with the primary node. If the psync , then a full sync is required.
■ 從復制斷開重連時會用到，默認1MB
In parameter group:
■ repl-backlog-size: default 1MB (Modifiable: Yes) The size, in bytes, of the primary node backlog buffer. 增加這個數值，可以減少全同步的機率，但會增加數據不一致的機會。
■ repl-backlog-ttl: defaul 3600s (Modifiable: Yes) The number of seconds that the primary node will retain the backlog buffer.
■ repl-timeout (Default: 60) (Modifiable: No)
AOF buffer (not important in ElastiCache)
aof持久化使用的緩存和aofrewrite時產生的緩存用。
!!! 注意: ElastiCache Reids Version 2.8.22 之後版本，已不支援AOF功能。
Q: What’s different between hard limit and soft limit?
There are two kind of limits Redis uses (hard limit and soft limit)
The hard limit is a fixed limit that when reached will make Redis closing the client connection as soon as possible. 當超過 hard limit 時，會盡快去關閉連線。
Instead, the soft limit is a limit that depends on the time, for instance a soft limit of 32 megabytes per 10 secs means that if the client has an output buffer bigger than 32 megabytes for, continuously, 10 seconds, the connection gets closed. 超過 soft limit 時，同時會檢查時間長度是否超時。例如連續10秒超過 32MB。
How to troubleshouting COB? 排查方式
redis > client list
id=141285 addr=172.31.15.23:47206 fd=11 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=26 qbuf-free=32742 obl=0 oll=0 omem=0 events=r cmd=clientChecking current connected Client IP address (查目前連線中的 IP)
■ omem : Redis 收到 Redis client 的请求后，回覆的内容会放到内存中，也就是 omem。
■ ool: output list length (replies are queued in this list when the buffer is full)” 數量(筆數)
Q: 為什麼 omem 或 ool 會增加? Why omem or ool got increased?
正常現象，因為請求一個比較大的回覆，所以接受本來就會相對久一點。
發送很多命令來，Redis 處理完了，但 Redis client 還沒接受完。 (例如使用 pipe line，同時請求多個命令，Redis 都處理完了，但 Redis client 的接收速度可能會跟不上。)
Redis client 的底層可能有一些狀況 (網絡抖動、性能受限.. 等)，所以接收回覆的效率相對變差。
Q: 有什麼方法能監控? How to monitor Client output buffer.
定期監控 `client list` 的 omem，但 client list 是一個開銷比較大的命令，當 client 數多時要特別注意。
定期監控 `info client` 中的 client_recent_max_output_buffer 這個命令會記錄最近最大的 omem 值，開銷相對 `client list` 小，但缺點是不會知道是哪個 client。
監控 `info memory` 中的 `used_memory_overhead` 這個值能瞭解到，Redis 用來存放數據結構的內存是否有突然的變化。(有變化說明數據結構有變多或變少)
Redis command: CLIENT LIST
Q: 為什麼 replication output buffer 會增加? Why replication output buffer got increased?
■ High loading
■ Reached network limits
■ Full-sync
■ New replica adding caused Full-sync.
Q:為什麼 normal output buffer 會增加? Why normal output buffer got increased?
■ Client instance network limit reached
■ Client-side high loading
■ Client-side using pipeline
■ Request larger key
■ Client-side package lost or unstable
Q: 為什麼 pub/sub output buffer 會增加? Why pub/sub output buffer?
■ Client-side didn’t close session
Q: 如何Kill idle connections to Redis?
redis 127.0.0.1:6379> CLIENT LIST
id=9115408 addr=10.0.1.252:56348 fd=95 name= age=23 idle=23 flags=N db=14 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=9115336 addr=10.0.2.151:52018 fd=37 name= age=107 idle=107 flags=N db=14 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get$ sudo pip install redis$ cat killredisconn.py
----
import redis
import reidle_max = 300r = redis.Redis(host="localhost", port=6379, password=None)
cl = r.execute_command("client", "list")pattern = r"addr=(.*?) .*? idle=(\d*)"
regex = re.compile(pattern)
for match in regex.finditer(cl):
if int(match.group(2)) > idle_max:
r.execute_command("client", "kill", match.group(1))
----$ python killredisconn.py
