Managing a Redis Stream

Redis streams have been my favorite Redis functionality when building real-time multi-process applications lately. Its data retention feature makes it more robust than Redis PUBSUB, and multiple consumers and consumer groups make it easier to use compared to Queues.

Redis streams act as a log data structure. Adding more entries to streams increases it’s memory and could impact some operations. To limit the amount entries, you can tell Redis to capped a stream with a limited count of entries.

XADD mystream MAXLEN ~ 1000 * <key> <value> [<key <value> ...]

Redis will trim the stream from the oldest entries when it reaches the number of entries specified in MAXLEN . The ~ tells Redis that the user isn’t exactly requesting the max capacity to be 1000. The stream could have more entries than specified in the command before Redis starts deleting entries.

Since trimming is an expensive operation, you can design your applications to delete messages after retrieved from the stream with XDEL . This might not be ideal if you have external logging services that also consume from the streams. Currently, Redis streams only support max size but there’s been a bit of discussion on a time-based deletion of entries in a Redis stream here: https://github.com/redis/redis/issues/4450.

Redis EXPIRE.

Yes, you can add a TTL on a stream with EXPIRE command. Take note that this works on the whole stream, not on the individual entries. Although you can design your application to create multiple streams based on a key - think of Kinesis shards. Then, set the TTL for each stream based on some configured time. You can also combine MAXLEN with EXPIRE commands to have both time-based and memory-based management of each stream.

These are just a few stream management concepts. Take a look at https://redis.io/topics/streams-intro to know more about Redis streams.