Redis — A Cache beyond Key-Value Store

> Aloha Honua REDIS

We all know a solid Cache plays a huge importance in building scalable cost-effective systems. In this day and age of instant gratification, be it ordering food online or booking a cab instantly; a good caching strategy could help you build good customer experience.

While there are umpteen articles that talk about caching in general. In this blog, I would like to touch upon a few aspects of Redis that leaves every other solution behind.

This is not a “What is” or “How To” but a “What could you do” article.

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes with radius queries and streams. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster. — What is Redis?

My personal journey with Redis began in 2011. I had explored it during its nascent stages when I was a one of those naïve engineers who was thrilled to push the boundaries with a piece of technology. Once I had learned to use it, I aspired to solve every other problem with it. Sounds familiar right? :-)

If Volkswagen is “Das Auto” — Redis is “Das Cache”, you may call it bias, I call it Trust ;-). As you scroll through I would be presenting to you classic problems which prove Redis is no Snake Oil, while one could argue there are better ways solving these problems. Let me tell you I am not running for Presidential Debate, this is just a perspective on the Technology’s versatility.

Key-Value Store, Obviously Duh.

Remember the classic philosophy of ‘doing one thing but doing it well’? I am more than delighted to say that Redis meets all its expectations. It solves an age-old problem of storing and fetching key-value pairs in a distributed environment. What more do you need?

127.0.0.1:6379> set foo bar
OK
127.0.0.1:6379> get foo
“bar”

Session Store

If you are building applications with Spring/Spring boot and want to deal with user sessions that could be shared across multiple nodes; essentially to manage session keys in a distributed system, the spring framework provides very good integration with Redis for session stores. It is quick and ‘grandma easy’.

Spring Session - Spring Boot

Throttling and Rate Limiter

Quite often if you offer SAAS or similar service, the world outside would be communicating with you via API. Now as you know every API call is expensive and a cost to the organization in terms of resource and bandwidth consumption. It may also have the potential of being misused. In such a case, a classic age-old solution of Throttling and Rate limiting could help curtail infrastructure and resource costs.

Just in case you search for a rate limiter on Github and end up finding a project, there is a pretty good chance it uses Redis for storage. This is where Redis is very versatile with its atomic operations like INCR

go-redis/redis_rate

The use cases even extend to storing article views, page view counts and unique visitors.

Leaderboards

Let us say you are an e-commerce site and you want actively keep track of top 10 items that have sold the most.

or

You have a service like Twitter or Instagram and you want to keep track of the top trending hashtags

or

Or you are a Gaming company and would want to keep track of top scorers across different games or verticals.

Redis has a very strong, sorted set collection that could be leveraged for any of the above 3 use cases. We will take the gaming use case to elaborate this further.

Let us say I want to find out the top scorers for ZELDA

127.0.0.1:6379> zadd ZELDA 100 "Alpha"
(integer) 1
127.0.0.1:6379> zadd ZELDA 200 "Bob"
(integer) 1
127.0.0.1:6379> zadd ZELDA 300 "Charlie"
(integer) 1
127.0.0.1:6379> zadd ZELDA 400 "Bravo"
(integer) 1
127.0.0.1:6379> zadd ZELDA 500 "Echo"
(integer) 1
127.0.0.1:6379> zadd ZELDA 600 "Zulu".......

I have added scores for all players hooked on to Zelda online. Here Zelda refers to a namespace and the values are player names with their respective score. the zadd command is a set upsert command

Now let’s say I want to query out the top 3 players for Zelda.

127.0.0.1:6379> zrevrange ZELDA 0 2 // Get 3 players from the SET 
1) "Zulu"
2) "Echo"
3) "Bravo"

Hurray! But what about their scores?

127.0.0.1:6379> zrevrange ZELDA 0 2 WITHSCORES
1) "Zulu" // Player
2) "600" // Score
3) "Echo"
4) "500"
5) "Bravo"
6) "400"

Wow, is there more?

Well, let us say now Alpha being alpha moves ahead of the curve with 1000 points.

127.0.0.1:6379> ZINCRBY ZELDA 1000 "Alpha"
"1100"

Let’s refire the query, shall we?

127.0.0.1:6379> zrevrange ZELDA 0 2 WITHSCORES
1) "Alpha" 
2) "1100" 
3) "Zulu"
4) "600"
5) "Echo"
6) "500"

There you have it, Alpha rules ;-)

Redis even exposes all these operations and commands through a standard language-specific client library.

While I do not intend to patronize the real world gaming leaderboard as the challenges quadruple with scale, the idea here is that if you are not worried about scale, do not reinvent the wheel.

Geo-Spatial Queries

Another powerful feature with Redis is its ability to perform geospatial queries within its Data set. I have personal experience in using this for hyperlocal advertising.

A hyperlocal system is essentially a software that finds the nearest service for incoming geo coordinates.

Being in the mobility space, a classical use case for a hyperlocal service could be to find drivers nearest to a passenger. While there are much-sophisticated systems built to address this efficiently, I do not want to shy but demonstrate how we achieve this through Redis.

To begin let us first load the driver details within the store.

127.0.0.1:6379> GEOADD BENGALURU 77.695139 12.937376 "Clark Kent"
(integer) 1
127.0.0.1:6379> GEOADD BENGALURU 77.597844 12.985010 "Barry Allen"
(integer) 1
127.0.0.1:6379> GEOADD BENGALURU 77.590762 12.979934 "Bruce Wayne"
(integer) 1
127.0.0.1:6379> GEOADD BENGALURU 77.574747 12.977229 "Diana"
(integer) 1
127.0.0.1:6379> GEOADD BENGALURU 77.702258 13.202677 "Hal Jordan"
(integer) 1

Now let us say I want to find out the driver closest to a particular location (77.653424 12.913261) within a 10 km range.

127.0.0.1:6379> GEORADIUS BENGALURU 77.653424 12.913261 10 km WITHDIST 1
1) 1) "Clark Kent"
2) "5.2577"
127.0.0.1:6379>

Clark Kent at your service sir

There you go, simple and efficient. Again these queries can be consumed by APIs and libraries that are totally polyglot.

Sets and Decision Engine

Redis SET is the most powerful beast in its arsenal. My prior experience was using it for Ad targeting. While this deserves a much detailed section, I will try to be as brief as possible.

An ad server is nothing short of a decision engine. You have a given set of input parameters that go through a rule processor that eliminates, filters and selects the best possible contextual ad to be served to the end-user end-user. Mathematically it's a SET AND OPERATIONS .

Let us take a hypothetical example that we have to pick up the most relevant advertisement based on Gender and Time of the day

We have configured the system to adhere to the following rules

For a Female audience show Ad1 or Ad2 or Ad3

For a Male audience show Ad4 or Ad5 or Ad6

If its Monday Show Ad1 or Ad6

127.0.0.1:6379> SADD FEMALE AD1 AD2 AD3
(integer) 0
127.0.0.1:6379> SADD MALE AD4 AD5 AD6
(integer) 1
127.0.0.1:6379> SADD MONDAY AD1 AD6

Now let us assume that this system is able to derive that a female member has visited the site and the current day is Monday. A simple interaction would determine which ad to show based on the context of DOW and the Gender

127.0.0.1:6379> SINTER MONDAY FEMALE
1) "AD1"

Here you go, Ad1 is the most relevant criteria for the given signal and condition.

Myth Busters

Redis is not persistent — Well who said that? There is a well-detailed article describing the same https://redis.io/topics/persistence , with small configuration changes you can flush data to more persistent stores at periodic intervals. There are different persistence options too. For a more reliable persistence you could write CRON jobs to push the RDB files (A binary file where Redis stores all data)to AWS S3 or Glacier for durability.

Deletion and purging are cumbersome — Absolutely not. Redis supports TTL for a large data sets within its cache, you can set a TTL for eviting keys and values from the cache at a given interval

All right, What’s Next?

Well, a large part of this article content was inspired by a very old article published on high scalability in 2011. You may have heard that quote “When life gives you lemons, make lemonade”, you now have recipes for Sorbet, Candies and more.

If you would like to know more please keep those claps coming. I promise to cover the following subjects before we colonize Mars.

1. Redis running in a Cluster, Replication.
2. Use cases with PubSub, Queueing.
3. BitOperations and its most relevant use case.
4. GeoHashes an extension to GeoSpatial queries.
5. Hyperloglog.

Until then Aloooha!