Redis Sorted Sets — Building Real-time Mood-Based Recommendation System

Redis is a powerful, open-source, in-memory data structure store. It is widely known for its speed and efficiency. Redis supports various data structures such as strings, lists, sets, hashes, and more. One of the key reasons Redis is so fast is that it keeps all data in memory, allowing for quick read and write operations.

Redis is not just a simple key-value store. It’s more like a Swiss Army knife for data management. With Redis, you can build complex applications using its primitive data structures. This flexibility makes Redis both fun and practical for developers.

Sorted Sets are like sets on steroids! If you are not familiar with Redis Sets, I recommend reading the previous article about Redis Sets first.

An Introduction to Redis Sets — The Underrated Data Structure

In this article, I will be using Upstash’s free Redis cluster. You can use up to 10k requests per day on the free tier, which is a great place to start.

Redis is a great technology with many features that enable us to improve the quality of our product. However, setting up a resilient Redis cluster requires significant experience and time. Upstash, the sponsor of this article, is run by a group of experienced engineers and offers industry-standard features. Check them out.

What is a Sorted Set?

A Redis Sorted Set is a unique data structure that combines the best features of a regular set and a sorted list. Like regular sets, sorted sets ensure that all members are unique. However, each member in a sorted set is associated with a score, which is used to sort the members in ascending order. This unique combination allows for efficient retrieval of elements based on their score or rank, making sorted sets ideal for use cases such as leaderboards, priority queues, and real-time recommendations.

Sorted sets are fast and efficient. Redis uses a skip list and a hash table to implement sorted sets, ensuring that operations such as insertion, deletion, and range queries are performed in logarithmic time complexity, O(log N). Moreover, sorted sets are compatible with regular sets, meaning you can perform set operations such as intersections and unions between sets and sorted sets.

Basic Sorted Set Commands

ZADD — O(log N): Adds one or more members to a sorted set, or updates their scores if they already exist.

ZADD myset 1 "item1" 2 "item2"

ZRANGE — O(log N + M): Returns a range of members in a sorted set, by index.

ZRANGE myset 0 -1

ZREM — O(log N): Removes one or more members from a sorted set.

ZREM myset "item1"

ZSCORE — O(1): Returns the score of a member in a sorted set.

ZSCORE myset "item2"

ZUNIONSTORE — O(N log M): Computes the union of multiple sorted sets and stores the result in a new sorted set.

ZUNIONSTORE resultset 2 set1 set2

Implementing a Real-Time Mood-based Recommendation System

Let me first explain what the real-time recommendation system is going to be like.

Imagine a social media platform or a marketplace. Each product or post has some tags, and now these tags also have scores! These scores represent how strongly a product or post is related to a particular tag. With this setup, we can leverage an algorithm to calculate the user’s mood in real-time. Based on the calculated mood, we can dynamically determine the most relevant products or posts and recommend them to the user.

To calculate the user’s mood, there are many algorithms, but the simplest one is to just calculate the cumulative score of recently interacted posts or products.

For example, if a user is currently interested in ‘tech’ and ‘gadgets’, we can assign scores to various products or posts related to these tags. A product like a new smartphone might have high scores for both ‘tech’ and ‘gadgets’, making it a strong candidate for recommendation. By continuously updating the user’s mood and recalculating the relevance scores, we can provide personalized and timely recommendations that match the user’s current interests.

For example, if a user is currently interested in ‘tech’ and ‘gadgets’, we can assign scores to various products or posts related to these tags.

1. We predefine the relationship between tags and products, ensuring each product has a score associated with relevant tags.
2. Based on the user’s mood, we determine the magnifier of each tag, which indicates how important that tag is to the user at the moment.
3. By combining the user’s mood with the predefined tags in a sorted set using a sum aggregator, we can calculate the total relevance score for each product.

This approach allows us to find the most relevant items for that user. By continuously updating the user’s mood and recalculating the relevance scores, we can provide personalized and timely recommendations that match the user’s current interests.

A Simple Implementation using Sorted Sets

We can achieve this with vectors as well, but the simplest way is by using sorted sets and combining them.

First, we need to assign products with tags:

# id=0 [Smartphone]
ZADD tag:tech 10 0
ZADD tag:gadgets 8 0

# id=1 [Laptop]
ZADD tag:tech 9 1
ZADD tag:computing 7 1

# id=2 [Smartwatch]
ZADD tag:tech 7 2
ZADD tag:gadgets 9 2

# id=3 [Gaming Console]
ZADD tag:gaming 10 3
ZADD tag:tech 6 3

# id=4 [Tablet]
ZADD tag:tech 8 4
ZADD tag:gadgets 7 4

# id=5 [Smart Home Device]
ZADD tag:tech 6 5
ZADD tag:gadgets 8 5
ZADD tag:home 9 5

Then, with an algorithm, we need to calculate the user’s mood based on interaction with other posts. In the end, we will have a list like this:

tag:tech score=10
tag:gadgets score=5

So, tech’s score is 10 and gadget’s score is 5. We can use ZUNION or ZINTER (based on the number of products and the company's preference) to generate a new list tailored to the user's mood:

ZUNION 2 tag:tech tag:gadgets WEIGHTS 10 5 AGGREGATE SUM WITHSCORES

The ZUNION command is used to compute the union of multiple sorted sets and store the result in a new sorted set. In this context, we are using it to combine the scores of products based on their relevance to the tags 'tech' and 'gadgets' according to the user's mood.

Here’s a breakdown of each part of this command:

• ZUNION 2: This specifies that we are combining 2 sorted sets.
• tag:tech tag:gadgets: These are the names of the sorted sets we are combining.
• WEIGHTS 10 5: This applies weights to the scores of the elements in the sorted sets. In this example, the score from the tag:tech set is multiplied by 10, and the score from the tag:gadgets set is multiplied by 5. This reflects the user's mood, giving more importance to 'tech' (weight 10) over 'gadgets' (weight 5).
• AGGREGATE SUM: This specifies that the scores of the elements should be summed when they appear in both sets. Other possible aggregations are MIN (take the minimum score) and MAX (take the maximum score).
• WITHSCORES: This ensures that the resulting sorted set includes the scores of the elements.

The result will be:

1) "3"  (Gaming Console)
2) "60"
3) "1"  (Laptop)
4) "90"
5) "5"  (Smart Home Device)
6) "100"
7) "2"  (Smartwatch)
8) "115"
9) "4"  (Tablet)
10) "115"
11) "0"  (Smartphone)
12) "140"

Performance Considerations

Depending on the number of products or the use case, this approach may be CPU consuming. Like in the last article, we can use ZINTERSTORE or ZUNIONSTORE to temporarily cache these values and use the same list for multiple users.

To narrow down the products, we can also generate a set of featured products (or a sorted set to give the featured products scores as well) and combine that with other sets using an intersection.

Remember that in an intersection, the cardinality of the smallest set will determine the time complexity of the operation.

Conclusion

Redis is a Swiss Army knife for the early stages of development. For many of these examples, in a real-world application, there are dozens of teams working on each feature simultaneously to improve overall performance and customization. However, for most use cases, especially in the early stages of a startup, with some tweaks, we can achieve a significant boost to the overall quality of the product.