Building a Scheduler with Redis

Diego
Diego
Apr 30, 2018 · 3 min read

At busuu - a social network for language learning 👫👭- we’ve recently built a simple yet functional friend suggestion system. Typically, in this sort of social network, you might want to find someone who speaks the language you learn and ideally learns the language you speak. We call this a language pair

When you build product features in a fast-paced startup, you need to iterate, measure and adapt to change quickly. That’s why we strive for simplicity when building things 💡


Overview

Our friend suggestion system, is designed to be a relatively simple State Machine 🤔 Users can be in different states. They are not always eligible to be suggested as friend to someone. But they do only and after certain conditions are met (they have been active for certain days, have completed certain interactions, …)

Example of states

Users before being able to be suggested as friend needs to be scheduled for a certain period of time, and they get scheduled only when certain conditions are met. We’ve then decided to build a simple task’s (user) scheduler and we did that using Redis, a popular In-Memory data store

I like Redis, because it follows the same philosophy as Unix, it does one thing, and does it well. It offers at the same time a set of useful data structures and one of those is a Redis Sorted Set

Scheduler ⏱

Using a Redis Sorted Set is everything you might need to build a task scheduler. A Scheduler has the following primitives

  • schedule(task_id, expire_time)
  • get_expired_tasks()

Scheduling a user is that simple

import redis
r = redis.ConnectionPool(host='1.2.3.4', port=6379, db=0)
r.zadd('scheduled_users', value=user_id, score=time.time())

Polling the expired users, which will be now moved to the “recommendable” state. This code should be runned every few minutes, to check whether there are new expired users

expired = r.zrangebyscore('scheduled_users', '-inf', time.time())

Is there any downside? ⚠️

While using this approach you need to consider that adding items to the sorted set it’s a bit more expensive than just adding item to a standard Redis Set (O(1) vs O(logN)), because Redis needs to re-order part of the items

Trying to know if an item is into the Sorted Set is really costly O(N), it will need to scan ZSCAN the entire Redis Sorted Set to do so! 🤕

If you really need to check if an item is currently scheduled you might want to add it somewhere in the moment you schedule it (see step 1). Where it can be easily retrieved, like using a Redis HashMap

r.zadd('scheduled_users', value=user_id, score=time.time())
r.hset('scheduled_users_quick_lookup', user_id, time.time())

But remember to remove it when you remove the items from the Sorted Set, the two data structures must be consistent! 😠

Thanks for reading and go build your scheduler with Redis now! 😄


WengVox

WengVox is a blog series about software engineering

Diego

Written by

Diego

http://diego.pizza

WengVox

WengVox

WengVox is a blog series about software engineering

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade