Dynamic task scheduling with Redis and NodeJS
In my very first blog, I’m going to show you that how can we use NodeJS with Redis for dynamic tasks to be performed by a service on dynamic timings given by UI.
So, first thing, why I choose the Redis for job scheduling? As we all know that Redis can be a Swiss knife for your backend system. It has so many data structures like PUB/SUB, Streams, List, etc., that can be useful in different kinds of workloads with just a single installation and as it all works as an in-memory data store, it is blazingly fast.
If you’re not familiar with the Redis, I will link the beginner documentation or tutorials links in the description, which can be very helpful to get started with Redis.
For this use case, we will use the Redis Sorted Set data structure. Redis sorted set data structure is the go-to data structure for the leaderboard implementation, but we will use it for job scheduling for our backend here.
So the First command we will use of Redis sorted set is ZADD, which is used to submit any entry to a sorted set.
ZADD key score member
Key can be anything here that represents a specific sorted set, as a score we will use the timestamp of the task that needs to be performed and in the member, we can add the data that needs to perform a task on the backend side
const submitTask = (taskDetails, time) => {
redisClient.zadd('scheduler', time, taskDetails, (err, data) => {
console.log(err, data)
})
}const taskData = {
taskType: 'sms',
details: {
to: 'Boss',
message: 'Still working :)'
}
}submitTask(
JSON.stringify(taskData),
Math.floor(+new Date('2020-05-01 22:00:00') / 1000)
)
Like here I’ve submitted a task to send my boss an SMS at 11 PM that I’m still working ;-)
ALERT!! There is a chance to get fired based on worksheet :’(
Now comes the actual thing that we need to implement. How can we get the tasks that are needs to be performed now, so for that we will use Redis command ZREVRANGEBYSCORE
ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count]
When needed to pass the key with min & max parameters, so for the key, we will be passing the same key that we passed in the ZADD, and as the max parameter, we will pass the current timestamp, and as min, we will pass 0.
We can specify WITHSCORES parameters to get the timestamp of that tasks with the task details or else we will only get the task details which we have passed as a member, and we can pass LIMIT and OFFSET count to limit the results per call and cover tasks one by one..
const pollScheduler = () => {
redisClient.zrevrangebyscore([
"scheduler",
Math.floor(+new Date() / 1000),
0,
"WITHSCORES",
"LIMIT",
0,
1
], (err, data) => {
data = data[0]
if (!data) {
setTimeout(() => { pollScheduler() }, 1000)
return
}
data = JSON.parse(data)
console.log(data)// send the task details to servicepollScheduler() // fetch next task
})
}
Now when we have task details, we can do the action by the details specified in the task details. It can be a direct function call or it can be an enqueue operation to a queue in which some other service is performing the queue operations, it can be a Publish call to a PUB/SUB channel listening by a Subscribers, it can be a stream add operations.
Like here I’ve submitted that task to Redis list which is a go-to solution for queue service.
redisClient.rpush(
data.taskType,
JSON.stringify(data.details)
)Now the main operation to do is remove that processed entry from the sorted set for which we will use command ZREM
ZREM key member
Here key will be our scheduler key and the member will be our data as it was submitted
redisClient.zrem('scheduler', JSON.stringify(data), (err, data) => {
console.log(err, data)
})Here below I have mentioned a Repo of NodeJS with ExpressJS implementing this feature.
Below I have mentioned a Repo of NodeJS with ExpressJS implementing this feature.
We can use this one as a microservice. As a separate service, it can be a go-to solution that can be integrated into many systems, Use cases of this kind of scheduler are endless in many possible ways, Let me know your use case in the comments.
Hope you liked my very first blog and it will be helpful to you, we can also make the same functionality with Redis streams also but that’s a whole another blog itself, And yes don’t forget to Clap Because Clapping is good for health. 😉
Resources:
