How we scaled services containing scheduled tasks
CasaOne is a technology-powered contemporary furniture rental company with a mission to deliver the best rental experience to our customers. Our software services run a lot of scheduled tasks periodically in the background to ensure the inventory data is accurate, send email and text notifications to keep the customers informed about the furniture subscription, delivery, pickup, etc.
During the early days of the startup, we had started with simple in-process cron schedulers to get up and running fast. It looked like this
As we started serving more customers and cities, this approach didn’t scale well. We couldn’t run multiple instances of service, as it would have led to undesirable duplicate executions of the tasks. This hampered availability of our services, during peak load and service upgrades
We had to find a solution to run multiple instances which coordinated to schedule these tasks only once as desired. We couldn’t find any out-of-the-box library for distributed scheduling across multiple instances. Building and managing our own solution for distributed scheduling didn’t seem like the best investment of our time
We took a step back and started breaking the problem into 2 parts
- Scheduler for triggering the tasks at periodic intervals
- Distributed workers executing tasks from a task queue
This decomposition simplified it to 2 smaller problems with well-known architecture patterns and solutions. Part 2 of the problem, got reduced to the way we run other background tasks in the system using AWS Simple Queue Service(SQS). For part 1 of the problem, we used AWS CloudWatch rules to publish messages to SQS task queues at periodic intervals.
The final solution looks like this
We wanted to ensure this solution is easier to work with during development and deployment. Using localstack during development and Terraform for AWS infrastructure automation helped in achieving this goal. As an added benefit, force triggering any of these tasks was as easy as publishing a message to the task queue
This engineering effort to compose existing solutions saved us from a lot of development effort and helped us provide highly available service to our customers!
We are looking for great engineers to join our team. Checkout latest jobs here
