How we scaled services containing scheduled tasks

CasaOne is a technology-powered contemporary furniture rental company with a mission to deliver the best rental experience to our customers. Our software services run a lot of scheduled tasks periodically in the background to ensure the inventory data is accurate, send email and text notifications to keep the customers informed about the furniture subscription, delivery, pickup, etc.

During the early days of the startup, we had started with simple in-process cron schedulers to get up and running fast. It looked like this

As we started serving more customers and cities, this approach didn’t scale well. We couldn’t run multiple instances of service, as it would have led to undesirable duplicate executions of the tasks. This hampered availability of our services, during peak load and service upgrades

We had to find a solution to run multiple instances which coordinated to schedule these tasks only once as desired. We couldn’t find any out-of-the-box library for distributed scheduling across multiple instances. Building and managing our own solution for distributed scheduling didn’t seem like the best investment of our time

We took a step back and started breaking the problem into 2 parts

  1. Scheduler for triggering the tasks at periodic intervals
  2. Distributed workers executing tasks from a task queue

This decomposition simplified it to 2 smaller problems with well-known architecture patterns and solutions. Part 2 of the problem, got reduced to the way we run other background tasks in the system using AWS Simple Queue Service(SQS). For part 1 of the problem, we used AWS CloudWatch rules to publish messages to SQS task queues at periodic intervals.

The final solution looks like this

We wanted to ensure this solution is easier to work with during development and deployment. Using localstack during development and Terraform for AWS infrastructure automation helped in achieving this goal. As an added benefit, force triggering any of these tasks was as easy as publishing a message to the task queue

This engineering effort to compose existing solutions saved us from a lot of development effort and helped us provide highly available service to our customers!

We are looking for great engineers to join our team. Checkout latest jobs here

--

--

--

Engineering efforts to convert furnitures to utility

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Deepak Narayana Rao

Deepak Narayana Rao

Human being running on curiosity

More from Medium

Setting up Load balancer, Docker, Kubernetes in your local — System Design (part 5)

Running Serverless DynamoDB Plugin on M1 MacOS Monterey

Authentication & Authorization, Hashing in Web Applications

Deploy Node.js application over Google Cloud with CI/CD