Reducing Costs and Improving Efficiency: Transitioning from AWS RDS to a Self-Hosted Queue System

At Internshala, we compute digests for students to send them highly accurate recommended internships and jobs, making it easy for them to apply through our platform.

To generate accurate recommendations, our cron jobs perform heavy computations. Doing this in real-time while sending emails is not feasible, as it would slow down the rate at which we send digest emails to all our users. Instead, we compute the entire digest every morning around 4 AM for approximately 1 million users and store the content in AWS RDS. We then send these digest emails later in the day using AWS SES.

Internshala’s internal server handles heavy computations for accurate recommendations, storing results in AWS RDS for later email dispatch using AWS SES.

The Problem

Storing the computed recommendations in RDS posed several issues:

1. IOPS: Inserting data into RDS during computation, reading the data while sending emails, updating fields for sent mails and retries, and finally deleting these emails at the end of the day collectively spiked our IOPS to 4 million requests. To manage our IOPS requests, we implemented batching while inserting and deleting data, and even truncating the table if all the emails are sent, which helped reduce many IOPS requests to RDS.
2. Storage Costs: Storing email content in the database can reach up to 65 GB of data. This increased our storage costs significantly and led to more expenses for database snapshots. I recall one instance when a permission issue prevented the truncate command from working, causing the table size to grow to over 500 GB.

These daily costs accumulated and eventually resulted in high billing expenses.

Exploring Solutions

To mitigate these issues, we researched implementing a self-hosted queue instead of using managed services like RDS. Here are the approaches we considered:

• AWS SQS: This option did not suit our use case since we needed to store heavy email content for a longer period, whereas SQS can hold data for a maximum of 15 minutes.
• RabbitMQ & Apache Kafka: These options showed promise, and we are still evaluating them for a long-term solution.

The Interim Solution: Self-Hosted MySQL

For a quick solution with minimal changes — what we call minimum effort versus high value — we decided to self-host a MySQL server on our EC2 instance. Here’s how we implemented it:

1. Setting Up MySQL: We installed the MySQL server on our EC2 instance.
2. Configuring Doctrine: We adjusted Doctrine’s DB connection (the ORM used in our PHP application) to connect to the new MySQL server.
3. Process Optimization: We continued to process all digests using cron jobs, but now saved the data in the local MySQL database. When the time came to send emails, we retrieved the data and sent it via SES.

Internshala’s internal EC2 server runs application logic to process digests, storing data in a MySQL database on the same EC2 instance for optimized performance and efficiency.

Benefits Achieved

This approach provided several immediate benefits:

1. Reduced IOPS: By processing and storing data locally, we significantly reduced the IOPS load on our system.
2. Lower Storage Costs: Moving away from RDS eliminated the high storage costs associated with it.

Next Steps

We plan to create a dedicated EC2 instance for this task. Once our research for the best long-term solution concludes, we will implement it on this new EC2 along with all necessary monitoring, calling it our queue microservice.

Go-Live Date: 5th June 2024

Thank you for reading! I hope you found this article helpful. If you have any questions or suggestions, please leave a comment. Your feedback helps me improve.