How an ECS Beginner Migrated Heroku Infrastructure to AWS in 3 Months After 6 Years of Usage

Background and Motivation

In November 2023, I decided to migrate the infrastructure of MAMORIO, a lost-and-found tag prevention service, from Heroku to AWS, and successfully completed the migration on February 5, 2024. As a result, we achieved a cost reduction of approximately 40% for our infrastructure.

While the cost of AWS increased by 25%, the cost of Heroku, which previously accounted for half of our infrastructure costs, became zero.

We were using Heroku Enterprise and operating in the Tokyo region, so there were no particular issues with response speed or quality. However, we considered migration for two main reasons:

1. Cost reduction. While we had already gone to considerable lengths to reduce costs on Heroku, as Heroku is built on top of AWS infrastructure, we estimated that removing it could further compress our costs by a third. The fact that the contract was paid annually in a lump sum also gave the impression of a lack of flexibility.
2. The emergence of the LLM ChatGPT had increased my personal work speed and learning abilities, making the migration more feasible. The initial reason for introducing Heroku was the lack of engineers proficient in infrastructure within the company. While my main knowledge areas were iOS app, Rails, and React development, I was able to gain confidence in my abilities as AI became a sounding board for my AWS-related questions, eliminating the need to hire a technical advisor.

Environment Migration

Previously, we deployed our code to Heroku, but in the new environment, we decided to deploy in containers to ECS and adopt Fargate.

For the database, we chose to use PostgreSQL instead of Aurora to avoid lock-in and ensure portability.

While the free Heroku Scheduler had previously handled our scheduling needs with little conscious effort, after the migration, we had to replace it with ECS scheduled tasks specified in cron format.

For monitoring, we decided to leverage CloudWatch and create custom dashboards. While the NewRelic add-on we used on Heroku was an excellent tool, we could perform the most critical database performance tuning with RDS Insights, so we decided to discontinue its use based on cost considerations.

For managing asynchronous processing queues, we migrated Redis to ElastiCache.

Migration Plan

In our migration plan, we first set up the web server and related services on AWS in advance, and then temporarily stopped Heroku to take a database backup and migrate it using pg_restore.

We considered creating a separate database on AWS to follow the Heroku database for a seamless migration, but it was deemed challenging to establish this relationship spanning both environments. Additionally, as the realistic downtime for migration was deemed acceptable, we opted for the more established and robust restore method.

Based on this assessment, we planned the migration according to the following schedule:

We allowed a 10-day buffer before the renewal date of our annual Heroku Enterprise contract to accommodate potential migration failures and retain the option to continue operating on the current environment if necessary.

Load Testing

In the first stage of load testing, we constructed the environment on AWS and sent only the most processing-intensive requests from the production environment to ensure that performance and costs did not spike.

For MAMORIO, log entries of tag discovery information sent from users’ smartphones accounted for 90% of the system’s overall load, so we focused on testing with this data.

The load test was successful, and we gained confidence that the planned infrastructure configuration would be sufficient for the migration. We also obtained approval from the initially skeptical CEO.

Technical Consultation and Rehearsal

After the successful load test, we decided to receive technical consultation from Amazon engineers and conduct rehearsals.

Technical Consultation

We were able to receive free technical consultation from Amazon by informing them of our planned migration from Heroku.

While LLMs like ChatGPT are helpful, their knowledge is not always up-to-date, and they may not be aware of all the latest AWS services and instance options. Therefore, we sought advice to avoid potential pitfalls.

The consultation provided us with valuable insights. Initially, I had considered using Aurora Serverless for the PostgreSQL instance on RDS, aiming for a serverless solution like the Fargate web server. However, the advisor shared the following article and pointed out that Aurora Serverless might not be cost-effective for services without frequent access spikes.

https://techcon2023.dena.dev/session/session15/

Had we not received this consultation, we might have ended up paying for performance that did not match our needs, as changing the database after migration can be challenging.

We also received guidance on security-related services like AWS GuardDuty and WAF, which we had not paid much attention to while using Heroku, which was extremely helpful.

Rehearsal

We conducted multiple rehearsals to create a procedure for downloading the Heroku database backup and restoring it to the new database, as well as to measure the time required for these steps.

The goal was to eliminate as many considerations as possible and minimize the migration time, ideally reaching a point where we could follow the procedure we created on the migration day without having to think too much.

Through multiple rehearsals, we discovered that by leveraging the fact that Heroku’s private instance in the Tokyo region and AWS’s Tokyo region were on the same network, we could reduce the overall time by more than 70% by downloading the backup to the Heroku instance and directly uploading it to AWS.

This time reduction provided a buffer of less than an hour to restart the process in case of any failures during the time-consuming database restore process, without extending the service downtime.

We also identified that migrating the scheduler would require more effort than initially anticipated, allowing us to address this proactively.

Migration Procedure

After finalizing the migration procedure through rehearsals, we communicated the migration plan to the entire company and had the support team prepare a notification schedule for users.

The migration work began at midnight. We enabled Heroku’s maintenance mode, confirmed that all tasks were complete, and then proceeded with the database backup and restore.

After the work was completed, we changed the domain’s destination, successfully concluding the migration.

Result

While we incurred some costs for setting up and conducting load tests and rehearsals in the two months leading up to the migration, this process provided peace of mind, and we were able to reduce the company’s overall monthly costs by 40%.

Conclusion

Heroku is not a bad service by any means. For organizations with limited personnel, its compact management console and ability to scale environments with simple operations, even for engineers without infrastructure expertise, is a significant advantage. Additionally, Heroku’s add-ons allow for easy functionality enhancements, and its UI is more refined than AWS.

The general notion that startups and small businesses should prioritize acquiring more customers through new development rather than optimizing costs, which may only yield marginal gains, still holds some truth.

However, for those considering migrating from Heroku to AWS, I would strongly recommend receiving technical consultation from Amazon and conducting rehearsals until all uncertainties are eliminated.

First, the Amazon technical consultation can help avoid selecting infrastructure that does not align with your performance needs.

Even engineers who consider themselves AWS experts may have outdated knowledge, as new instance options for services like EC2 and RDS are constantly being introduced. Consulting with experts can help dispel misconceptions (e.g., assuming that Aurora Serverless is a cost-effective option because Fargate is a good, affordable choice).

Additionally, repeatedly rehearsing can shorten the migration time and uncover potential issues in the migration process that may not have been identified during the load testing and technical consultation stages.