AWS Account Migration: From Single Account to Account per Environment Architecture
The article explains the process that a company — Track160 — went through to migrate from a single AWS account architecture with three environments (develop, staging, and production) to an account-per-environment architecture. The new architecture would provide better isolation and control over infrastructure, as well as enable better use of AWS’s multi-account strategy. The article details the benefits of this architecture, such as enhanced security, streamlined resource management, and improved cost optimization. The article also explains the tools used for migration, such as Terragrunt, AWS DataSync, and AWS Certificate Manager, and the migration journey from updating DNS to cross-account data migration. Overall, the article provides insights and guidance for anyone considering a similar migration process.
“From One to Many: Our AWS Account Migration Journey”
Migrating to the cloud can be a daunting task, especially when dealing with a complex infrastructure that includes multiple environments. “Track160” recently faced this challenge when we realized that our existing architecture, consisting of a single AWS account with three environments (develop, staging, and production), was no longer sufficient for our needs. This setup made it difficult to manage costs, enforce security, and scale our applications, prompting us to seek a more advanced solution.
After conducting thorough research and consulting with AWS experts, we decided to migrate to an “account per environment” architecture, where each environment would have its own AWS account and resources. This approach would provide us with better isolation and control over our infrastructure, as well as enable us to take advantage of AWS’s multi-account strategy.
In this article, we will detail the process we followed to migrate our production environment to the new architecture. We will explain the challenges we encountered, the tools we used, and the lessons we learned along the way.
Benefits of Account per Environment Architecture
Enhanced security
Moving from a single account architecture to an account per environment architecture allowed us to improve the security of Track’s AWS environment. By allocating a separate account to each environment, we eliminated the risk of data breaches across accounts. Additionally, this architecture facilitated access control and permissions management for each environment, increasing security.
Streamlined Resource Management
The account per environment architecture provided Track160 with better visibility into the resources used in each environment. This structure made it easier to manage and allocate resources, such as EC2 instances, RDS databases, and S3 buckets. We were able to optimize resource usage and reduce costs.
Improved Cost Optimization
The account per environment architecture allowed for more precise cost tracking for each environment. Track160 could monitor their spending on each account and set budgets for each environment, which was not feasible with the previous architecture. By splitting the costs across different accounts, the client was able to optimize their expenses and manage their finances more effectively.
Migration Tools
During the process of migrating a single AWS account with three environments to an account-per-environment architecture, we utilized several tools to streamline the process and ensure a smooth transition. The following tools served specific purposes in the migration process:
- Terragrunt: A tool used for managing infrastructure code and creating AWS resources in each account.
- MongoDB: A database management system used for transferring data from the old environment to the new environment.
- AWS DataSync: A data transfer tool used for transferring data from one S3 bucket to another.
- AWS Route 53: A DNS management tool used for routing traffic to the new environment.
- AWS Certificate Manager: A certificate management tool used for generating SSL/TLS certificates for the new environment.
- AWS Elastic Load Balancer: A load balancing tool used for distributing traffic across EC2 instances in the new environment.
- AWS CloudFront: A content delivery and caching tool used for the new environment.
- AWS S3: A tool used for storing static files and backups.
- AWS EFS: A file system tool used for storing shared files between EC2 instances.
- AWS IAM: An access management tool used for securely controlling access to AWS resources.
- AWS CLI: A command-line interface tool used for automating tasks and managing resources.
Each tool played a vital role in the migration process, from managing infrastructure code and databases to transferring data and securely controlling access to AWS resources. In the following sections, we will delve into the details of each tool and how we used it to accomplish the migration.
Migration Journey
Setup description
For better understanding — below is an overview directory tree of the Terragrunt module
├── develompent
│ └── us-east-1
│ ├── acm
│ ├── alb_internal
│ ├── alb_public
│ ├── app_db_proxy
│ ├── app_import_api
│ ├── app_import_lb
│ ├── app_mobile_api
│ ├── app_solver_orchestrator
│ ├── app_tactical_api
│ ├── app_terrarium_frontend
│ ├── app_terrarium_main_worker
│ ├── app_terrarium_workers
│ ├── aws_backup
│ ├── cloudfront_frontend
│ ├── cloudfront_mobile_api
│ ├── cloudfront_tactical_api
│ ├── efs
│ ├── kms
│ ├── pipelines
│ ├── redis
│ ├── route53_records_private
│ ├── route53_records_public
│ ├── route53_zones
│ ├── s3
│ ├── sg
│ ├── sqs
│ ├── vpc
│ ├── vpc_endpoints
│ ├── vpc_peering
│ └── waf_logging
├── production
│ └── us-east-1
│ ├── ...
├── staging
│ └── us-east-1
│ ├── ...
└── wrappers
├── ...DNS Update
After creating the environments, our systems were tested on [*.prod.track160.co](http://prod.example.com/)m.
At that point, we needed to switch the well known application urls to point to the new ones -
The DNS update process was a critical step in the migration process as it involved updating the domain name system to point to the new production environment. The goal was to change the domain name from “prod.example.com” to “example.com” while ensuring minimal downtime and avoiding any potential loss of data.
To accomplish this, we began by editing the “production/us-east-1/regional.hcl” file and replacing all instances of “ prod.track160.com” with “ track160.com”. This ensured that the new environment was properly aligned with the updated domain name.
Next, we manually removed the “prod.track160.com” and “internal.prod.track160.com” hosted zones from the production account using the AWS Management Console. This ensured that the old DNS entries were no longer active and could not interfere with the new environment.
Using Terragrunt, we then applied the “route53_zones” module to recreate the hosted zones, taking note of the name servers for “track160.com” in the zone output to be used later in the process.
In the old account, all the DNS records were exported and imported into the “track160**.com” hosted zone. The record type was set to NS, the name was set to “track160. com**”, and the values were set to the name servers from the previous step. This ensured that all DNS records were transferred to the new environment and that the domain name was properly pointed to the new environment.
As the final step in the migration process, we used Terragrunt (run-all apply) to apply all the remaining changes, which included reissuing the SSL certificate, reconfiguring the applications to work with the new AWS accounts, and changing the load balancer rules to ensure proper routing of the traffic.
The SSL certificate was reissued using AWS Certificate Manager and assigned to the new environment’s load balancer. We made sure that the certificate was properly configured to ensure secure communication between the client and the server.
We then reconfigured the applications to work with the new AWS accounts by updating the database connection strings, API endpoints, and other configuration parameters. Using Terragrunt, we applied these changes across all the environments, ensuring consistency and reliability.
Finally, we changed the load balancer rules to ensure that traffic was properly routed to the new environment. We used AWS Elastic Load Balancer to balance the traffic across the EC2 instances in the new environment, ensuring high availability and scalability.
By using Terragrunt to apply all the remaining changes, we were able to ensure a smooth and efficient migration process. The client’s applications were now running smoothly in the new environment, and they were able to take advantage of the improved security, resource management, and cost optimization benefits of the new architecture.
In conclusion, the migration from a single account architecture to an account per environment architecture was a significant undertaking, but it ultimately resulted in significant benefits for the client. If you are considering a similar migration, it is crucial to plan and execute the process carefully to avoid any disruptions.
Cross Account Data Migration
To migrate data between cross-account S3 buckets and EFS, we utilized AWS DataSync, a data transfer tool. The process involved setting up an agent on each account, configuring the source and destination locations, and performing the data transfer. AWS DataSync allowed us to transfer data across AWS regions and accounts securely and efficiently. To learn more about how we used AWS DataSync in our migration process, you can check out this link for more about cross account and AWS DataSync: https://aws.amazon.com/blogs/storage/transferring-file-data-across-aws-regions-and-accounts-using-aws-datasync/
Data migration was a crucial step in the overall migration process of our client’s AWS architecture. It was a critical part of the process as it involved transferring all the necessary data from the old environment to the new environment without any loss of data. We had to plan and execute this step carefully to ensure minimal downtime and avoid any potential issues.
One of the primary considerations we had to take into account was how to optimize downtime. We had a 40TB file system that included the database and various types of binaries, such as videos and PDFs. This made it impossible to move the 20GB database files while the system was live, and therefore, we had to schedule a downtime for this particular task. We carefully planned the downtime to minimize any potential impact on the client’s operations.
In addition to scheduling downtime, we also had to copy some of the database and binary files in advance to allow for testing in the QA environment. This allowed us to ensure that the system was working correctly before migrating it to the production environment. We made sure that the sync between these files and the production environment was properly maintained to avoid any potential issues.
Overall, we took great care to ensure that the data migration process was smooth and efficient, and we successfully transferred all the necessary data to the new architecture without any issues. We utilized several tools, including AWS DataSync, to transfer the data securely and efficiently across AWS regions and accounts.
In conclusion, the data migration process was a critical part of our client’s migration to an “account per environment” architecture. It was a complex process that required careful planning and execution to ensure minimal downtime and avoid any potential loss of data. However, with our careful planning and execution, we were able to transfer all the necessary data to the new architecture successfully, allowing the client to take advantage of the improved security and resource management benefits of the new architecture.
Overall and Conclusion
Once the development environment was successfully migrated, we moved on to the staging environment, which was more complex. We followed the same process as with the development environment, but we had to be more careful not to disrupt the development process. Finally, we migrated the production environment, which required the most attention to avoid any additional downtime.
Track160’s migration from a single AWS account architecture to an account per environment architecture was a significant undertaking, but it ultimately resulted in significant benefits for the client. The new architecture provided enhanced security, streamlined resource management, and improved cost optimization. We utilized several tools, including Terragrunt, MongoDB, AWS DataSync, AWS Route 53, AWS Certificate Manager, AWS Elastic Load Balancer, AWS CloudFront, AWS S3, AWS EFS, AWS IAM, and AWS CLI, to streamline the migration process and ensure a smooth transition.
