The Migration Journey From Self-Managed MongoDB to DocumentDB @SSENSE
Managing database systems on EC2 instances comes with its own set of challenges and MongoDB is no exception. From handling operational overhead to ensuring security and scalability, organizations often face significant hurdles when maintaining their self-managed MongoDB environments. In this article, we’ll explore the SSENSE migration journey from self-managed MongoDB clusters to AWS managed service offering Amazon DocumentDB (with MongoDB compatibility). We’ll dive into the advantages of migrating to DocumentDB, discuss the migration process in detail, and highlight lessons learned along the way.
MongoDB @SSENSE
At SSENSE, we have been building and running applications that use MongoDB as storage since 2016. As the adoption of MongoDB grew within the company, manual creation and management of the database clusters was no longer feasible.
The SSENSE DevOps team used a combination of tools such as Terraform (for provisioning) and SaltStack (for configuration management) to automate the creation and maintenance of the self-managed MongoDB clusters that ran on top of AWS EC2 instances.
One of the key challenges with self-managed MongoDB clusters was the operational overhead. Although the setup was mostly automated, edge cases would involve manual intervention. Any troubleshooting would require deep expertise in MongoDB and once again manual involvement was unavoidable. Non-functional requirements such as backup automation and patch management were not uniformly implemented across all the clusters, which created a potential risk for the business.
The final nail in the coffin of the self-managed MongoDB clusters was the new non-functional requirements that required the implementation of encryption at rest and in transit, as well as the need to externalize and rotate database credentials. All of the above wouldn’t be possible or easily achievable without a significant investment to rewrite the existing automation and migrate the data to new conformant clusters.
What is Amazon DocumentDB?
At SSENSE, we believe certain cloud-native off-the-shelf products can provide resilient and scalable solutions, and can free up our engineers’ valuable time, allowing them to focus on bringing further capabilities to our e-commerce customers. As most of our infrastructure runs on AWS, adopting Amazon DocumentDB was a natural choice.
The main advantage of Amazon DocumentDB is the reduction in operational overhead. DocumentDB is a fully managed service, eliminating the need for manual setup, patching, and backups.
Security is a top priority for any database solution, and DocumentDB offers enhanced security features. Encryption at rest ensures that our data is safeguarded in storage, TLS encryption in transit ensures that the data is transmitted securely, and network isolation helps protect against unauthorized access.
Scalability and availability are also critical, particularly for a growing business. DocumentDB provides independent scaling of storage (up to 64TB) and compute (up to 15 replicas in a cluster), allowing us to scale the database resources separately as needed. DocumentDB also employs automated failover mechanisms, ensuring maximum availability with minimal downtime during hardware or software failures.
The following diagram from AWS documentation illustrates the decoupled architecture of the DocumentDB cluster.
Executing the Migration Process
At SSENSE, we opted for an “Online Migration” approach in order to minimize downtime and data synchronization risk. A migration project typically requires careful planning and execution across several steps. The following diagram illustrates the high-level steps involved during the migration.
First, set up a new DocumentDB cluster in the desired AWS region as per AWS best practices. Once the cluster is up and running, configure the necessary security rules and access controls to ensure a secure environment.
Next, data migration can be accomplished using the AWS Database Migration Service (DMS). DMS enables full load as well as continuous replication from the self-managed MongoDB cluster to DocumentDB. During the migration, it’s crucial to monitor the replication process and address any issues promptly.
The DocumentDB Index Tool from Amazon DocumentDB Tools is a valuable resource for re-creating indexes on DocumentDB (as DMS only does data migration). To create the index in DocumentDB, we used the DocumentDB Index Tool to dump and restore the indexes on the target database.
Thorough testing is vital to the success of the migration. It’s important to conduct load testing to evaluate DocumentDB’s performance under realistic conditions. Proper node sizing considerations should also be made to ensure optimal resource allocation for the new cluster. The following tool can help with sizing and cost estimation: https://sizing.cloudnativedb.com/.
Finally, if the application architecture permits, switching to the DocumentDB endpoint can be done gradually by changing the read endpoint and subsequently the write endpoint. If application architecture doesn’t permit, the change would be a single atomic change of the endpoint to point to the DocumentDB cluster endpoint. At SSENSE, application architectures required a single atomic change. Application owners booked a maintenance window during the time of least connections to the MongoDB database and the switchover of the endpoints was performed with close monitoring of the production workloads.
Lessons Learned
While migrating to DocumentDB brings numerous benefits, it’s essential to be aware of potential challenges and important considerations.
Architectural differences between DocumentDB and MongoDB at the storage layer may impact application performance. Latency considerations, such as disk IO and network round trips, should be taken into account (we observed 2–4ms latency increases after the migration). It’s vital to perform thorough testing before the switchover. In addition, storage layer differences might impact the infrastructure cost as DocumentDB charges per I/O operation against the storage layer. Extra attention should be paid to DB index optimization to minimize the amount of reading and writing to your cluster’s storage volume.
Recreating indexes on DocumentDB requires special attention. Some index types, supported by MongoDB, may not be available or may have different behaviors in DocumentDB. It’s crucial to thoroughly analyze and test index functionality during the migration process.
DocumentDB emulates MongoDB APIs. Some MongoDB commands and operations might not be available in DocumentDB. You can use the following tool to check API compatibility. It’s important to test your application during the migration process to identify and fix any incompatibilities.
Connect to the DocumentDB cluster as a Replica Set and take advantage of the secondaryPreferred option to start using read-replicas without modifying the logic of your application to separate reads from writes.
A brief outage may occur during the DocumentDB failover. This is a normal part of the failover process and can be mitigated by setting appropriate expectations with stakeholders and implementing retry mechanisms in applications.
Final Thoughts
Migrating from self-managed MongoDB clusters on EC2 to Amazon DocumentDB offers numerous advantages, including reduced operational overhead, enhanced security, and improved scalability and availability. With the automated management of routine tasks, Tech teams can focus on their core business features.
At SSENSE, we completed the migration of 14 self-managed MongoDB clusters on EC2 to AWS DocumentDB, with no customer impact, within 3 months. By following a well-defined migration process and considering potential challenges, we were able to seamlessly transition from self-managed MongoDB clusters to DocumentDB, while minimizing downtime and ensuring optimal performance.
Next Steps
To learn more about Amazon DocumentDB, visit the official AWS website and documentation: https://aws.amazon.com/documentdb/, https://docs.aws.amazon.com/documentdb/latest/developerguide/what-is.html
Explore AWS resources and tutorials to gain hands-on experience with DocumentDB: https://documentdb-immersionday.workshop.aws/
Editorial reviews by Catherine Heim & Mario Bittencourt
Want to work with us? Click here to see all open positions at SSENSE!
