Migrating Operational DB to the cloud without downtime

This post is part of the series on doing safe database migrations using the 4-phase approach. Other posts in the series are (a) Rearchitecting a SpringBoot application for database migration (b) Data streaming from DynamoDB to ElasticSearch (c ) Data streaming from DynamoDB at scale to ElasticSearch.

Risk free data migration explains the 4-phase approach.

4-phase data migration

We want to migrate all our existing users data from Cassandra to DynamoDB because we prefer managed databases now. We rearchitected the application so it can use both databases in the 4-phase data migration approach.

As a refresher, here are the 4 phases:

(i) Dual writes to Cassandra & DynamoDB, Read from Cassandra. Turn on migration job

(ii) After migration is complete, Dual writes to Cassandra & DynamoDB, Read from DynamoDB

(iii) All data is migrated and verified. Write to DynamoDB, Read from DynamoDB

(iv) Delete Cassandra

We are entering phase (i) of dual writes. Our dual writes mechanism is set up. Next step is to turn on a migration job that will systematically move all data from Cassandra to DynamoDB. There are several approaches to do this. If we were allowed to take downtime, we could scale up Cassandra and DynamoDB, then run EMR jobs that would read and transform the data and output them on the respective DynamoDB…