How AWS Data Migration Service Helps

What is Data Migration service?

AWS Database Migration Service (AWS DMS) is a cloud service that makes it possible to migrate relational databases, data warehouses, NoSQL databases, and other types of data stores. You can use AWS DMS to migrate your data into the AWS Cloud or between combinations of cloud and on-premises setups.With AWS DMS, you can discover your source data stores, convert your source schemas, and migrate your data.

Migration Process
Plan → Assess → Convert → Migrate

At a high level, when using AWS DMS you do the following:

1. Create a replication server.
2. Create source and target endpoints that have connection information about your data stores.
3. Create one or more migration tasks to migrate data between the source and target data stores.

A task can consist of three major phases:

1. The full load of existing data
2. The application of cached changes
3. Ongoing replication

How DMS Works?

AWS Database Migration Service (AWS DMS) is a managed migration and replication service that helps move your database and analytics workloads to AWS quickly, securely, and with minimal downtime and zero data loss. AWS DMS supports migration between 20-plus database and analytics engines.

The only requirement to use AWS DMS is that one of your endpoints must be on an AWS service. You can’t use AWS DMS to migrate from an on-premises database to another on-premises database.

There are two types of Database Migration topologies

Homogeneous: The source and destination databases are same type. Example: Oracle to Oracle, Postgres to Postgres.

Heterogeneous: The source and destination databases are different type. Example: Oracle to Postgres, MySql to Postgres.

Prerequisite

1. VPC and subnets for EC2 Replication instance.
2. Security Group to allow access to Replication EC2 Instance. There is no specific port requirement.
3. Create the source Postgres database instance and store the credentials in the secret manager.
4. Create a IAM role that grants Amazon DMS permissions to access the specified secret (and any required KMS encryption key)
5. Create the target Postgres database instance and store the credentials in the secret manager.
6. S3 bucket and folder to store pre migration assessment report.
7. IAM role to access S3 bucket.

In this demo we are migrating from Postgres to Postgres. There is no requirement for SCT(Schema conversations tool).

For Heterogeneous migration, AWS SCT or DMS Schema Conversion to convert your source database schemas.

During the Database Creation click store the credentials to Secret Managers

Source Database

Destination Database

Bucket to store pre migration assessment report.

IAM Role for secret manager and S3 Access.

Steps to Migrate data from Postgres to Postgres

Step 1: Create a replication instance using the AWS DMS console

A replication instance performs the actual data migration between source and target endpoints. Your instance needs enough storage and processing power to perform the tasks that migrate data from your source database to your target database. How large this replication instance should be depends on the amount of data to migrate and the tasks your instance needs to do.
To create a replication instance using the console

1. Sign in to the AWS Management Console and open the AWS DMS console.
2. On the navigation pane, choose Replication instances, and then choose Create replication instance.
3. On the Create replication instance page, specify your replication instance configuration:

• For Name, enter DMS-instance. (Follow the EC2 instance naming convention)For Name, enter DMS-instance. (Follow the EC2 instance naming convention)
• For Description, enter a short description for your replication instance (optional).

• For Instance class, leave dms.t3.medium chosen.
  The instance needs enough storage, networking, and processing power for your migration. For more information about how to choose an instance class, see Choosing the right AWS DMS replication instance for your migration.
• For the Engine version, accept the default.
• For Allocated storage (GiB), accept the default of 50 GiB.
  In AWS DMS, storage is mostly used by log files and cached transactions. For cache transactions, storage is used only when the cached transactions need to be written to disk. Thus, AWS DMS doesn’t use a significant amount of storage.
• For VPC, choose the VPC name.

4. Choose the Advanced security and network configuration tab to set values for network and encryption settings if you need them:

• For the Replication subnet group,Choose the subnet from the VPC chosen above.
• For Availability zone, choose us-east-1
• For the VPC security group(s), choose the created replication instance security group if it isn’t already chosen.
• For AWS KMS key, leave (Default) aws/dms chosen.

5. Leave the settings on the Maintenance tab as they are. The default is a 30-minute window selected at random from an 8-hour block of time for each AWS Region, occurring on a random day of the week.

6. Choose Create.

AWS DMS creates a replication instance to perform your migration.

Replication Instance

Step 2: Specify source and target endpoints

To specify a source endpoint and database endpoint using the AWS DMS console

1. On the console, choose Endpoints from the navigation pane and then choose Create Endpoint.
2. On the Create endpoint page, choose the Source endpoint type.
3. In the Endpoint configuration section, enter a name for Endpoint identifier.(like dev-postgres-database)
4. For Source engine, Choose database engine type Postgres (Like mysql, postgres,oracle)
5. For Access to endpoint database, (Choose any one of the option below) — Choose AWS Secrets Manager , provide the value Secret ID, IAM role which you have created before. or Choose Provide access information manually. Verify that the Port, Secure Socket Layer (SSL) mode, User name, and Password are correct.
6. Choose the Test endpoint connection (optional) tab. For VPC, choose the VPC name.
7. For Replication instance, Choose the replication Instance created before.
8. Choose Run test. After you choose “Run test”, AWS DMS creates the endpoint with the details that you provided and connects to it. If the connection fails, edit the endpoint definition and test the connection again. You can also delete the endpoint manually.
9. After you have a successful test, choose “Create endpoint”.
10. Specify a target database endpoint using the AWS DMS console. To do this, repeat the steps preceding, with the following settings:

• Endpoint type: Target endpoint
• Choose database engine type Postgres
• Endpoint identifier: enter a name for Endpoint identifier.(like dev-postgres-database)
• Target engine: Choose database engine type Postgres
• Repeat step 5.

Endpoint connection test must pass to Run the Migration Task.

both endpoints created

When you have finished providing all information for your endpoints, AWS DMS creates your source and target endpoints for use during database migration.

Step 3: Create a task and migrate data

To create a migration task and start your database migration:

1. In the console navigation pane, choose Database migration tasks, and then choose “Create task”. The Create database migration task page opens.
2. In the Task configuration section, specify the following task options:

• Task identifier: Enter a name for task (like dev-database-replication-task)
• Replication instance: Choose the created replication instance.
• Source database endpoint: Choose the source endpoint created before
• Target database endpoint: Choose the target endpoint created before
• Migration type: Choose Migrate existing data and replicate on-going changes.

3. Choose the Task settings tab. Set the following settings:

• Target table preparation mode: Do nothing
• Stop task after full load completes: Don’t stop

4. Enable Validation (Optional) : If you want AWS DMS to compare the data between the source and the target immediately after it performs a full data load, then enable validation. Validation ensures that your data was migrated accurately, but it requires additional time to complete.

5. Enable CloudWatch logs : DMS task logging uses Amazon CloudWatch to log information during the migration process.

6. Choose the Table mappings tab, and expand Selection rules. Choose Add new selection rule. Set the following settings: 1 Rule needed. You can include and exclude the schemas.

• Schema: Enter a schema
• Schema name: dms_sample

7. Choose the Pre Migration assessment tab (optional), Enable pre migration assessment run : A pre migration assessment warns you of potential migration issues before starting your migration task. Premigration assessments generally have minimal impact on your databases and take minimal time to run.

• Bucket Name to store the report
• Bucket folder
• IAM role to access S3 bucket.

8. Choose the Migration task startup configuration tab, and then choose Manually later.

9. Choose Create task.

Pre-Migration Assessment report. Stored in S3 Bucket.

Migration is successful.

AWS DMS then creates the migration task and starts it. The initial database replication takes about 10 minutes. Make sure to do the next step in the tutorial before AWS DMS finishes migrating the data.

Step 4: Test replication

To test replication

1. While your migration task is running, you can monitor the progress of your database migration as it happens:

• In the DMS console navigation pane, choose Database migration tasks.
• Choose Task Name created before
• Choose Table statistics.

2. After replication completes, query the target database for the records. Verify

Pricing

All data transfer into AWS Database Migration Service is free, and data transferred between AWS Database Migration Service and databases in Amazon RDS and Amazon EC2 instances in the same Availability Zone also is free. Standard AWS data transfer rates apply when you migrate your source database to a target database in a different Availability Zone, Region, or outside of AWS.

AWS DMS Schema Conversion (AWS DMS SC) is a free-to-use, fully managed feature of AWS Database Migration Service (AWS DMS) — only pay for the storage used.

Standard pricing apply for the replication Instance used.

Monitoring

Monitoring is an important part of maintaining the reliability, availability, and performance of AWS DMS and your AWS solutions.AWS provides several tools for monitoring your AWS DMS tasks and resources, and responding to potential incidents:

AWS DMS events and notifications: AWS DMS uses Amazon Simple Notification Service (Amazon SNS) to provide notifications when an AWS DMS event occurs, for example the creation or deletion of a replication instance. AWS DMS groups events into categories that you can subscribe to, so you can be notified when an event in that category occurs.

Task status: You can monitor the progress of your task by checking the task status and by monitoring the task’s control table. Task status indicates the condition of a AWS DMS task and its associated resources.

Amazon CloudWatch alarms and logs: Using Amazon CloudWatch alarms, you watch one or more task metrics over a time period that you specify. If a metric exceeds a given threshold, a notification is sent to an Amazon SNS topic. CloudWatch alarms do not invoke actions because they are in a particular state.

Time Travel logs: To log and debug replication tasks, you can use AWS DMS Time Travel. In this approach, you use Amazon S3 to store logs and encrypt them using your encryption keys. You can retrieve your S3 logs using date-time filters, then view, download, and obfuscate logs as needed. By doing this, you can “travel back in time” to investigate database activities.

AWS CloudTrail logs: AWS DMS is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, IAM role, or an AWS service in AWS DMS. CloudTrail captures all API calls for AWS DMS as events, including calls from the AWS DMS console and from code calls to the AWS DMS API operations.

Database logs: You can view, download, and watch database logs for your task endpoints using the AWS Management Console, AWS CLI, or the API for your AWS database service.

Use Cases

1. Move to managed databases: Migrate from legacy or on-premises databases to managed cloud services through a simplified migration process, removing undifferentiated database management tasks.
2. Remove licensing costs and accelerate business growth: Modernize to purpose-built databases to innovate and build faster for any use case at scale for one-tenth the cost.
3. Replicate ongoing changes: Create redundancies of business-critical databases and data stores to minimize downtime and protect against any data loss.
4. Improve integration with data lakes: Build data lakes and perform real-time processing on change data from your data stores.

Conclusion

AWS Database Migration Service is a handy tool for Migrating Databases from On-premises to AWS Cloud, Other Cloud to AWS Cloud and AWS Cloud to AWS Cloud. The migration method can be homogeneous or heterogeneous. It can help you for Full load, Full load and On-going replication or On-going replication type migrations. For heterogeneous migrations you can use SCT(Schema conversion tool) or DMS Schema Conversion for source server schema changes.

References

1. https://aws.amazon.com/dms/
2. https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html
3. https://docs.aws.amazon.com/dms/latest/userguide/CHAP_BestPractices.html

buy me a Tea