How to migrate DBs with NodeJS and TypeScript

Migration of a database is a process of applying changes to the database. For example, in SQL it’d be creating/dropping/altering of tables, etc. Here is why you’d want such a process in your application:

• Source control for your databases. Usually, you put all the changes of the repository in the same repository as your application code. Thus, you control your databases the same way as you do with the codebase, along with all the advantages of the source control.
• Reproducibility. Ideally, you may take an empty database and apply all the changes from the very beginning, and you’ll end up in the same state as your production databases. It is especially helpful when you have multiple environments: development, testing, production, etc.

Now, I got to this topic when trying to find a solution for migration of DynamoDB for an application on NodeJS written TypeScript. Here are my thoughts on that.

Why Not CloudFormation?

The first and the most important issue I faced with that approach: I must be able to apply the changes locally exactly the same way it’s done on production. However, I cannot use CloudFormation outside of AWS for migrating a local instance of DynamoDB.

The second reason — flexibility. Sometimes you do need to apply changes that cannot be done within the CloudFormation template changes. For example, you may remove some record from a table, or you may want to call some other service before making changes, etc.

The third — worse reproducibility. On production you apply changes one by one: today you add a table, tomorrow you remove it, etc. CloudFormation won’t do the same, instead, it takes the target and current states of DynamoDB and calculates the changes that are needed to be made. Thus, you might end up in different states in different environments.

Umzug For the escue

The umzug lib is a framework agnostic migration tool for Node.JS. The tool itself is not specifically related to databases but basically provides a clean API for running and rolling back tasks.

Umzug is not a tool just for migration of DynamoDB, it may be a tool for migration of anything. Add an ability to store umzug’s book-keeping data in DynamoDB with the help of an add-on, and you will have the perfect solution (not only) for my case.

But… It does not support TypeScript well. Don’t get me wrong, it has typings, but it’s not enough to fully support TypeScript. Umzug expects a set of JS files as its input, so in case of TypeScript, you first need to compile each of the migration scripts which is not quite convenient, especially if you bundle your backend (because you’d have to bundle each of the migration scripts as well).

Fixing Umzug

Ideally, I want to have a single function migrate that accepts all my migration functions and then just applies them. For example:

migrate(migrations, umzugOptions).catch((err: any) => {
    console.error(err);
    process.exit(1);
});

I’ve created such a function migrate, it applies the changes in 4 steps:

1. Generates stub JS scripts in order to satisfy umzug. 1 script for each function.
2. Starts umzug.
3. Umzug runs the given scripts.
4. Each script spawns another process running the same migration entry point script (using the same exact command from the step 1), but passing additional environment variable containing the name of the corresponding migration function.

It’s hacky, but it works very well. You can reuse the migrate function in your project if you like by using the npm lib called umzug-typescript-helper.

What’s Next?

We will talk about bootstrapping your own serverless CI/CD pipeline on AWS. All the steps from the push to your repository to the delivery on production through testing.

Comments, likes, and shares are highly appreciated. Cheers! ❤️