Typescript: Automated Data Migration and Data Seeding
Ever heard the term Data Migration or Data Seeding?, and no, it does not have anything to do with birds nor plants.
Data Migration
Data Migration means defining a data structure on a machine or database. Say you set up a new database to store your tables of your phone contacts. Each row would contain the contact’s name, contact’s phone number, and contact’s picture. Now before we can store them to the database, we need to tell the database what do we want to store (contact’s name, contact’s phone number, and contact’s picture), what kind of data we want to store (contact’s name would be a string of characters, contact’s phone number would be an integer, and contact’s picture would be a text, etc.), restrictions, constraints, and many more. This process is called Data Migration.
For this example, I would use a table from my PPL project called a User. User store information about our project’s user and contains 10 attributes.
Now since we are using typescript, we would use ts-node to run our typescript, and typeorm as our typescript library to help us program database operation in typescript. Next we use <packetManager> ts-node migration:create -n <tableName>. Since we use yarn as our packet manager, we use yarn ts-node migration:create -n User.
After running that command, it would create a new file, we call migration files. Here is User’s migration file. If you know a bit of SQL scripts, we could see that on line 8 is a string of SQL scripts use to create our table.
Next, we need to run this typescript to our database. To do that we use <packetManager> typeorm migration:run. This command would run all migration files in a designated migration directory to a set up database. To configure these settings, we use a file called ormconfig.js. Here’s an example of it.
Now we got Data Migration done, that’s all to it right? No, not yet. We need to seed our data.
Data Seeding
Data Seeding is an act where we insert data into a database. After telling the database what and how we want to store, next is we store those data. To do this we could write a typescript similar to this.
the data constant is used to write what are the data that we want to store to our database, while the run function is used to run the whole data seeding. An example of data would be something like this.
To run the seeding, we run our seed.ts with ts-node with something like this.
<packetManager> ts-node seed.ts
Automation
After creating our migration files and seeding, we could automate this process as to our liking. We could use gitlab-ci.yml to automate this on gitlab’s runner or in our local environment, use scripts like bash to automate these process.
Here is an example of a bash script we use to drop, migrate, and seed our dummy database for testing purposes.
