PostgreSQL to MongoDB Migrator — Batteries Included!!!
DATABASE MIGRATION ACROSS PLATFORMS — Got your goosebumps yet?
Well, long story short; cross-platform database migrations equals sleep talking, distress and long day works with coffee; and what good does it do? We will just end up writing hours and hours of scripts to conquer the end-result. However, It is of one-time-use-only, which lets you think to yourself; “All this horsepower and no room to gallop?”
PostgreSQL to MongoDB:
Be it a platform change, or maybe, it’s due to the organizational growth or perhaps bad coding, or perhaps you have got your own microservices all set in, dwelling on the JSON objects; you might have had to switch from relational to noSQL databases. Switching can be tedious, I hear you and here lies the solution to all your worries.
Behold! Enter the Pg2Mongo:
Pg2Mongo is an open source database migration tool between PostgreSQL and MongoDB, written on pythonV3 which gives you an exclusive control over the migrations.
First Steps:
The initial step is to make sure you have access to both the PostgreSQL and MongoDB servers. Upon cloning the repository, make sure you install the requirements for the pg2mongo to run.
For demonstration-sake, let’s try to migrate the dataset provided along with the pg2mongo for us to play-around.
Configuration setup:
And now, all we got to do is to set up the instructions for the migrator to wrangle. The configuration file is at the location — ‘pg2mongo/pg2mongo.yml’ and it goes as follows:
The preliminary sections such as extraction and commit are self expanatory, stating the configuration settings for the extraction and commit databases. The component Migration is where all the magic happens!
The following section explains what the individual components are all about:
INIT_TABLE:
Initial table from which data needs to be migrated. This could be a prime table such as a transactions table with a primary key having multiple foreign constraints to other tables of the PostgreSQL database. FOR EACH ENTRY IN THIS TABLE, THE LINKING OF OTHER TABLES WILL HAPPEN WHILE DEFINING THE TABLES.
INIT_KEYS:
KEYS of the init_table (aliases can be given using ‘as’)
SKELETON:
Skeleton is an empty raw python dictionary assignment which will transform to a mongodb document, upon migration
TABLES_ORDER:
The order by which the TABLES section needs to be executed for each of the entry from INIT_TABLE
TABLES:
Set of PostgreSQL tables enlisted along with condition and corresponding mapping. In the case of lists inside a dictionary, list can be mentioned. Mapping is where, the association of skeleton to the table keys is defined. The value assignments are python compatible; hence, they are defined by using ‘%s’ and other python based variable transformation functions can be used over here.
COLLECTIONS:
This is where the push of the skeleton to the corresponding MongoDB collection takes place.
With all the instructions in place, it’s time to wrangle. You may invoke the migration by keying in the following command.
And off she goes!!
Originally published at https://www.datawrangler.in on December 10, 2017.
