Tricks to restore large mongo collections on small machine
Mongo is an open source no-sql database which stores data in documents and collections. We use mongodb at Synapse because its developer friendly and allows us to churn out api’s in no time. It also plays well with Django and we can code up features without worrying about schema and migrations.
Taking backups and restoring them to work with latest data is part of daily life at Synapse. Mongo provides mongodump and mongorestore command line utilities to make that easy for the developers.
mongorestore is a command line utility by which one can restore some or all the collections. mongorestore loads a collection or database in memory and then puts it in database. It also creates the indexes for the restored collection afterwards. You might say what’s the problem with this?
Sometimes when the size of collection is too large it may result in insertion error:EOF or unexpected input:EOF exceptions causing mongo-db instance to crash.
The error is not suggestive but it’s related to available RAM on the machine. Mind you — its the RAM which matters instead of hard drive space. So how did we solved this?
There are two possible solutions to this problem -
To restore the collections one by one.
To create/update the swap space on the machine.
We tried the first approach and were successful till we encountered a 1.3 GB collection. We were working on a small machine with 512 MB RAM and Ubuntu 16.04.
We tried the next approach of creating the swap space of 2 GB and we were able to restore the database. Here are the steps to create the swap space -
Read this article for more information on setting up permanent swap space on Ubuntu 14.04.
Mongo allows us to backup and restore databases and collections. If you happen to face insertion error:EOF it can be a memory related issue and trying out one of the above listed solution can get you out of trouble.
But remember that creating swap space can slow down the things because reading from file is way slower than reading from RAM. Swap is a short term solution and one should upgrade the RAM to a larger one to avoid such issues in future.
Mongo is working out quite well for us. We are able to roll out new features without worrying too much about schema and migrations. :)