Back up and restore Mongo DB(hosted on AWS EC2) with a shell script, cron, and AWS S3

What is our requirement to solve?

1.Dump one/multi MongoDB database

2. Sync the backups with Amazon S3 with s3 bucket created

3. Run the command every night with a cron job

4. Restore the Mongo database from s3 bucket

Back up and upload into S3 from a mongo running server:

Step 1) Create a shell script to dump the database

cd ~

mkdir mongobackupscripts

cd mongobackupscripts

nano mongo_db_backup.sh

Paste in the following script :

#!/bin/bash

DIR=`date +%d-%m-%y`

DEST=~/mongo_db_backup_folder/$DIR

mkdir $DEST

mongodump -h localhost:27017 -d my_mongo_db_name -o $DEST

Note: my_mongo_db_name is the local dbname, mongo_db_backup_folder is the folder in which we will backup the mongo db, DIR = date +%d-%m-%y (date and time stamp used as a folder name so that the folder name doesn’t duplicate)

Chmod (chmod is the command and system call used to change the access permissions of file system objects sometimes known as modes.) the script so its can be executed

chmod +x ~/mongobackupscripts/mongo_db_backup.sh

Note: if we need multiple databases to be backed up, the last line should be duplicated for each local database (mongodump -h localhost:27017 -d my_mongo_db_name_1 -o $DEST).

Step 2) Create a shell script to sync the database backups with Amazon S3

1. Create a script file using touch or nano

nano mongo_db_sync.sh

1. Create an s3 bucket and replace the name: my-bucket-name-for-mongo-backup
2. Paste & save in the following script in mongo_db_sync.sh file

#!/bin/bash

/usr/local/bin/aws s3 sync ~/mongo_db_backup_folder s3://my-bucket-name-for-mongo-backup

chmod the script so its can be executed

chmod +x ~/mongobackupscripts/mongo_db_sync.sh

Note : verify the path for aws file is present /usr/local/bin/aws or /usr/bin/aws update the same in above file

Step 3) Create the folder for the database dumps to go

cd ~

mkdir mongo_db_backups

Step 4) Configure the AWS Command Line Interface

Install AWS Command Line Interface

sudo apt-get install awscli

aws configure

Add the aws access and secret key

Configure Credentials in Config

cd ~

mkdir .aws

nano config

Paste in your AWS key_id and secret_access_key, like below

[default]

aws_access_key_id=XXXXXXXXXXXXXX

aws_secret_access_key=XXXXXXXXXXXXXX

Note: By adding or configuring the aws access and secret key we need to validate the read and write permission is granted for s3 bucket to upload and retrieve the files

5) We need to install python-pip before we can install the AWS CLI with pip

sudo apt-get update

sudo apt-get -y install python-pip

curl “https://bootstrap.pypa.io/get-pip.py" -o “get-pip.py”

Response:
% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 1892k 100 1892k 0 0 23.1M 0 — : — : — — : — : — — : — : — 23.3M

Step 6) Configure the cron job

crontab -e

Paste in the following commands at the bottom of the file

30 20 * * * ~/mongobackupscripts/mongo_db_backup.sh >> /root/temp/backup.txt

32 20 * * * ~/mongobackupscripts/mongo_db_sync.sh >> ~/temp/sync.log

This will run the backup script at 8:30 pm and sync the files for Amazon S3 on 8:32 pm, which is scheduled for every day of every month.

Restore the uploaded Mongo into the mongo running server:

Create a folder restore it in the destination mongo instance machine

mkdir restoremongo

cd restoremongo

fetch the latest folder from the s3 bucket

aws s3 cp s3://s3bucketname/foldername/ . — recursive

run mongorestore commands to restore databases

mongorestore -d dbnameforwhichithastorestore foldernameinwhichdbisinjsonextension/

You will see the success message on the Mongo database importing

Author: Mohan Raj R
VP Engineering & AWS enthusiastic
Code Matrix Software India Pvt Ltd
www.codematrix.org