How to setup** AWS backup with python.**

What this article is about

Amazon EC2 is one of the most popular cloud platforms. There are two Amazon-based options to make backups:

  1. You can make a copy of your whole server data with all system files and packages you’ve already installed. More time-consuming.
  2. Configure your system to store all important data in separate volume (Amazon Elastic Block Store volume). Next, you make backups of that volume only. Less time-consuming.

In this article you will see how to implement authomatic EC2 backup (both options) with python.

Before we start, you’d better go and read about boto, as we’re going to use it heavily.

Possible approaches on AWS for backups

WHOLE IMAGE (AMI) BACKUP

  1. Create an Amazon instance, install all required packages and set up applications you need.
  2. Attach separate volume to your instance and mount it to /mnt/data.
  3. Configure this applications in such a way that all your precious data will be saved somewhere in the mounted volume.
  4. Check everything is working fine, stop your applications and delete all garbadge files that were probably generated during testing.
  5. Create an image of your instance.

Now, you can generate new instances from this image. They are restored from snapshot, so no need of additional configuration or setup. This is the main reason to make a whole image backup.

Restoration process

So you already have an image of your Amazon instance and you want to test it. With python code below you can make a copy of your backup instance. It will also create and mount a volume for your data.

You can also do it manually: just launch image for particular instance in AMI’s tab in AWS console (FIXME: тут надо картинка)

Link to the code

Some explanations for that code:

Create config file first. You will store all required settings for scripts there. Example is here.

Now let’s comment some code.

We put loggers to all important code parts to make debugging simpler.

logger = logging.getLogger('maas')

You may notice that some of our code is wrapped in try/except blocks; it’s important to have because when we try to connect to server something bad (e.g., loss of internet connection) may happen. So we try to repeat the request a couple of times. In the worst case, we will get info about error in log files.

When instance is started, we can automatically execute some pre-saved commands. You can set up commands to execute on startup manually: just go to the instances tab, select desired instance and change user data. All commands from this field will be automatically executed every time server starts.

To automate this — edit USER_SCRIPT_TEMPLATE variable:

  1. mount volume connected to your instance
  2. you can execute some commands you need.

You can place all your commands after ‘mount -a’ command. ATTENTION: debugging here is really painful, so you should be very attentive when you write such scripts..

USER_SCRIPT_TEMPLATE = """#!/bin/bash -ex
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

## setup the ebs volume for data.
avail_blk=`lsblk -n -oNAME,MOUNTPOINT | grep -v '/$' | grep -v 'xvda' | awk -F' ' '{{print $1}}'`
if [ -z "$avail_blk" ]; then
echo "Don't have a mounted data blk device."
exit -1
fi

update_needed=`file -s /dev/$avail_blk | awk -F':' '{{print $2}}'`
setup_fs=`echo "$update_needed" | egrep -e '^[[:space:]]+data$' |wc -l`

if [ $setup_fs -eq 1 ]; then
echo "Setting up a file system for /dev/$avail_blk"
mkfs -t ext4 /dev/$avail_blk
fi
cp /etc/fstab /etc/fstab.orig
echo "/dev/$avail_blk /mnt/data ext4 defaults,nofail,nobootwait 0 2" >> /etc/fstab
mount -a
"""

Probably your application needs permission on mounted directory, so you can set them. Also, you should create all folders your application will use.

USER_SCRIPT_TEMPLATE += """
chown -R www-data:www-data /mnt/data/
mkdir /mnt/data/... # here you have to create all required folders for your app
touch /mnt/data/... # here you have to create all required files for your app
chown -R www-data:www-data /mnt/data/*
"""

Also you can change some settings for your app if you want.

USER_SCRIPT_TEMPLATE += """
# here you can change some settings files for your app
echo "{some_variable_to_pass}" >> # path to system file
"""

As in the snapshot we have all our applications stopped, you should start all of them again in the script:

USER_SCRIPT_TEMPLATE += """
# here start all your apps
sudo service supervisor start
supervisorctl start all
"""

In the code we use conn variable to make requests to Amazon server. In code block below we try to create our instance:

conn = boto.ec2.connect_to_region(
AWS_REGION,
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY
)
reservation = conn.run_instances(
AWS_AMI_IMAGE_ID,
instance_type=instance_type,
key_name=AWS_KEY_NAME,
security_group_ids=AWS_SECURITY_GROUPS,
subnet_id=AWS_SBNET_ID,
block_device_map=bdm,
user_data=user_data,
)

And then wait until it is created:

instance = reservation.instances[0]
while instance.update() != "running":
time.sleep(5)

And at last check if it works as expected:

assert instance.ip_address is not None
assert instance.update() == "running"

DATA VOLUME BACKUP

When you have volume connected to your instance, the only entity for you to make a backup is volume. You don’t need to backup the whole instance, as all changeable data is stored at volume. This will save your time and resources. Here’s how to do this:

Backup process

With the code below you can make backups of data volumes of all your instances.

You can create snapshots instance-by-instance manually in volumes tab:

Link to the code

Some code explanations:

Here we get list of all our Amazon instances:

instances = conn.get_only_instances()

Here we try to get data volume id for each instance. If it exists, we create a new snapshot from it. We ignore instances without id.

for instance in instances:
data_vol_id = extract_non_root_id(
instance.get_attribute("blockDeviceMapping"))
if data_vol_id:
manage_snapshots(conn, data_vol_id)

After new snapshot is created, we delete oldest snapshots in the delete_old_snapshots function, ignoring four newest ones. We can also provide one additional snapshot we want to keep (exclude_snap).

volume = conn.get_all_volumes([vol_id])[0]
snapshots = volume.snapshots()
snapshots_sorted = sorted(
[(s, s.start_time) for s in snapshots],
key=lambda k: k[1]
)[4:]
for snapshot, start_time in snapshots_sorted:
if snapshot.id == exclude_snap:
continue
info = "deleting snap {id}: {desc}".format(
id=snapshot.id, desc=snapshot.description
)
logger.info(info)
snapshot.delete()

Run manage_instances_snapshots a couple of times from python shell. For each volume you will have up to four snapshots.

Restoration process

You can restore backups manually: just create an image from snapshot on snapshots tab:

But a script will make massive backups much easier. Link to the code

This process is very similar to instance restoration process above. The core difference here is that you have to stop your current instance.

We do it in fuction try_to_stop_ec2_instance FIXME: link. First we have to try to stop it.

conn.stop_instances([instance_id])
reservations = conn.get_all_reservations([instance_id])
instance = reservations[0].instances[0]

Then we wait until it is stopped:

while instance.update() != "stopped":
time.sleep(5)
return "stopped"

Then we do the same as instance restoration process above. You should create a new instance from image and then mount your backuped data volume to it. After it is mounted you can add any code that you want to execute on this instance startup, we explained it more detaily in our previous article about whole image backups, in explanation of restoration process. Again probably you should set up special right to this data. At last you should start all your apps.

To restore, you should call function try_to_create_ec2_instance with id of volume snapshot from which you want to restore data (backup id) as first argument and id of current instance as second argument (because we have to stop this instance).

instance = try_to_create_ec2_instance(non_root_snap_id, instance_id)

As you can see, logic here is almost the same as in the first case, so you can use analogy.

Afterword

We really hope this article helps you. But, if you’re not sure you can easily setup backups on the Amazon — we can do that for you, fast and affordable. Please, contact us, we will answer any questions!

or:

we really hope this article is useful for you. But if something is not clear — we can setup backup system on Amazon, or answer any questions. Please, contact us

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.