Fast and Secure Backups to S3 with Restic

We all know how important backups are, but even today we seldom have a reliable and secure backup solution. Some people have a small NAS device at home that they backup their data to, but how useful is that in the case of theft or fire?

Restic is an open source backup application written in Go. It runs on all modern operating systems including Windows, Mac, and Linux. There is no graphical interface as it is a console application, but this guide will teach you how to install and configure it for regular backups to the cloud.

Installation

Note: This guide is tailored towards Mac and Linux users. I may mention some Windows specific steps for installation, but the rest of the guide assumes a Unix style OS.

To get the latest stable release of Restic, visit their Github releases page. On Linux and Mac, extract the downloaded archive and move the Restic binary to /usr/local/bin as restic. On Windows move the application into your C:\Windows\System32 directory. This will put Restic in your path, making using it as simple as typing restic.

Installing Restic from Github on Ubuntu Linux

Mac

You can also quickly install by running brew install restic.

Debian/Ubuntu

sudo apt-get install restic -y will install, but the version provided might be many versions behind the latest release.

S3 Bucket and IAM Configuration

Start off creating an S3 bucket to store your backups. Log into your AWS console and create a new bucket to store your backups. Feel free to choose whatever name you want for your bucket. I highly recommend enabling versioning and encryption. Restic also encrypts your data, but you can’t be too safe. And triple check to make sure your bucket is not public.

Review of my new bucket for backups

Now that we have a bucket to store our data, now we need to create a dedicated AWS user for Restic. Go ahead and do this now, don’t use your personal keys even though it might be quicker to do.

Create a new user name restic and grant it programmatic access.

Create a custom JSON policy for the user. Use the snippet below, replacing BUCKET_NAME with your bucket’s unique name.

Click Create policy to customize the permissions for the new user

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::BUCKET_NAME"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::BUCKET_NAME/*"
        }
    ]
}

Once created, store the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in a safe place. We’re gonna need it soon.

Restic Configuration

Generate a Repository Key

Restic protects your data by encrypting it. This way even if somebody gains access to your S3 bucket, the still cannot read your data. Generate a strong 24+ character password using LastPass’s password generator. Store a copy of this password in a secure location because without it, your backups will be useless.

Configuration File

Using your favorite text editor create a file named .restic.env in your home directory. Inside this file, we are going to add lines for each of the 4 required environment variables. Use the example block below for yours, replacing the values with your own.

export AWS_ACCESS_KEY_ID="AKIAJNFRE7YAHGDUMD"
export AWS_SECRET_ACCESS_KEY="EtE7mGPqcbjyPT4KLXX1qVDUkAVhA"
export RESTIC_PASSWORD="!emP*#IWFEoy8%@zp9qQpIHo"
export RESTIC_REPOSITORY="s3:https://s3.amazonaws.com/BUCKET_NAME"

Let’s protect this file from other users by changing the permissions on the file to read-only by your user with chmod 400 ~/.restic.env. To activate these environment variables into your current bash session run source ~/.restic.env. I don’t like having credentials in my shell all the time. This method allows me to only have them active when I need them.

Creating, securing, and testing my .restic.env file

Initializing Your Repository

Before Restic can backup anything, you must first initialize its repository. This only has to be done once. To do so run source ~/.restic.env ; restic init.

Contents of an empty initialized repository

Your First Backup

Let’s create a text file to backup and go through a few basic Restic commands to demonstrate backup and restore procedures.

• Create a sample file name backuptext by running echo "My first file to backup" > backuptext.
• We’re going to set our required environment variables with source ~/.restic.env
• Backup the file by executing restic backup backuptext.

Creating and backing up our test file

• Overwrite backuptext with echo "My modified file" > backuptext.
• Backup the modified file. restic backup backuptext.
• List the snapshots in your repository. restic snapshots.

• Confirm the contents of backuptext. cat backuptext
• Restore from the first snapshot with restic restore SNAPSHOTID -t restored-data replacing SNAPSHOTID with the first snapshot listed when you ran restic snapshots. The file will be restored into a directory named restored-data.
• Confirm contents of restored file. cat restored-data/backuptext

Restoring file from backup

Clean Up Test Snapshots

We can cleanup our test snapshots with the restic forget --keep-tag="whatever" --prune command. Since we haven’t tagged any of our backups, it will delete all of these because the --keep-tag flag tells Restic to only keep backups with the tag whatever. The --prune flag tells Restic to also delete the data from S3, which isn’t done by default.

Deleting all snapshots

Real Backups

Now that we have a working Restic configuration, it’s now time to create a more useful setup. This walk-through will backup /etc, /usr and /root directories. You typically would not backup /usr, but I’m using it as a source for lots of data. This backup will be scheduled to run every 15 minutes. Since Restic only uploads changed files, subsequent jobs will run fast after the initial upload is done.

Specifying Files to Backup

We’re going to create a file called restic.files in our home directory. This file will list all the files and directories we want to back up. We could specify them on the command-line like we did earlier, but a file makes it easier to work with once the list starts expanding. List each directory on a separate line in the file.

/etc
/root
/usr

Running Our Initial Backup

Once again we’ll setup our environment variables with source ~/.restic.env. To run our backup using our restic.files list we’ll use the --files-from /root/restic.files flag. Restic has issues using the ~ shorthand, so we have to type the full path to the file. The --tag seed flag tags the snapshot as a seed. This is a freestyle label that can be used for organization. The full command is restic backup --files-from=/root/restic.files --tag seed.

Our seed backup

We’ll run the command again. Notice how quickly it runs this time. 3 seconds compared to 50 seconds for our initial backup. This is because Restic can quickly determine if a file is already stored in S3 and only upload new and changed files. It also does chunking, which means if parts of a very large file are changed, only the changed chunks need to be uploaded.

Subsequent backups are very quick

Automating Our Backups with Cron

Since Restic backups run fast after the initial seed, it’s very useful to run backups very frequently. For our example, let’s have cron run backups every 15 minutes. We’ll also configure the job to write output and errors to restic.log and restic.err in your home directory.

• Run crontab -e to edit your user’s crontab.
• Add the following line */15 * * * * . ~/.restic.env ; /usr/local/bin/restic backup --files-from=/root/restic.files --tag automated 2>> ~/restic.err >> ~/restic.log. Notice we use . instead of source. This is a nuance of cron using a POSIX shell and not bash.
• After the next quarter-hour, check your snapshots, restic snapshots, and confirm the task completed as expected.
• If no new snapshot is created, check the restic.err file for errors.

Automated snapshot listed

Conclusion

You now have a nice automated backup solution to S3. For most users, simply backing up your home directory will be all you need. On a business system, backing up /etc, /var, and specific data directories will be needed.

If you would like an estimate for how much S3 storage you are using, restic status will return that information.

Root directory structure of an active repository

In upcoming articles, I will expand on this one by giving instructions on adding an automated backup to a Western Digital My Cloud NAS and for backing up AWS Elastic File System (EFS) shared filesystems.