Cron backups into remote chroot using rsync

A lot of us use cron & rsync -e ssh for backups. On the backup server, we’d love to chroot for many obvious reasons, but chroot breaks rsync. I’m sure you’ve tried & seen something like this:

rsync error
protocol version mismatch -- is your shell clean?

Here’s my full solution on automated backups using chroot.
TLDR; if you’re only interested in making rsync work, head near the end.

Photo by Taylor Vick on Unsplash, edit ΔO

On Your Production Server

TLDR; You can use my script for generating backup command, creating ssh key pair and installing a cron job:

bash <(curl -s "https://tools.deltazero.cz/server/setup.backup.sh")

Or do it by yourself… On a server called Ergos (w/ hostname ergos.deltazero.cz), I have a script like this in /usr/local/bin/backup:

#!/bin/bash
PARAMS="-avh --exclude '*/__*'"
DIRS="/home /etc /var/log"
SERVER=my.backup.server.com
BUSER="backup_${HOSTNAME%%.*}" # backup_ergos
sudo rsync $PARAMS -e ssh $DIRS $BUSER@$SERVER:/

Don’t forget to sudo chmod +x /usr/local/bin/backup 😉

As the script uses sudo, it’ll auth with /root/id_rsa, which you can generate with:

sudo ssh-keygen -t rsa

Finally, make a root cron job for this:

sudo crontab -e
30 04 * * *     /usr/local/bin/backup
...

On The Backup Server

1. Enforce a chrootonto all users of the group backup to their home directories

# add these to /etc/ssh/sshd_config
Match Group backup
  ChrootDirectory     %h
# then reload ssh service
sudo service ssh reload

2. Create backup user for server Ergos

sudo useradd backup_ergos -g backup -s /bin/sh -d /home/backups/Ergos

3. Create backup user directory (owned by root, as needed for chroot)

sudo mkdir /home/backups/Ergos

4. Create .ssh/authorized_keys

cd /home/backups/Ergos
sudo mkdir .ssh
cat | sudo tee -a .ssh/authorized_keys
## copy paste contents of Ergos' /root/.ssh/id_rsa.pub 

5. Create directories for backup (backup user owned)

sudo mkdir home etc log
sudo chown backup_ergos:backup ./*
# this doesn't include hidden folders like .ssh

6. Create minimal chroot environment to run rsync

You need to create a minimal environment within the chrooted dir in order to make rsync do.

Run this at your chroot:

bash <(curl -s "https://tools.deltazero.cz/server/setup.chroot.for.rsync.sh")

Or do it by yourself:

# pwd = your chroot dir for backup user on the backup server
# see what libraries to copy
ldd /bin/sh
  # linux-vdso.so.1 (0x...)
  # libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x...)
  # /lib64/ld-linux-x86-64.so.2 (0x...)
ldd /usr/bin/rsync
  # linux-vdso.so.1 (0x...)
  # libacl.so.1 => /usr/lib/x86_64-linux-gnu/libacl.so.1 (0x...)
  # libpopt.so.0 => /usr/lib/x86_64-linux-gnu/libpopt.so.0 (0x...)
  # libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x...)
  # /lib64/ld-linux-x86-64.so.2 (0x...)
# linux-vdso is a virtual lib, copy all the others into chroot
# all /bin/sh libs are also present for /usr/bin/rsync
# so I'll just copy those
sudo mkdir bin usr usr/bin usr/lib usr/lib/x86_64-linux-gnu lib lib/x86_64-linux-gnu lib64
sudo cp /usr/lib/x86_64-linux-gnu/libacl.so.1 ./usr/lib/x86_64-linux-gnu
sudo cp /usr/lib/x86_64-linux-gnu/libpopt.so.0 ./usr/lib/x86_64-linux-gnu
sudo cp /lib/x86_64-linux-gnu/libc.so.6 ./lib/x86_64-linux-gnu
sudo cp /lib64/ld-linux-x86-64.so.2 ./lib64
# copy the binaries
sudo cp /bin/sh ./bin
sudo cp /usr/bin/rsync ./usr/bin
# look at the mess we've done
ls -lA
# drwxr-xr-x  2 root         root   4096 Jul 14 14:23 bin
# drwxr-xr-x 91 backup_ergos backup 4096 Jul  9 20:50 etc
# drwxr-xr-x  7 backup_ergos backup 4096 May 18 15:34 home
# drwxr-xr-x  3 root         root   4096 Jul 14 14:22 lib
# drwxr-xr-x  2 root         root   4096 Jul 14 14:23 lib64
# drwxr-xr-x 12 backup_ergos backup 4096 Jul 14 00:00 log
# drwxr-xr-x  2 root         root   4096 Apr  7 22:51 .ssh
# drwxr-xr-x  4 root         root   4096 Jul 14 14:22 usr

E voilá! It makes some mess, but
You can rsync into chrooted environment now 🤩

# run your backup cmd now
# first time, you'll need to auth known_hosts fingerprint
backup

More time savers for Linux servers

See my tools for 5-min Debian / Ubuntu server deployment:
https://github.com/deltazero-cz/linux-tools/tree/master/server

What do you think of scripts like these? Let me know in the comments 🙂