This week, it’s all about rsync and how you can use it to backup your system.
Before you start reading, I made use of this github repository for my rsync backups. You will also be able to do this on any linux version, or on any mac os device.
What is rsync?
To start off, rysnc is an open source app that helps you sync your backups or storage to an external hard drive. (for my use-case). Find out more about rsync here — https://rsync.samba.org
Before i go anywhere. Feel free to pop by and look at these rysnc example scripts that can inspire you to start your own “I need to backup my drives!” craziness. — https://rsync.samba.org/examples.html.
With Rysnc, you can back up your computer to a central backup storage on the cloud, or an external disk. Removing that single point of failure if and only if your machine fails, you’d still have two-three places you can retrieve your backups from.
Today, I learned to make use of rsync to back up my drives. I also managed to automate this process with the use of a simple script. (crontabs + rysnc!)
Rsync with crontab
Crontab allows me to set a command to be run at a particular time monthly, or daily if i want. Here’s an example I made to backup my local machine’s desktop to my external hard disk.
# Crontab's format of usage
* * * * * command to execute
│ │ │ │ └─── day of week (0 - 6)
│ │ │ └──────── month (1 - 12)
│ │ └───────────── day of month (1 - 31)
│ └────────────────── hour (0 - 23)
└─────────────────────── min (0 - 59)# Eg:
0 12 28 * * rsync -r ~/Desktop/ /Volumes/brianexternal
The above runs the the rsync command every 28th of a month exactly at 12:00.
To set up a crontab, all you gotta do is to run the command crontab -e on your terminal, and just type in the above, based on whenever you want the command to run.
Here’s how you get first get started and dip your heads into the backup game.
1.Identify where you want your backups to go, and what exactly you want to back up.
In my case, I had my hopes on backing up my old code repositories and websites I use to create when I was still exploring what HTML was all about, I also have a remote machine I have with the best cloud service provider out there — OVH!
They offer free DDOS-protection, and it’s simply great value for the servers with a high compute power, ram and for a cheap price. There is no downside to being a OVH customer. I’ve been with them for 3–4 years now, and they never dissappoint
2.Set up your ssh keys onto your remote-server, if applicable (otherwise, you can do backups to an external hard disk without ssh keys — 🔐 )
I set up my ssh keys a long time ago, and if you’re not sure how, read this! (PS: you can ignore the second part, if you don’t want to use an ssh-agent to manage your keys)
Next, once you’re sure you can connect up to your machine, set up the crontab & commands for backing up.
3. Set up the correct crontab & commands for your own backup/s
In my case, I set up the crontab to run every monthly, and here’s a tool I really recommend to help visualise your crontab syntax
Next thing to do is to set up some sort of logging system, to help you identify any errors with your crontab. Since this is a backup cronjob, it could mean there may be permissions that may require proper credentials on the remote server, or files that may be skipped or not copied over.
I found this article, which explained a lot about crontab logging.
./rsync_tmbackup.sh -p 22 ~/Desktop/ user@hostname:/var/www/html/backup > ~/Desktop/brian.log 2>&1
The above command executes and copies all folders from ~/Desktop to my cloud server’s directory /var/www/html/backup, all while outputing the results of it’s log into a brian.log file on my Desktop.
(PS: If you are using the rsync-time-backup, make sure you create a backup marker on the first initial run — If you have the above command, the logs will tell you the exact command — shown below!)
ssh -p 22 username@hostname 'mkdir -p -- "/var/www/html/backup" ; touch "/var/www/html/backup/backup.marker"'
(optional) Test and see if cronjob/crontab works
All I have to do now is to link this up to a cronjob, and check if it works, at all. I wasn’t really sure how a crontab worked in the first placed, despite all the information online. I had to test it out for myself to see. Here’s what i did.
# env EDITOR=nano crontab -e
58 19 * * * curl bchewy.me > ~/Desktop/testcron.log
I opened the crontab -e with nano, and did a simple curl command, along with sending it’s output into a log file so I could visualise what was going on. Now I was sure that the cronjob would run properly, so I had to add my crontab for rsync, and made sure it ran properly.
4. Running my crontab initially to see if it works
# env EDITOR=nano crontab -e
@monthly cd ~/Desktop && ./rsync_tmbackup.sh -p 22 ~/Desktop/ username@hostname:/var/www/html/backup > ~/Desktop/brian.log 2>&1
Once that was added, I had to verify to see if it was indeed there with
crontab -l then, you should see that the cronjob exists.
I decided to then run my crontab command for the first time, to see it works.
It works! — woohoo!
You should see this final complete message once the backup is done, and this is again dependent on internet speed, size of your backup & several other factors. (i saw mine in my logs, since i output all the results there)
Number of files: 351425
Number of files transferred: 303538
Total file size: 5.56G bytes
Total transferred file size: 5.54G bytes
Literal data: 5.54G bytes
Matched data: 0 bytes
File list size: 8150509
File list generation time: 10.743 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 4.01G
Total bytes received: 6.97Msent 4.01G bytes received 6.97M bytes 1.02M bytes/sec
total size is 5.56G speedup is 1.39
rsync_tmbackup: Backup completed without errors.
I hope I helped you get a quick-start into setting up a rsync backup, understanding it better like how I understand it. Now, I’ll be backing up my files monthly, and my backup strategy is to have at least two points where my files are at all the time.
I’ve also put all the links and references I looked at when creating this guide/post. Please check the original authors & their resources.