How I Replaced Dropbox with Rsync and Why

Photo by Ian Battaglia on Unsplash

Who doesn’t love cloud storage services like Dropbox?

Imagine installing one simple program on your computer that automatically synchronizes all your personal documents to a server as backup.

With a free-of-charge storage plan of 2GB of free storage, this is a very convenient alternative to using USB flash drives to store your personal data. I mean, be honest with me, how easily can those flash drives get stolen — or lost.

Most of the people reading this have used a cloud storage service provider to store their personal information at some point in their lives. There is this magical satisfaction in the assurance that your data is definitely backed up on a server that will not get lost or easily compromised (as long as you used a good password and a plus if you applied Multi-Factor Authentication).

However, in the past month I discovered that there was…a price to pay for this convenience.

So here are the security problems with services using Dropbox:

1. It is generally unwise to store your personal data on a centralized server. The more people store their personal data on a centralized server database, the more financially profitable it is for the server to get hacked. Dropbox is definitely a strong case incident of this rule. If you really care about your security, then you should never store your personal data in a place that is financially profitable to get hacked.
2. If your data is left unencrypted, any system adminstrator that has root privileges can see your content. Usually, companies look forward to your data so they can sell your data to other companies.
3. No, the AES-256 encryption is not hack-proof. It is actually not that hard to hack AES-256 as long as you have physical access to the computer that hosts the encrypted file and executes the AES algorithm to protect said files. Just read this security report. Unfortunately for you, the cloud storage user, your cloud storage provider is the one that has physical access to their own remote server. You obviously do not. Companies usually do not have as much of an incentive to automatically protect your data by default with encryption, since they do make money off your personal data. They are a business and need to make money off of the “free-of-charge” space they gave you somehow. Any sysadmin, especially the root user, can see your personal files.
4. Companies usually demand large sums of money for more storage space. On their own servers. You can very easily setup your own server at home and store your personal information on your own hard drives.

You just got pwned.(Photo by vipul uthaiah on Unsplash)

Note: In response to the timing side-channel attacks that AES-256 is vulnerable to, the Poly1305-ChaCha20 cipher-authentication system was invented. But still, even the Poly1305-ChaCha20 will not protect you against other side-channel attacks, as the research paper in the previous sentence admits. At the time of this writing, there is absolutely no cipher that is resistant against all known forms of side-channel attacks. For now, the Poly1305-ChaCha20 is the most up-to-date cipher for encrypting large files. But again, it is still vulnerable to EM side-channel attacks. This attack only really works if you have physical access to the server.

3. Dropbox’s main selling point, seamless remote-syncing, is nothing new. Advanced Linux and Unix users have been performing the same feat for years before Dropbox was a startup. A simple linux command that is capable of replacing Dropbox is the rsync command. Like Dropbox, rsync can efficiently first zip files up before they are sent online to the server. Unlike Dropbox, rsync gives you full control over to which server your data is being sent to.

Ever since I started using Rsync I honestly stared at every cloud service in the world and wondered for what reason I ever used them in the first place.

Here are my cents on why I switched from Dropbox to Rsync:

1. Now that I use Rsync, I can store as much data as I please as long as I have the hard drive space for it. Thankfully, my personal Rsync folder is only a little more than a Gigabyte of storage. But still, it is a powerful feeling that I can literally store hundreds of gigabytes of information in that one miracle folder if I wished to free of charge. And then automatically backup that same folder and all the machines I wish.
2. The Slant community officially voted Rsync as a better algorithm for file syncing than Dropbox. I will post the Slant rant right here:

“Only the changed parts of files are synced. For instance, if a long log file increases by just a few lines of text, a small diff will be sent to and saved in the archive. Rsync also compresses data in transit.” — Slant Community

3. I can send my own files to my own personal server that is protected by end-to-end encryption, free of charge, when using it with an *encrypted SSH connection. Now that the data is on my own personal, decentralized server, there is much less financial incentive for a hacker to try to hijack my information. It just doesn’t make business sense for a hacker to steal data from a lone user’s computer than it is to steal from a vast database like the one Dropbox has. Moreover, now that the data is only available on my personal computer, I will not have to worry about companies selling my personal data.

*Note: Hackers have hijacked the SSH servers of personal users in the past. ESET published a report on this. The basic moral of their story is simple:

1. Disable Remote Root Logins completely. To login as the root user, you must have physical access to the actual server computer and login directly.
2. Disable Password-Based Logins. Only enable Public-Key based authentication.
3. Consider adding Multi-Factor Authentication into your SSH logins. Remember those six digits you have to copy from your phone and paste onto your browser before you get to sign in? Yes, that’s what I mean. Except this time, you are doing it in the terminal. Consider using OAuth Toolkit.

Meanwhile there are real-life cloud storage providers that require a financial price for this level of quality in service.

Are you convinced to switch to Rsync yet? Great! Let’s get started.

1. Our first job is to find a Linux server computer. This is much easier than you think. Today, Linux is so lightweight you can literally run the OS from a USB flash drive. I strongly recommend running ArchLinux from USB. The link in the previous sentence has instructions that are so well written, I will simply let ValleyCat teach you guys (and gals) that. The directions are so well-written, you can quickly figure out how to install Arch on an actual computer afterwards. Lightweight distro that has strong documentation support and community. Especially community.
2. The rest of this guide will assume you are determined to run the server from an ArchLinux USB. Remember, you will need to configure your bios so it is set to boot from a USB flashdrive as first priority. You will have to lookup your own computer model’s BIOS/UEFI settings to figure out how to do this. There are a lot of reasons you should start out running a server from a $5 16GB-USB flash memory. It is less risky for you to handle a USB flash memory stick than a full blown hard drive that probably set you back ~$50. (Yes, I broke one trying to do just that at first). And the flash drive is portable. So if you are on the go, your server (your bootable USB flash drive) can easily go with you.
3. If you followed ValleyCat’s guide properly, you should now have a bootable USB flash drive to run ArchLinux from. The next step is to setup a GPG key. This is where my tutorial begins.
4. First, manually setup your own simple (yet reliable firewall) based on ArchWiki’s excellent guide: Simple stateful firewall. Adjust port numbers as necessary to suit the portnum_of_choice for SSH logins.
5. You should change the hostname of your ArchLinux USB server. Treat this as if you are setting up your own website name. What would be the name at your website? Would you name it “coolserver.org”? Or will you name it “secret_files.com”? Think about this. For the rest of this tutorial, your website name will be referred with the placebo name_of_website.
6. Figured out the name of your server? Great. Open /etc/hosts:

sudo vim /etc/hosts

7. This file should only have one line: the name of your website.

coolname.[org|edu|com|gov|...]

The “coolname” in the name of your website is called the hostname of your server. Everything after the decimal point ‘.’ is called the localdomain

Now open /etc/hosts. If you followed ValleyCat’s advice, then all of you have to do is change the following line:

127.0.1.1    coolname.[org|edu|com|gov...]    coolname.[org|edu|com|gov...]

Basically you replace everything after the “127.0.1.1” with the name of your website.

1. First login into your ArchLinux bootable USB flash drive as root user. To setup your own SSH server on ArchLinux, you will need to download the “openssh” package in ArchLinux:

sudo pacman -S openssh rsync

If using RHEL/CentOS:

yum install openssh
yum install rsync

2. You must now configure the /etc/ssh/sshd_config file:

sudo vim /etc/ssh/sshd_config

This is the file that configures how your SSH server will function.

Edit the very top of the file as follows:

Protocol 2
Port portnum_of_choice

I strongly recommend that you make portnum_of_choice as high as possible. Port numbers range from 0–65535 since the port number is stored in the computer as a 16-bit unsigned number.

Now go down the file and find “#Ciphers and keyring” and add the following:

# Ciphers and keying
ciphers chacha20-poly1305@openssh.com
macs hmac-sha2-512-etm@openssh.com
KexAlgorithms curve25519-sha256@libssh.org

The SSH server is now configured to host communications with it using the Poly1305-ChaCha20 cipher and authenticated with the hmac-sha2–512-etm algorithm to verify the information matches that claimed to be sent by the client. Moreover the SSH server is configured to encrypt keys using the curve25519-sha256 algorithm.

5. Scroll down the file until you find the line: “PermitRootLogin”. By default, this says “yes”. But if you read the ESET report, you now know this is a terrible idea. Change “yes” to “no”:

PermitRootLogin no

Scroll down the file and edit the following line as follows:

PasswordAuthentication yes

Scroll down the file and find the line:

ChallengeResponseAuthentication yes

and change it to:

ChallengeResponseAuthentication no

Note: The only exception to setting “ChallengeResponseAuthentication” to “no” is if you are setting up Multi-Factor Authentication as an additional security check before completing the SSH-login. But that is a tutorial for another day.

Normally, Linux and Unix users store everything in their personal home directory. I strongly encourage you to create a brand new username with its own home directory. Do NOT give this username you are about to create sudo privileges nor root privileges:

useradd -m -G users rsync_username

This command will make a brand new username (replace rsync_username with the username of your choice) and give its own personal home directory.

This new user will not have access to the files in your original home directory.

Quickly set the password for rsync_username by typing:

passwd rsync_username

Head to the home directory of this new user as root user by typing:

cd /home/rsync_username

Create the Rsync directory:

mkdir Rsync
chown -R rsync_username:rsync_username Rsync/
chmod -R 777 Rsync/

What “chmod -R” just did is that it made all the files and subdirectories in Rsync/ have the following permissions:

drwxrwxrwx 44 rsync_username rsync_username 40960 Jan 15 18:53 Rsync/

Did you see what was in bold? That is exactly what you should see starting from the Rsync/ directory and EVERY file and subdirectory within Rsync/.

In order to be able to sync files flawlessly from a remote shell login, you will have to allow all file permissions throughout the entire directory.

This is why the Rsync directory is being put in its own separate directory and not your own. It is for safety reasons.

By logging into the home directory of rsync_user you will be the rsync_user and will never be able to look at the contents of your real home directory.

Still, I must confess there are plenty of security risks in this technique. The rsync_user will still be able to see all files at the root directory of your hard drive. This is why chroot-jailing your rsync_user directory is a good idea. It will set the root directory for rsync_user to be the rsync_user home directory itself, preventing rsync_user from going to any directory above its home directory.

This is why I told you not give neither sudo nor root privileges to rsync_user. It is very dangerous. I will talk about chroot-jailing later, since I still have not learned how to do it myself. For today’s article, I will teach you how to SSH login using only an RSA private-public key system.

Go back to your /etc/ssh/sshd_config file in your ArchLinux USB server and edit the following line as follows:

AllowUsers rsync_username

This ensures only the person logging in as the exact correct username can login to your machine. If a machine tries to login as any other username, they will automatically be denied permission.

Scroll down and edit the following line as follows:

X11Forwarding no

X11 is a program that allows Desktop GUIs to function. It is actually possible to open GUI windows that are actually running from the server onto the client’s computer. Imagine opening LibreOffice on your server and being able to see the application window running on the client computer you are SSH logging from? Isn’t that great. NO! Its not. It’s SLOOOOOW. You don’t need this feature to run rsync. So disable it with the above line.

Next we will set the limit to how long the SSH connection is allowed to remain idle without any response. Edit the following lines as follows:

ClientAliveInterval 300
ClientAliveCountMax 2

By setting “ClientAliveInterval” to “300”, you are telling your SSH server that if no activity is detected for 5 minutes (or 300 seconds) continously, the SSH server will send a request to your client asking for a response.

By setting “ClientAliveCountMax 2”, you are telling SSH server that it will try what you do in “ClientAliveInterval” twice in a row before forcibly shutting down the SSH login. Maybe you forgot to logout of your SSH connection. This ensures the SSH server will do it on your behalf if you forget.

But wait, you only configured your server to have this encryption. What about your client computer. That’s the computer that will ssh login to your SSH server.

In the terminal, type:

systemctl enable sshd
systemctl start sshd

Check if the sshd service (ssh daemon service) is running by typing:

systemctl status sshd

If you do not see the tags “active (running)”, it did not work. Go back and fix your mistakes.

So go to your client computer (that is another Linux or Unix computer and if not that use Cygwin on Windows). Now go into the file /etc/ssh/ssh_config file (yes that’s ssh_config without the ‘d’ after ssh) and make sure the following lines are in that file:

Port portnum_of_choice
ciphers chacha20-poly1305@openssh.com
macs hmac-sha2–512-etm@openssh.com
KexAlgorithms curve25519-sha256@libssh.org

This seals the deal for end-to-end encryption between you the client and your SSH server.

Now, if you have read the ESET report carefully you would quickly realize that password-based logins are in general a bad idea. To overcome this, you should set the /etc/ssh/sshd_config in your ArchLinux bootable USB server to only accept Public-Private Key-Based Authentication for SSH logins. I saved this step for last on purpose because its the most difficult one.

If you have followed all the steps so far, you should be able to login to your ArchLinux USB bootable flash drive with a simple SSH login and password for rsync_username:

ssh -p portnum_of_choice rsync_username@name_of_website

If you followed all the steps above correctly you will be greeted with a prompt that talks about ECSDA fingerprints. Just say yes to everything.

You will then be asked for the password for logging into the SSH server as rsync_username. Go ahead and type in the password.

If all is well, every line of your terminal should begin with rsync_username@name_of_website. You have successfully logged in.

But wait, you are not done. No, being able to login with a password does not cut it. You must disable SSH-based password logins and instead use PubKeyBased Logins. Perhaps you don’t care about additional security. Okay, but if you want rsync to work properly without you having to type the rsync_username password everytime you want to update the directory, then listen up.

I will now pass the mic to RedHat which has a great article on generating a GPG 4096-bit RSA-key from scratch. Afterwards, check out Ryan Lue’s tutorial on editing the key properly to be suitable for SSH Authentication.

Now, I will pass the mic to Ryan Lue who will instruct you on how to create a GPG key for authentication.

First Note: In his tutorial, Ryan Lue recommends to append the following in ~/.bash_profile:

export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket)
gpgconf --launch gpg-agent

I advise you to append the two lines in ~/.bashrc instead. This did a far better job of guaranteeing that the two lines above were executed by BASH before you logged into BASH. If you choose to keep BASH your default shell like I strongly recommended you too, then append the following two lines in BASH:

export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket)
gpgconf --launch gpg-agent
fish #or the command for whatever shell you prefer

Replace “fish” with the name of the shell of your choice. Maybe you will like the “zsh” (Z Shell) more.

Second Note: please make sure your default shell is the BASH shell. If it is not. His instructions will not work properly. Even if that was not true, I have noticed that there are BASH scripts designed to make the SSH encryption/decryption steps work properly. If your system is not BASH by default, you will need to change the default shell to BASH on the client computer you intend to SSH login from:

chsh -s /bin/bash client_username

But you can easily keep using your favorite shell seamlessly. Just add the name of your favorite shell to the very bottom of the .bashrc file. For me, its fish, so my .bashrc looks like the following:

#
# ~/.bashrc
#
# If not running interactively, don't do anything
[[ $- != *i* ]] && return
alias ls='ls --color=auto'
PS1='[\u@\h \W]\$ '
export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket)
gpgconf --launch gpg-agent
fish # or the command of whatever shell you prefer

Whatever shell you like to use, follow all of Ryan Lue’s steps exactly.

You by now should have finished Ryan Lue’s tutorial.

Just one last thing about Ryan Lue. I did mention you can easily keep BASH as your default shell but you can still login to your favorite shell on top of that to make it seem like your favorite shell is your default shell. If you followed my advice of calling your favorite shell directly from .bashrc, just make sure that when you use your favorite shell the following works:

ssh-add -l

This should list the RSA SHA2 key for your SSH authentication, regardless of the shell you are currently in. If you do not see anything, then press Ctrl+D to exit your favorite shell and fallback into BASH.

Following Ryan Lue’s instructions, you need to type the following in BASH:

gpgconf --kill gpg-agent
source ~/.bash_profile

If you followed my advice on taking advantage of .bashrc to automatically login to your favorite shell at login, then you should be able to now type

ssh-add -l

and see your RSA SHA2 hash in your favorite shell.

If not, then press Ctrl+D to exit your favorite shell to fallback into BASH.

Do the following again in BASH:

gpgconf --kill gpg-agent
source ~/.bash_profile

Now type:

ssh-add -l

directly in BASH. You should definitely see the SHA2-RSA hash for your GPG-authentication key. This has the label “[A]” when you type:

gpg -k --with-keygrip

Logout of your client computer and head to your ArchLinux USB server

1. Login to your ArchLinux USB server as root user.
2. You should already be in rsync_user’s home directory. If not, you better make the home directory for rsync_user now. In case it does not already exist (it should!), quickly logout of rsync_user, login as root user, and do the following:

mkdir /home/rsync_username
chmod 700 /home/rsync_username
chown -R rsync_username:rsync_username /home/rsync_username

What chown just did is that it made the true owner of the rsync_username home directory (and all its subdirectories and files)…rsync_username.

It also set the group owner to rsync_username.

Now check the file permissions

3. You need to create the .ssh directory.

mkdir /home/rsync_username/.ssh

Type the following:

ls -la /home/

Find the line that has rsync_username in it:

drwx------ 17 rsync_username rsync_username 4096 Jan 15 20:06 rsync_username/

Pay special attention to the “drwx…” part. Only you as a user and no one else, not even anyone that is part of group rsync_username (specified as the second rsync_username in the line) should have the privilege to open that file. If does not only say “drwx” for the first part of that line, then change the directory permissions as follows:

4. Change the chmod permissions of the .ssh directory

chmod 700 /home/rsync_username/.ssh

5. Create the “authorized_keys” file in the /home/rsync_username/.ssh directory.

touch .ssh/authorized_keys && chmod 600 .ssh/authorized_keys

Now logout of your ArchLinux USB server computer and return to your client computer.

1. Login to your username whose home directory contains your .gnupg/ directory.
2. Double check that:

ssh-add -l

Returns the SHA2-hash for your RSA Authentication key.

3. Type:

ssh-copy-id rsync_username@name_of_website

If all is well, you should be prompted for the password for your rsync_username. Type your password.

4. If all is well, the terminal will literally ask you to try SSH logging in without having to type the password.

5. The above step will automatically transfer the public key (that corresponds to your private RSA key) to your ArchLinux USB server.

6. If all is well, the terminal will literally ask you to try SSH logging in without having to type the password. But WAIT. You need to configure /etc/ssh/sshd_config on your ArchLinux USB server first.

1. Login to your ArchLinux USB server as root user.
2. Open the /etc/ssh/sshd_config file
3. Edit the following two lines so they look exactly as follows:

PasswordAuthentication no
PubkeyAuthentication yes

Now the only way to SSH login is if the private-pubkey system actually works.

4. Type:

systemctl restart sshd

5. Logout of your ArchLinux USB server and go to your client computer.

1. Login as the user of your client computer whose home directory has the .gnupg/ directory for your ArchLinux SSH login.
2. Configure your ~/.ssh/config file. If you wish to use your name of website to ssh login. Edit the following template at the top of the file:

Host name_of_website
Hostname IP.Address.Of.Website
User rsync_username
Port port_num

Add the same of set of lines below for each user that you wish to be able to login as on your SSH server.

3. Type:

ssh rsync_username@name_of_website

You should now have logged into your remote ArchLinux server.

4. Go ahead and explore your ArchLinux USB SSH server computer for 5 minutes and then logout once you are done celebrating with Ctrl+D.

1. Remember when you made your Rsync directory on your ArchLinux server. Now you must make a replica Rsync directory in the home directory that contains the .gnupg/ directory that contains the private key to login to your ArchUSB server.

Repeat the same steps you did to create your Rsync directory, except this time you are doing it in your client computer.

Remember, when typing:

ls -la

In your client computer’s home directory, you should see this for Rsync on your client computer:

drwxrwxrwx 44 client_username client_username  40960 Jan 15 18:53 Rsync

Not only should Rsync have these file permissions, but ALL files and subdirectories in Rsync should have these file permissions in your client computer. Oh, and by the way, you realize these same principles apply to the Rsync directory on your server, right?

The Rsync directory on your ArchUSB server should ALREADY look like this by now:

drwxrwxrwx 44 rsync_username rsync_username 40960 Jan 15 18:53 Rsync/

and yes, ALL the files and subdirectories in your server’s Rsync directory should look like that.

And now you are ready to start rsyncing. Put a test file in the Rsync directory on your client computer:

touch test.txt ~/Rsync/

Now here is the basic format I use for rsync:

rsync -AXzarp --info=progress2 -e 'ssh -p port_num' /home/client_username_that_has_.gnupg_directory/Rsync/ rsync_username@name_of_website:/home/rsync_username/Rsync/

This pushes all the files and subdirectories from your client’s Rsync/ directory into your server’s Rsync/ directory.

I’m sorry let me break down all those flags:

Rsync General Syntax: rsync [OPTION…] SRC… [DEST]
-A Preserves Attribution Control List
-X Preserves Extended Attributes
-z Compresses data before it it is sent online to the destination
-a It is the equivalent of the flag combination: -rlptgoD. It recursively preserves all permissions
-r Recursively gurantees synchronization of all files AND subdirectories in the target SRC directory, if SRC is a directory
-p Preserves partial permissions. Okay, this one is redundant here. But I decided to include it because it made the whole flag system easier to memorize for me.
--info=progress2 It will show you the download percentage progress live.
-e Specify command line option

What this means is that your server’s Rsync/ directory should either add or modify existing files in your ArchServer’s Rsync/ directory to look exactly like those in your client’s computer. There is one limit to this powerful feature: it will never delete in your ArchServer’s Rsync/ directory.

Did you see those forward slashes ‘/’ after Rsync “Rsync/” that you completely thought were annoying and redundant throughout this blog. They were put their for just reason.

If you leave out the forward slash ‘/’ after “Rsync” in the above command, Rsync will not only copy the contents of your client’s Rsync directory, it will literally copy the entire “Rsync/” directory in its total completeness.

That means that in your Rsync/ directory on your ArchUSB server you will literally see this:

drwxrwxrwx  2 rsync_username rsync_username 4096 Jan 15 22:25 Rsync/

Instead of this:

-rwxrwxrwx  1 rsync_username rsync_username       19 Jan 15 18:53 test.txt

Do not make the mistake I made the first few times.

One last note. When you wish to delete files from your ArchUSB server’s Rsync/ directory, you will need to include the delete flag in your command:

rsync -AXzarp --delete --info=progress2 -e 'ssh -p port_num' /home/client_username_that_has_.gnupg_directory/Rsync/ rsync_username@name_of_website:/home/rsync_username/Rsync/

This will not only add or modify existing files in your ArchServer’s Rsync/ directory, it will also delete any files and subdirectories in your server’s Rsync/ directory that do not exist in the client computer. So be careful with this command. I only encourage doing the command above when you are cleaning your Rsync/ directory on your server.

But what if you need to pull from your Archserver to your client computer since the Rsync/ directory on the server is more up-to-date than what’s on your client computer.

Well, you simply reverse the SRC… [DEST] directories. Just read the manual page for rsync:

rsync [OPTION...] SRC... [DEST]

So if you need to update your local Rsync/ directory on your client computer based on your ArchServer’s Rsync/ directory, you switch the source and destination:

rsync -AXzarp --info=progress2 -e 'ssh -p port_num' rsync_username@name_of_website:/home/rsync_username/Rsync/ /home/client_username_that_has_.gnupg_directory/Rsync/

If you need to delete files and directories on your client’s local Rsync/ directory based on your ArchServer’s Rsync/, you include the delete flag:

rsync -AXzarp --delete --info=progress2 -e 'ssh -p port_num' rsync_username@name_of_website:/home/rsync_username/Rsync/ /home/client_username_that_has_.gnupg_directory/Rsync/

And that is it. That’s how I manage my synchronization. If you wish, you can research how to setup a crontab to automate rsyncing your Rsync/ directory, but I like doing this manually on purpose, especially with the FISH shell at my fingertips.

Disclaimer: You will only be able to SSH login ONLY if you are in the same LAN as the server. If you wish to login outside of the LAN, you will need a VPN connection to your SSH server. But that is an article for another day.