A gentle introduction to rsync, a free, powerful tool for media ingest

It strikes me that many media professionals and filmmakers don’t use rsync, the free and open-source command-line utility available across Mac, Linux, and Windows. It’s a powerful, fast, lightweight, and mature tool that’s perfect not only for simple one-off media ingests, but also for complicated transfers involving many camera cards or hard drives. It can even be automated for reliable, efficient backups. Though it’s most commonly used by IT professionals, it’s highly relevant for filmmakers. DITs, data wranglers, and post-production professionals of all kinds will find it invaluable. Admittedly, the learning curve for rsync may be much steeper than that of other professional media ingest applications that have GUIs, but proficiency with rsync feels like a superpower.

What is rsync and why would I use it?

Arri’s “Data Management” page describes rsync:

Rsync is a free terminal application. It requires a computer expert to run a verified data transfer and even then, is rather cumbersome to use. If it is not already installed on your computer, you can find it at http://rsync.samba.org.

As far as I can tell, rsync remains superior to pretty much every other professional application for media ingest, like Imagine Products ShotPut Pro, Red Giant Offload, DaVinci Resolve’s Clone Tool, Pomfort Silverstack, or CopyToN. Each of these applications are great in their own rights, and they deliver what they promise, but they can be slow, expensive, and CPU-intensive. In contrast, rsync is fast, completely free of charge, and computationally lightweight.

Checksum verification, resuming interrupted transfers, and regular, automated backups

All professional-grade ingest apps use checksum verification, and rsync is no exception. Kylee Peña has written at CreativeCOW about the function and importance of checksum verification for media transfers. In short, according to Peña, checksum verification “is a method for detecting errors when copying a file from one place to another.” Checksum verification is an efficient way of ensuring that absolutely every byte in a source properly transfers to its destination. If you care about the integrity of your media, you should not be merely dragging and dropping files from one hard drive to another via Finder or Explorer, which offers no such protection.

To run a checksum on a file is to use a cryptographic method to give that file a kind of “fingerprint.” The “fingerprint” is a checksum, and has the form of a string of letters and numbers. If two large files were to differ even just by one bit, their checksums would almost always be drastically different. In the context of media transfers, checksum verification is the method of comparing a file’s checksum on source and destination media to make sure that every bit transfers properly.

Like other professional commercial media ingest programs, rsync verifies every bit it transfers by checksum, but rather than computing checksums separately on both the source and destination, it uses a “rolling checksum,” which is faster.

For a typical transfer, rsync compares filenames and file timestamps on the source and destination directory trees to assess which files should be transferred. However, if there’s any question of whether one or more files are in fact different but might share identical filenames and identical timestamps, rsync can be run in such a way so as to compare all the checksums on both the source and destination directories and then transfer only the items whose checksums differ.

Because of the comparisons across source and destination directories, whether by filename and timestamp only, or also by checksum, rsync can effectively resume transfers that have been halted or interrupted without having to transfer whatever data might have already successfully been transferred before the interruption. This also means that if you configurersync to run in an automated way, bandwidth isn’t wasted copying the same files repeatedly.

Log files and accountability

As with the other professional ingest applications, rsync can provide extensive log files, so that everyone involved in the chain of custody of the footage can document exactly how and when they would have transferred footage. The log files show all of rsync's activity during its transfers. If you need to go figure out whether a particular item was transferred, you can go check the log file.

Cross-platform

Because rsync can run on bash on any UNIX or UNIX-like system, it can be used on Apple macOS, on Linux, or on Microsoft Windows. macOS is UNIX and it’s default shell is bash, Linux distributions typically includebash for their default shells, and now, via the Windows Subsystem for Linux on Windows 10, bash can run on Ubuntu on Windows.

Command line basics

To understand how to use rsync, you need to understand how to use the command line, and more specifically, bash.

Broadly, there are two kinds of computer interfaces: command line interface (CLI), and graphical user interface (GUI). In the history of computing, CLIs were created first, and may be familiar via movie clichés like in War Games. Nowadays, for non-developers, GUIs are what most people use most of the time.

Using the command line is a way of interacting with the computer by entering lines of text that follow specific rules and syntaxes. What shell your computer uses and what specific program you’re using dictate what rules and syntaxes you need to use. Whereas a GUI might display windows, panels, buttons, dials, pictures, and text, the command line requires an understanding of what words and characters to type so that the program can run as you want.

So why use a CLI? The Linux Foundation’s introductory course explains that GUIs make easy tasks easier, but CLIs make difficult tasks possible. rsync’s set of features is impressive by itself, but the fact that it can be used in conjunction with bash’s other features and other command line programs like cron make it amazingly powerful and versatile.

For the purposes of this article, we’ll assume that you’re using the Terminal app within macOS specifically, but the core insights regarding bash and rsync apply to bash on Ubuntu on Windows as well as major Linux distributions.

bash basics

Navigating directories

When you first open a Terminal window, you will be using bash, which is macOS’s default shell. Within the Terminal, you can navigate among different files and folders on your system. These are the same files and folders you’re familiar with seeing in macOS Finder, but because this is the command line, you’ll have to navigate into and out of directories with specific bash commands.

In bash, you’re always operating within some folder on your system. This is your “present working directory.” You can always check what directory you’re in by typing pwd and hitting Enter.

Typically, when you first open Terminal, you’ll be in your user folder, like /Macintosh HD/Users/yourusername/. If you type ls and hit Enter, Terminal will list out the contents of the present working directory, which, if you’re in your user folder, would typically be the default subfolders:

Applications
Desktop
Documents
Downloads
Movies
Music
Pictures
Public

To navigate into one of these directories, type cd followed by the name of the directory itself. If you want to enter the Movies folder, you’d type cd Movies and hit Enter. Then, if you were to typepwd followed by the Enter key, Terminal would return /Macintosh HD/Users/yourusername/Movies/. So that’s your new present working directory.

To navigate back up a level to /Macintosh HD/Users/yourusername/, you can type cd .. and hit Enter.

This is an easy way of navigating to specific directories that are one level below or above your present working directory, because you’ll only have to type the immediate folder name, like Movies or Documents, but you can also navigate to any specific directory anywhere on the system by typing out the full pathname. So if you’re in /Macintosh HD/Users/yourusername/Desktop/Folder1, but you want to get to /Macintosh HD/Users/yourusername/Movies/Kubrick, you could get there by typing cd /Macintosh\ HD/Users/yourusername/Movies/Kubrick, and hitting Enter. You can also drag and drop a file or folder from a Finder window into a Terminal window, and the full pathname of the file or folder will populate, wherever the text cursor is in Terminal.

For convenience, the tilde character, ~, is a shortcut in a pathname for /Macintosh HD/Users/yourusername.

Spaces

It’s worth pointing out here that spaces in directories or filenames must be treated specially. Because bash’s syntax for commands uses spaces in specific ways, if a pathname or filename has a space in it, you must put a \ character ahead of the space to “escape” the space. “Escaping” the character is the term for specifying to the shell that the next character should be thought of as the exact character that’s being input specifically, and isn’t being interpreted as a space for bash's own syntax. If a file you see in Finder is named My Great Video Clip, that file would need to be written in bash as My\ Great\ Video\ Clip.

man pages

Within bash, there’s actually a compendium of documentation for most of the programs you’d want to use. These are the man pages, so-named for “manual.” If you’re ever curious about how particular commands work or what syntax to use to make a program work, you can get to a program’s manual by typing man [program] and typing Enter.

So, to see rsync's manual, you’d open up a Terminal window and type man rsync, and then hit Enter.

To navigate through man pages, press f to advance one page, press d to advance one-half page, press b to go back one page, or press u to go back one-half page. To exit a man page, press q.

You can also view an HTML version of rsync on the Web.

Unfortunately, many man pages function as detailed documentation for seasoned veterans to reference, without any kind of introductory explanations for beginners — rsync’s man page is no exception, which is why I’m writing this guide.

Typical syntax for commands in bash

In general, programs in bash typically take the form:

[name of program] -[flags] [relevant item] [relevant other item]

While this is an oversimplification, and many programs work differently, in general, the different sections of a command are separated out by spaces, and the man pages will document how to use the program.

[name of program] is whatever program you need to use. -[flags] are options to use to modify how the program runs. [relevant item] and [relevant other item] are whatever files or folders are being worked on by the program. The program might be moving, copying, joining, splitting, or renaming items, or reading or changing something within the items themselves. If you’re referencing an item within the present working directory, you can just type out the name of the item by itself, but if the item you’re referencing exists somewhere else on the system, outside the present working directory, you’ll need to type out the full pathname.

Compiling the latest version of rsync for macOS

Apple’s licensing issue with GPLv3

Before we dive into the specific details of how to use rsync, we actually need to take time to compile and install the latest version of rsync for macOS. As of this writing, bash on Ubuntu on Windows ships rsync 3.1.1 from Ubuntu 16.04.02, and current, major distributions of Linux ship the latest version, rsync 3.1.2, but macOS only ships an outdated version, rsync 2.6.9.

Apple only includes rsync 2.6.9 within the default installation of macOS because it’s the last version of rsync that used GPLv2, which is a specific software license for open-source software. After 2.6.9, the authors of rsync changed the license to GPLv3. Without getting involved in the GPLv2 vs. GPLv3 religious wars, just know that Apple refuses to ship GPLv3 software, so to use the latest and greatest version of rsync on macOS, you’ll need to compile it yourself.

How to compile rsync 3.1.2 on macOS

Whenever you plan to use a new macOS workstation, or if you have just completed a clean installation of macOS, you’ll need to download, compile, and install the latest version of rsync, which as of this writing, is 3.1.2. This only needs to happen once, and then you won’t have to think about it again.

Before upgrading to 3.1.2, you can check what version you’re running by entering rsync --version.

In a Terminal window, navigate to the Desktop to make it your present working directory.

cd ~/Desktop

Then, download rsync from its website.

curl -O http://rsync.samba.org/ftp/rsync/src/rsync-3.1.2.tar.gz

The .tar file format is a “tape archive” file, often referred to as a “tarball.” It’s a file format for material that has been compressed, similar to the more commonly used .zip. So, to access the content inside this tarball, you’ll need to uncompress it with the tar utility.

tar -xzvf rsync-3.1.2.tar.gz

This will create a new folder on the Desktop, your present working directory. The Desktop should now have a folder named rsync-3.1.2. Since you’re done uncompressing the tarball, you can go ahead and delete it.

rm rsync-3.1.2.tar.gz

Navigate into the rsync-3.1.2 directory.

cd ~/Desktop/rsync-3.1.2

Next, prepare the software. For the next few commands, you may be prompted to download Xcode to proceed. Xcode is an integrated development environment (IDE) created and maintained by Apple.

./prepare-source

Then, configure the software.

./configure

Now, begin compiling it.

make

Then, move all of the assorted files for the program to their proper locations within the operating system.

sudo make install

sudo is a way of escalating security privileges. You’ll need to enter your user password to verify that yes, you do want to install this software. This is proper security architecture. You wouldn’t want malicious programs to be able to install themselves without requiring a password. When the command line prompts you to enter your user password, it won’t display that you’re typing characters, but the characters are registering. When you’re done typing your password, hit Enter to proceed.

In case you’re curious, George Brocklehurst has written about how the commands ./configure, make, and make install work.

Once the sudo make install command completes, quit the Terminal application. Open Terminal again and enter rsync --version again to verify that you’re running 3.1.2.

Once you’ve verified that 3.1.2 is properly installed, you can delete the folder on your desktop by running rm -rf ~/Desktop/rsync-3.1.2.

Using rsync for media ingest

rsync has many powerful features, but one of the simplest and most effective ways to use it is for a simple media ingest — ingesting the contents of camera cards or hard drives to other hard drives, RAIDs or NAS devices. It’s useful on a DIT cart or within a post-production facility.

Using flags

rsync has a handful of flags that are particularly useful for media ingest. Recall that flags in commands are used to exercise particular options for programs.

Flags in rsync can take two forms — single letters, which can be collapsed together after one hyphen, and whole words or phrases, which typically are separated out by spaces, and preceded by two hyphens.

Flags are case-sensitive.

If you wanted to use the flags -a, -v, and -z, you could run either the command rsync -avz [source] [destination] or rsync -a -v -z [source] [destination]. It’s usually just more convenient to collapse all of the single-letter flags into the former form.

A suggested rsync command for a one-off ingest

One effective way to use rsync for an ingest would be to use the command as follows:

rsync -aihW --progress --log-file=[destination]/ingest.log [source] [destination]
  1. The -aflag, which also has the long form --archive, is a great general-purpose copying mechanism. Unfortunately, the manual isn’t very clear for beginners, but the -a flag is a way of copying pretty much everything you’d want to copy from your source directory to your destination directory. It informs rsyncto go through every directory recursively and queue all of the important attributes to be transferred to the source destinations — the files themselves, as well as file system metadata like Date Created, Date Modified, etc.
  2. The -iflag, which also has the long form --itemize-changes, shows some details about what’s happening to each file. For a typical ingest, this will just show that you’re copying a particular item, but for instances in which an item already exists on the destination, rsyncwill inform you that the item is already “up to date.”
  3. The -h flag, which also has the long form --human-readable, changes the readout from bits to bytes, which is a bit more intuitive.
  4. The -W flag, which also has the long form --whole-file, forces rsync to forgo its famous “delta-transfer algorithm.” The delta-transfer algorithm is used primarily for textual data, like code, scripts, or databases, and transfers just the portions of the files that have changed, which saves bandwidth. It’s a nice feature for textual data, but isn’t relevant for media files, so there’s no need to waste the computational power.
  5. The --progress flag shows progress in the Terminal window for the current transfer. The man page actually has a pretty good description for this flag, “This option tells rsync to print information showing the progress of the transfer. This gives a bored user something to watch.”
  6. The --log-file flag creates a log file. If you just list something like ingest.log after the =, rsync will append information about what it’s doing into the log file named ingest.log in the present working directory. You can also use a full pathname after the = to append activity into a log file within any directory on your system. If you need to specify a long pathname for a log file outside your present working directory, you can drag and drop the file into Terminal from Finder. Dragging and dropping from Finder will populate the full pathname for the file wherever the text cursor is in Terminal, and will even automatically escape spaces within the pathname with \ as needed.
  7. Lastly, the [source] and [destination] portions of the command specify exactly your source and destination. Your source can be a file, folder, or the contents of a folder. As with the--log-file flag, you can use the GUI to drag a file or folder from a Finder window into the Terminal window to populate the file’s full pathname, which will also automatically escape spaces within the pathname with \ as needed.

The presence of or absence of a / at the end of a directory path distinguishes a directory from the contents of a directory. Using FolderA as the source will transfer FolderA and all of it’s contents into the destination, but specifyingFolderA/ as the source will transfer only the contents of FolderA into the destination directory.

Suggestions for how best to use log files

Log files are great accountability tools for DITs, data wranglers, or anyone in the chain of custody for valuable footage. If ever a particular clip goes missing in the post-production process, you can go back and consult the log file to verify that a particular file was indeed transferred at a particular time.

.log file is just a kind of text file that’s specially designated by the operating system. In macOS, the Console app opens log files by default, and provides some specialized tools to navigate through them, because they can often get quite long. There’s nothing particularly special about them, though — .log files can also be opened up in any text editor.

If the particular .log file you specify in the --log-file flag after the = doesn’t yet exist, rsync will create a new .log file for you and begin to log activity into that file.

Sometimes it’s useful to create a completely blank .log file before you start any kind of rsync transfer, because you can then use the GUI to populate the long pathname instead of having to type it out manually. For instance, if you know you want to log the activity of several different rsync transfers into one log file, you could have a Finder window open, side by side with the Terminal window, and then drag a blank .log file from Finder into the Terminal window once your text cursor is immediately after the =.

To create a blank log file, first cd into the directory where you want the log file to exist. Then, use the touch command to name your new blank log file.

If you wanted to create a log file named ingest.log on your Desktop, you’d enter:

cd ~/Desktop
touch ingest.log

Comparing source and destination directories by checksum

If you believe that you might have some files in the source and destination directories that are indeed different but somehow have identical filenames and timestamps, you can add in the -c flag to compare everything within the source and destination directories by checksum to determine what items to transfer. Be forewarned though — this is extremely computationally taxing, and can take quite a long time, especially for large media files, so you only want to use it if you absolutely have to, and only on the smallest possible directories in question.

Queuing multiple rsync transfers

One of the great advantages about using rsync inside of bash is that, aside from being able to do one-off ingests and transfers, you can also use bash's syntax to queue up multiple commands to run successively.

Let’s say you’re ingesting a day’s media to a hard drive, and you have two camera cards that you need to transfer to one hard drive. Perhaps the maximum write speed of the hard drive is 100 MB/s, but your camera cards in your camera card readers can read at 400 MB/s.

If you were to open up two different Terminal windows and start ingesting both cards to the one hard drive simultaneously, your performance would suffer below 50 MB/s for each card, because the drive would effectively be fighting itself trying to write data to two different directories.

A faster and more effective workflow would be to use bash's syntax to queue up two different rsync commands — to ingest the first card entirely, taking full advantage of the hard drive’s full 100 MB/s speed, and then to automatically ingest the second card completely, continuing to take advantage of the hard drive’s full 100 MB/s write speed.

The bash manual describes in-depth the different ways this can be accomplished with different “control operators,” but for our purposes, one of the easiest and most effective ways to queue commands together is with one particular control operator, the double ampersand: &&.

Stringing commands together with && allows each command to execute sequentially, while also verifying that each command is executing properly as they go. Using && will mean that if any one particular command in the sequence errs, the whole sequence of commands will halt, at which point you can view the error message and diagnose whatever is wrong. Maybe your hard drive is full, or maybe you typed something wrong, but you’ll be able to pause and investigate.

Let’s say we have two camera cards, A001 and A002, that we want to ingest to Hard-Drive. We would need two rsync commands:

rsync -aihW --progress --log-file=/Volumes/Hard-Drive/ingest.log /Volumes/Camera-A001 /Volumes/Hard-Drive

and

rsync -aihW --progress --log-file=/Volumes/Hard-Drive/ingest.log /Volumes/Camera-A002 /Volumes/Hard-Drive

To queue one after the other, in one line, we’d combine the two commands with &&:

rsync -aihW --progress --log-file=/Volumes/Hard-Drive/ingest.log /Volumes/Camera-A001 /Volumes/Hard-Drive && rsync -aihW --progress --log-file=/Volumes/Hard-Drive/ingest.log /Volumes/Camera-A002 /Volumes/Hard-Drive

The camera card Camera-A001 would transfer into Hard-Drive, and the activity would get logged into ingest.log, and then Camera-A002 would transfer into Hard-Drive, and that activity would get logged into ingest.log as well.

You could string together as many commands as you need. If you had another camera card mounted, Camera-A003, you could run:

rsync -aihW --progress --log-file=/Volumes/Hard-Drive/ingest.log /Volumes/Camera-A001 /Volumes/Hard-Drive && rsync -aihW --progress --log-file=/Volumes/Hard-Drive/ingest.log /Volumes/Camera-A002 /Volumes/Hard-Drive && rsync -aihW --progress --log-file=/Volumes/Hard-Drive/ingest.log /Volumes/Camera-A003 /Volumes/Hard-Drive

You’ll notice that as you queue up more and more commands in this way, you’ll find it difficult and annoying to look at a giant block of text. It can be helpful to separate out each command with the escape key — after typing &&, type a space, then \, and then hit Enter. The escape key will let you press enter without starting to execute your list of commands. So instead of typing out command1 && command2 && command3 into one jumbled mass of hard-to-read text, you can type out:

command1 && \
command2 && \
command3

Separating each command into its own line keeps everything nice, clean, and legible.

Staggering rsync transfers for maximum speed and efficiency

Typically, it’s best practice during production for a DIT or data wrangler to back up camera cards in triplicate. With a little planning and deliberation, you can queue up rsync jobs in different Terminal windows on the same computer, and keep all transfers running at their maximum speed.

Let’s say, for a day’s shoot, we have nine different camera cards to ingest:

  1. A001
  2. A002
  3. A003
  4. A004
  5. A005
  6. A006
  7. A007
  8. A008
  9. A009

For simplicity’s sake, let’s assume that these are all the same size, and that they’re all full.

Let’s also assume that you want to get all nine cards onto three different hard drives that are identical makes and models, and have the same capacities:

  1. Hard-Drive-1
  2. Hard-Drive-2
  3. Hard-Drive-3

Finally, let’s also assume that all of the camera cards can read faster than the hard drives can write — the write speeds of the hard drives are the bottlenecks.

What’s the fastest and most efficient way to run through all of these cards, in such a way that each hard drive is receiving the contents of each camera card at maximum speed? The solution is to queue up the rsync transfers in a staggered way.

With some careful planning, we can set up our commands so that:

First,

  1. A001, A002, and A003 ingest onto Hard-Drive-1; while
  2. A004, A005, and A006 ingest onto Hard-Drive-2; and
  3. A007, A008, and A009 ingest onto Hard-Drive-3.

Then,

  1. A004, A005, and A006 ingest onto Hard-Drive-1; while
  2. A007, A008, and A009 ingest onto Hard-Drive-2; and
  3. A001, A002, and A003 ingest onto Hard-Drive-3.

Finally,

  1. A007, A008, and A009 ingest onto Hard-Drive-1; while
  2. A001, A002, and A003 ingest onto Hard-Drive-2; and
  3. A004, A005, and A006 ingest onto Hard-Drive 3.

How would we accomplish this? You’d mount all the cards and hard drives, and then open three different Terminal windows.

In one Terminal window, you’d write, for Hard-Drive-1:

rsync -aihW --progress --log-file=/Volumes/Hard-Drive-1/ingest.log /Volumes/A001 /Volumes/Hard-Drive-1 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-1/ingest.log /Volumes/A002 /Volumes/Hard-Drive-1 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-1/ingest.log /Volumes/A003 /Volumes/Hard-Drive-1 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-1/ingest.log /Volumes/A004 /Volumes/Hard-Drive-1 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-1/ingest.log /Volumes/A005 /Volumes/Hard-Drive-1 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-1/ingest.log /Volumes/A006 /Volumes/Hard-Drive-1 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-1/ingest.log /Volumes/A007 /Volumes/Hard-Drive-1 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-1/ingest.log /Volumes/A008 /Volumes/Hard-Drive-1 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-1/ingest.log /Volumes/A009 /Volumes/Hard-Drive-1

In the next Terminal window, you’d write for Hard-Drive-2:

rsync -aihW --progress --log-file=/Volumes/Hard-Drive-2/ingest.log /Volumes/A004 /Volumes/Hard-Drive-2 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-2/ingest.log /Volumes/A005 /Volumes/Hard-Drive-2 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-2/ingest.log /Volumes/A006 /Volumes/Hard-Drive-2 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-2/ingest.log /Volumes/A007 /Volumes/Hard-Drive-2 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-2/ingest.log /Volumes/A008 /Volumes/Hard-Drive-2 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-2/ingest.log /Volumes/A009 /Volumes/Hard-Drive-2 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-2/ingest.log /Volumes/A001 /Volumes/Hard-Drive-2 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-2/ingest.log /Volumes/A002 /Volumes/Hard-Drive-2 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-2/ingest.log /Volumes/A003 /Volumes/Hard-Drive-2

In the third Terminal window, you’d write for Hard-Drive-3:

rsync -aihW --progress --log-file=/Volumes/Hard-Drive-3/ingest.log /Volumes/A007 /Volumes/Hard-Drive-3 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-3/ingest.log /Volumes/A008 /Volumes/Hard-Drive-3 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-3/ingest.log /Volumes/A009 /Volumes/Hard-Drive-3 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-3/ingest.log /Volumes/A001 /Volumes/Hard-Drive-3 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-3/ingest.log /Volumes/A002 /Volumes/Hard-Drive-3 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-3/ingest.log /Volumes/A003 /Volumes/Hard-Drive-3 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-3/ingest.log /Volumes/A004 /Volumes/Hard-Drive-3 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-3/ingest.log /Volumes/A005 /Volumes/Hard-Drive-3 && \
rsync -aihW --progress --log-file=/Volumes/Hard-Drive-3/ingest.log /Volumes/A006 /Volumes/Hard-Drive-3

Then, you’d cycle through each Terminal window and hit Enter to start these three transfers simultaneously. Your CPU will be running three different rsync threads, but recall that your speed bottleneck won’t be CPU power — it will be the write speed of the hard drives. With this method, each hard drive will be writing at its own maximum write speed.

Automated backups with cron

While a full explanation is outside the scope of this article, I’ll just mention that one more practical application of rsync is to use it within another command line application, cron. If you’re curious, can can go learn how to usecron from its man page. Essentially, cron lets you run commands at specific time intervals. Because rsync's -a flag keeps track of what files in a destination directory already exist, scheduling rsync to run regularly with the -a flag is a great way to backup files regularly, without wasting time or bandwidth on material that has already been backed up.

Conclusion

I hope you find this explanation useful. Please leave a comment to get in touch with me for any comments, questions, or respectful criticism.