How to Backup Files Using Node.js and Rsync

Tate Galbraith
Jan 7 · 6 min read
Image for post
Image for post
Photo by Markus Winkler on Unsplash

Managing backups is the technological equivalent to the big, ugly elephant in the room. Everyone knows its there, but nobody wants to deal with it. What’s worse, there are hundreds of different ways to implement a backup solution. You can purchase some off-the-shelf software, use an open-source solution or roll-your-own custom project. There are pros and cons to each type of solution and depending on your environment and the sensitivity of data you might not even have much choice.

If you’re tired of using third-party tools that don’t perform the way you want and you have the flexibility to settle on a bespoke path forward you should consider building your own solution. When you craft a custom backup solution you have total control over when, how and where the data gets backed up. The best part is you can write it using languages and programs you’re already familiar with.

In this article we’ll focus on building a custom backup solution. We’ll use familiar, free tools to accomplish this and the code will be straightforward and easily extensible. Node.js is a massively popular backend built on JavaScript and allows you to integrate your solution with almost any modern framework. If you haven’t heard of Rsync before, it is easily one of the most robust and efficient backup applications available. We’ll combine these two elements to create a simple, effective backup application you can tailor to your exact needs. Let’s get started.

For this guide I’m going to assume you already have Node.js installed, but if you don’t head over to nodejs.org for instructions. If you’re using macOS and have brew installed, you can simply run:

brew install node

First let’s setup a new project folder (feel free to drop the -y to specify your own package details):

mkdir backup && cd backup
npm init -y

Now add the following entry to your package.json in the “scripts” section to run the app quickly using npm start:

"scripts": {
"start": "node index.js"
}

Create an index.js file in the same directory. This is where we’ll store our application code. Next, we’ll setup packages and build the app.

For this project we will need the rsync package. Let’s add it using npm now:

npm install rsync

Now edit the index.js file created earlier and add the following code:

const Rsync = require('rsync');
const rsync = new Rsync();
const rsyncSource = './test_data/src/';
const rsyncDest = './test_data/dest/';
function runRsync() {
rsync.flags('avzP');
rsync.set('delete');
rsync.source(rsyncSource);
rsync.destination(rsyncDest);
return new Promise((resolve, reject) => {
try {
let logData = "";
rsync.execute(
(error, code, cmd) => {
resolve({ error, code, cmd, data: logData });
},
(data) => {
logData += data;
},
(err) => {
logData += err;
}
);
} catch (error) {
reject(error);
}
});
}
(async () => {
console.log('starting rsync');
let output = await runRsync();
console.log(output);
console.log('rsync complete');
})();

Let’s work through what some of this code is doing, piece-by-piece.

const Rsync = require('rsync');
const rsync = new Rsync();
const rsyncSource = './test_data/src/';
const rsyncDest = './test_data/dest/';
...

First, we pull in the rsync package and instantiate Rsync. This produces a new Rsync object that we can set some state on before running our backup operation. Following the new object we create two variables for holding our desired source/destination backup directories relative to the current directory. For this project we’ll just specify some test directories (more on these later, but note the trailing slash on the end of the path).

function runRsync() {
rsync.flags('avzP');
rsync.set('delete');
rsync.source(rsyncSource);
rsync.destination(rsyncDest);
...

Next, we create our backup function. This function is called runRsync and handles configuring and executing rsync. Here we can specify how we want rsync to handle the process and whether or not to mirror the directories, etc. The delete option will keep the directories in sync and remove files in the destination that were deleted in the source. For an extensive list of options the official rsync package GitHub is available here.

  ...  return new Promise((resolve, reject) => {
try {
let logData = "";
rsync.execute(
(error, code, cmd) => {
resolve({ error, code, cmd, data: logData });
},
(data) => {
logData += data;
},
(err) => {
logData += err;
}
);
} catch (error) {
reject(error);
}
});
}
...

After we configure rsync the next step in our runRsync function is to execute the actual process itself. For this we call rsync.execute. Since .execute uses callback functions we’ll want to wrap it in a Promise so we can easily use async/await later on. To learn more about promises and why they’re useful, check out Understanding Promises in JavaScript by Gokul N K.

The (data) and (err) callbacks stream output data from the underlying process. Since we don’t want the promise to resolve quite yet, we add the buffered output to a variable outside the .execute function.

The (error, code, cmd) function is called once the process has completed so we’ll resolve our promise here. For this we just return an object containing all of our data from both the streams and the final error code, etc. The cmd variable will actually show us the raw rsync command as it was executed.

In the event the rsync process fails to launch or the rsync object is configured incorrectly, our catch block should handle the error. This is where we’ll reject the promise with whatever error has occurred.

(async () => {
console.log('starting rsync');
let output = await runRsync();
console.log(output);
console.log('rsync complete');
})();

Finally, we call an anonymous async function at the end of our file and place our desired function, runRsync, inside. We use await in order to be sure that runRsync completes before proceeding to the next line. This last anonymous function will allow you to place other async functions before or after the call to runRsync and have each action occur sequentially based on the use of await. This is helpful if you want to have more verbose logging or send some kind of alert that the backup has completed. For this example, we’ll wait for runRsync to complete and then output the log.

Now that we have our application code wrapped up, we can start testing out our backup strategy. Earlier in our application code we specified our test source and destination directories in variables. Let’s put some test data in those directories now so we can see rsync in action:

mkdir -p test_data/src test_data/dest
touch test_data/src/file1 test_data/src/file2

Now that we have some files to backup, run the application using npm start and wait for the job to complete. Once finished, you should see the log output in the console and the same files in the dest directory as in src.

starting rsync
{
error: null,
code: 0,
cmd: 'rsync -avzP --delete ./test_data/src/ ./test_data/dest/',
data: 'building file list ... \n' +
' 0 files...\r3 files to consider\n' +
'./\n' +
'file1\n' +
' 0 100% 0.00kB/s 0:00:00 (xfer#1, to-check=1/3)\n' +
'file2\n' +
' 0 100% 0.00kB/s 0:00:00 (xfer#2, to-check=0/3)\n' +
'\n' +
'sent 178 bytes received 70 bytes 496.00 bytes/sec\n' +
'total size is 0 speedup is 0.00\n'
}
rsync complete

You should see log output similar to that above. We can see our returned object with the error details, return code, raw command and log output. If you run the app again, rsync won’t have anything new to do so the log will be slightly different and the app should execute faster.

Using rsync with this simple Node.js app unlocks a realm of possibilities for building your own custom backup solution. This example provides a solid foundation for designing an even more full-featured app on your own. You could add more detailed log output, support for multiple backup directories and even robust alerting or reports. An additional bonus is that since this is written in Node.js you have the ability to integrate it into other existing Node.js applications or popular frameworks. This example could be built into an API, web backend and much more.

Thank you for reading. After finishing this article I hope you’re able to see how rolling your own backup solution doesn’t have to be a massive, complicated undertaking. Check out the rsync man page for even more details on how the underlying program functions.

The Startup

Medium's largest active publication, followed by +775K people. Follow to join our community.

Tate Galbraith

Written by

Software Engineer @mixhalo & die-hard Rubyist. Amateur Radio operator with a love for old technology. Tweet at me: https://twitter.com/@Tate_Galbraith

The Startup

Medium's largest active publication, followed by +775K people. Follow to join our community.

Tate Galbraith

Written by

Software Engineer @mixhalo & die-hard Rubyist. Amateur Radio operator with a love for old technology. Tweet at me: https://twitter.com/@Tate_Galbraith

The Startup

Medium's largest active publication, followed by +775K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store