Media Composers: How To Organize (And Back Up) Your Data

Lacking a good backup system in place will come back to haunt you sooner or later.

If you’ve been using a computer for some time, it’s quite likely that you’ve heard over and over again that you should back up your data. And you might in fact already have some kind of system in place, local or offline, manual or automated, full or incremental. But, as a creator, how safe is your critical data exactly? Are you backing up everything you really should? And are you really ready in case something happens?

At a deeply personal level, we hope our data (photos, video clips, music, documents…) will never disappear, because the emotional toll of losing some or all of it would be devastating. MacOS has had a consumer-centric backup system, Time Machine, for more than ten years now, and since it came out with Mac OS X Leopard, it has helped many people easily set up a straightforward backup of their personal files.

The original Time Machine interface (screenshot courtesy of MacWorld)

As working professionals dealing with huge amounts of data, however, where that data is backed up and how quickly we can expect to retrieve it can become quite a complex issue. It is essential to plan well: outside of the emotional factor, losing any project-related data due to a poorly thought-out backup system could lead in worst cases to missed deliveries, damaged reputation, and financial liability. It might not destroy a career, but the ripple effects could be no less devastating that losing your personal photos. Simply put, it should never happen.

There are three common ways of losing data: technical failures (a drive that fails, for example), “acts of God” (natural disasters, fires, etc.) and human errors. Unfortunately, no matter how careful you are, one of these is bound to happen at some point: you can be lucky for some time, but you won’t escape basic statistics. Drives ultimately die — yes, even SSDs; natural disasters happen whether you want it or not; and no one is ever completely free from human error. As such, and based on more than a decade of experience dealing with large set of data and standard common sense practice, I designed my current rig so that, a) it insure the lowest chance possible of data being lost, b) it allows for easy and fast data recovery in case something does get lost, and c) all of that, at a cost I can actually afford.

Your Files, Organized

But before I talk about backing up data, it’s important to think about how you organize the different type of data that you use on a regular basis. As media composers, we typically deal on our workstations with a host of different files:

1. System files (anything that allows your system to run)
2. Personal documents (anything not related to specific projects)
3. Project-related files (DAW session files, audio files, documents, etc.)
4. Voluminous Project-related files (video, most typically)
5. Sample libraries
6. Archives (old project files)

Each type of files has different needs when it comes to ideal read/write speeds, access frequency, and back up requirements. As always, personal preferences based on habits, usage and workflow will vary. Below is, for example, my current drive set up.

To understand the reasoning behind this design, here are some elements that I took into consideration:

1. The iMac’s internal drive cannot be easily upgraded, so it can’t ever get full. At the same time, it’s also by far the fastest drive in my setup (almost up to 500MB/s read speed). I elected to only use it for the following specific set of files:
   a. System-related files (macOS, preference files)
   b. files that should reside on the system drive for technical reasons or just to keep things tidy and simple (Dropbox, iCloud Drive, Photo library, Logic presets, etc.)
   c. Video files for current projects only. These files are voluminous in nature (especially when dealing with uncompressed Full HD video), but also require high read performance.
2. My Projects and Libraries drives are stored on SSDs loaded in a Multidock rack made by Blackmagic Designs. It holds up to four standard 2.5" SSDs and connects to the iMac via a Thunderbolt 2 cable. Short of having all my project files on my system drive (which wouldn’t have enough space for all of them anyway), this guarantees fast read/write speeds (around 350MB/s), allows for quick loading of Logic sessions (with a huge amount of sample libraries), and insures fast Disk I/O during playback and recording.
3. When a Project is completed, I move its Project-related files to the Project Archives USB 3 drive to make space for new projects. The same happens to the Project-related video files stored on the System Drive which are moved to a USB 3 drive named Video Archives.
4. As a general rule, I wanted all my “day-to-day” files to be on SSDs with the fastest possible connection. Mechanical USB 3 drives are relegated to my Archives (i.e. files I rarely need to access) and a manual backup of my Libraries drive (which I only execute after I installed a new library, a rare occurence). I only mount these drives when I need to use them — they are usually unmounted and powered down.

All Drives Die Eventually—even SSDs

Since the switch to SSDs, many people have assumed that these memory-based drives are much more reliable than mechanical drives because they contain no moving parts. That’s true, especially when used in harsh conditions, like mobile use (laptops, etc.). But at the end of the day, plenty of things can go wrong with an SSD. Connectors can fail. Controllers can fail. RAID systems can fail. Software bugs happen, malfunctioning firmware updates happen, power surges happen, data corruption happen, physical damages happen.

Human error is also an important factor to consider, as it knows no technical boundaries. If you delete a file or folder by mistake—or, worse, format the wrong drive or erase a partition—having a recent backup can be a life-saver.

In short, whatever the technology you rely on: back up your data.

What Should Be Backed Up Where, and How?

Although most online backup services are inexpensive and fast — provided you have a broadband connection—there are good reasons to delineate what kind of data should be backed up locally or online (or both). Let’s look at the advantage for each:

1. On a per-gigabyte cost, local storage is extremely cheap. It’s easy and fast to back up to, and it’s easy and fast to retrieve huge sets of files if you ever need to.
2. But local storage is also faillible. It depends on the user (you!) doing regular backups, or at least setting up the right processes for it to happen automatically for you—and making sure these processes are running properly. The backup drives could die (see above). Let me write this again for emphasis: all drives die eventually—even SSDs. Your backup drives are also limited in space and could get full, requiring new upgrades or a painful process of getting rid of older incremental backups. And if an “act of God” were to happen (fire, flood, earthquake…), especially if they’re kept on site, those backup drives aren’t worth much more than your main drives once they’re all physically damaged.
3. On the other hand, online storage is always there—or, at the very least, you trust the backup company for multi-level redundancy and uninterrupted access should you need to access your files at any time. Depending on the service, there are often no storage limitations—and incremental backups are more and more the norm, meaning you can preview and retrieve files from multiple past “time states” (which is useful if you deleted a file by mistake). And the monthly costs are typically very competitive to the one-time purchase of a physical drive.
4. But online storage also depends on (still) inadequate broadband connections. Take the painfully-slow upload for the first backup, for example: if you have a significant amount of files (say over a terabyte), this process can take weeks. Weeks! While that happens, it’s also a potential drain on your CPU. And retrieving large sets of files can take an inordinate amount of time—sometimes requiring that a drive be shipped to you instead of simply downloading files from your browser. (This is quite understandable, especially if you’re trying to restore terabytes of data, but it can add to a significant delay in being functional again.)

Here again, as you can see, you will have to weight different considerations to figure out what works for you and your workflow. But whatever happens, you should have at least one backup somewhere, and if possible, two in different locations.

If we go back to the earlier diagram of my backup setup, I applied the following thoughts:

1. Unless one has unlimited amount of time and money, every decision is a compromise. There is no perfect solution.
2. Consequently, and since my time to set up, execute, and monitor the backups is limited, I have to be smart in how I design the system—I must do what is necessary for my data to be safe, but I shouldn’t go overboard and create something that is financially out-of-reach, or that I won’t be able to properly maintain in the long run.
3. If I keep data that doesn’t need to be backed up, then there’s probably no reason to keep the data in the first place.
4. When it comes to data recovery, the only parameter that matters is: if the data is really gone, how much time am I willing to go without it?

Based on the above thinking, and on the way my data is organized, I made the following backup decisions:

1. My most critical information is stored in two places: My System drive and my Project drive. Both should be backed up regularly, locally AND online. Both backups should be incremental so that I can find a file or folder that I deleted by mistake.
2. In case of a crash of my System drive, I don’t just need to have the data backed up; I need to have a system ready to go again as soon as possible. Even with a recent backup, re-installing a macOS system with the drivers, plug-ins, music and audio software, and countless sample libraries necessary to a functioning music workstation can easily take a full day. For busy composers, that’s usually a day you don’t have. That’s why I have a clone of my System drive, ready to go at a moment’s notice. (I regularly keep it up-to-date.)
3. My Library drive, although an essential element of my system, holds data that is less critical. If the drive dies, I can be up and running in no time with the manually synchronized backup that I do each time I install a new sample library. Additionally, I can usually re-download the libraries themselves from the manufacturers’ websites. And finally, that data is quite voluminous: my 2TB drive is pretty much filled to the max. Having a simple local backup is fine for that purpose.
4. Finally, my Archive drives (Project and Video) contain important information but which does not need to be available immediately if it ever got lost. As such, these drives are only backed up to an online service.

As you can see, it’s important to figure out how to organize your data in a way that makes sense to you and your workflow. Only then will you be able to figure out how you should back it up, in a safe, efficient, and economical way.

Case Studies

Over the past 15 years that I’ve been in the industry—as an assistant, a creative consultant, a composer, and a music producer—I have seen and heard countless stories about data being lost. On a professional level, this can be devastating, and I’ve learned some very hard lessons because of it. I can safely say that it doesn’t just happen to others. As a matter of fact, remember how I wrote at the beginning of this article that, in the last six months, I’ve had two very close calls with losing some of my data forever? Well… read on.

The Dead System

One morning, when booting up my computer, I realized something was wrong: it was repeatedly freezing during the loading sequence. I kept manually restarting, but to no avail. And eventually, I saw this dreaded icon flashing before my eyes:

That’s not the kind of icon you want to see when you boot your Mac.

I wasn’t sure what happened or how it happened. All I knew was that my only solution at that point was to re-install macOS. And, aside from the usual system and productivity apps, that meant re-installing a lot of software, plug-ins, and libraries related to my work as a professional composer, such as:

1. DAW and music-related software such as Logic, Sibelius, Pro Tools
2. Samplers such as Kontakt, Machine, Vienna Ensemble, Play
3. Countless instrument libraries from Spitfire, Cinesamples, Orchestral Tool, Project SAM, Output, East West, Spectrasonics, Synthogy, VSL…
4. Countless audio plug-ins from Audio Ease, Waves, SoundToys, PCM, Melodyne…
5. And the usual but workflow-critical productivity apps such as Slack, Wunderlist, Dropbox, Metaserver, Backblaze, MindNode, Name Mangler, File Maker Pro, etc.

Re-installing an OS itself can take a couple hours, but re-installing all these apps and libraries can take days—especially if you have to download 2TB of samples all over the again. And I certainly didn’t have that kind of time: I had a critical delivery deadline that next day on an important writing gig!

Thankfullly, all my important system files were backed up on a Time Machine drive. Within three or four hours, I had my operating system and music rig up and running again. Since all my instrument libraries were previously installed on a separate drive, I didn’t have to re-download anything; and since my Project-related files were also safely stored on a separate drive, these didn’t have to be copied over. Although I did lose some time that day, I ended up delivering on time simply by going to bed a bit later that evening (likely around 3–4 AM).

That Time Machine back up? Worth every penny.

The lesson I learned that day, however, was that it could have happened under an even more critical schedule. Say for example that I had to deliver something that same afternoon. What would I have done, then? And that’s why, later that week, I decided to create a clone of my System drive (courtesy of econ’s ChronoSync), to insure I could be up and running even faster if ever this situation happened again.

The Dead Archive

As described earlier, I use a USB-3 drive to archive project-related files when I am done working on a gig (Logic sequences, Pro Tools sessions, audio files, etc). Although the drive is always connected to my computer, I turn it on and off via a pretty neat USB hub with port-per-port power switches.

A few months ago, I needed to access files from a previous project for a current gig I was working on, so I turned on my Project Archives drive… and nothing happened. The volume wouldn't mount, and the drive didn’t even appear in Disk Utility. I had low expectations, but I tried using another USB-3 cable, using another USB hub, or connecting the drive directly to the computer, to no avail. I soon realized the drive was doing a constant “click-y” sound which I knew to be pretty much the equivalent of a flat-lining hearbeat monitor. As a last resort, I ended up opening the enclosure to extract the mechanical drive and plug it into one of my empty USB-3 enclosures—I figured it was worth voiding the warranty (and therefore any hopes of getting a replacement drive for free) if I could at least access the files right away. Unfortunately, even in the new enclosure, the drive didn’t show any sign of life. It was dead.

At this point, my options were pretty limited. But because I did have a backup system in place, I still had options—and that’s what mattered. As described earlier in this article, all my archive files are backed up online on Backblaze. Finding the files I needed for my current project was as simple as browsing through my backed up files on their website.

Within 5 minutes I had received an email from Backblaze with a link to a compressed archive of my files, and I was able to keep working. It was that simple!

I also immediately ordered a new drive to replace the defective one, and all I needed now was to get the rest of the files that I had backed up on Backblaze. The problem was that the total space they took was almost 5TB—it would have taken weeks for Backblaze to compress it all, and probably a few more days for me to download them all. At this point, it was easier and faster to use their Hard Drive Recovery service.

The nice thing about this service is that although it costs $189 per drive (including shipping), if you send back the drive to Backblaze within 30 days, you will get a full refund—so it only ends up costing you the shipping fee to send it back! This is definitely a great option for voluminous backups. In my case, since the files were over 4TB, I had to order two drives.

That same day, Backblaze sent me an email to confirm the drives were being prepared.

Four days later, I received another email letting me know the drives had shipped.

And another four days later, on September 28th, I received the two drives which Backblaze had sent via FedEx.

The box includes the drive and a USB cable, all well protected with padding foam.

From that point on, it was as simple transferring all the files to my new Archive drive. (For security, the drives are encrypted; to be able to unlock them and access your files, you need to log in to your Backblaze account to get a drive-specific passcode.)

A few weeks later, I went to my local FedEx shop and shipped back the drives to Backblaze. And after a few days, I received a final email from Backblaze confirming they had refunded $189 for each drive I had ordered.

All in all, this was a great experience and I can’t recommend Backblaze highly enough. It’s a great service which is very easy to set up and which you completly forget about as it operates in the background. Yet it’s always there for you when you need it, and offers many options to get your data back. It’s also affordably priced. What’s not to love?

As you can see, having a proper backup system is essential, especially if you rely on your data for work. As media composers—and, I think, for most creators using a computer for a living—it’s absolutely critical for our data to be organized in a way that makes it easy to retrieve if something happens to it, whether it’s human error, mechanical failure, or some kind of natural disaster.

Storage has never been cheaper and, thanks to high-speed internet connections, online backup services are plentiful and easier to set up and use than ever. In short, there are no excuse to wait on setting up the ideal backup system for your data. As I hope my real-life examples demonstrate, it’s an insurance policy you can’t afford to live without. So… get to it!

I hope you found this article informative. As always, thanks for reading! Don’t hesitate to reach out via email for general comments or inquiries. You can also follow me on Twitter, Facebook or Instagram.

Finally, if you’d like to learn more about me and what I do, head on over to my website, where you can read my bio, listen to my reel, and check out my other album releases.