How to recover video from an SD card if you accidentally reformat it

Here’s a youtube video where I explain the same process

We all make mistakes. I recently deleted a whole folder of important video from an SD card, and then emptied the bin on my mac, permanently deleting everything without realizing I had no backup.

I cried and I panicked, but found a solution. This recovery method works whether you delete files or accidentally format a drive. Read on.

TL;DR testdisk + mp4fixer. Skip to the bottom for helpful scripts.

How I accidentally wiped my whole SD card:

In the good robot lab, we’ve got loads and loads of SD cards. Most of them carry some flavor of linux and power a raspberry pi, but a few live in cameras in case we need to capture a robot’s first baby steps, or first words or something.

Unlike our very cute robots, we ourselves are stumbling, bumbling humans, and that means we tend to be a little bit lazy, and that means we’re vulnerable to the consequences of our laziness.

On one fateful day, I was crooning over some really cute footage of our first prototype, Diana, wheeling around, sweeping up plastic for recycling, and just being, well, so doggone cute!

i should probably just get an actual dog

I was building a new bot and needed to format a card to load linux on it, so I stuck a new card in my laptop and hit format. All good!

While the new linux image was racing down silicon highway to burn itself on to the card, I went back to giggling at the adorable way the motorized camera mount on Diana makes her head move in an a, like, totally human way.

I couldn’t find the videos on the card anymore, though. And then I noticed that both cards attached to my computer were 32gb drives named NO NAME.

I formatted the WRONG CARD.

I stopped the image write and began sweating profusely. In addition to all of the cute videos of our pet robots, were a lot of one off shots we had planned to put in an upcoming kickstarter video.

Always make backups

Since you’re here, here’s a great post we should all tattoo on the insides of our brains about the importance of backups. Also, don’t treat your camera SD card as permanent storage. Don’t just tell yourself you’ll get to organizing and copying files over later. You won’t!

I panicked and sweated for about ten more minutes, and then slowly moved my way through the rest of the stages of grieving, and made my way eventually to acceptance.

But in case you don’t..

Then, I calmed the hell down and I remembered that sometimes, people design computers to be lazy as well. When you format a drive or delete a file, you delete it from an index which basically tells your computer where the 1s and 0s that represent the file are, but you don’t actually wipe those 1s and 0s, until you replace them later with another file. You delete the reference to the file, but not the data itself.

This means that you can use clever file recovery tools to magic back files if you accidentally delete them (even if you empty the trash!) or if you do a high level format. A low level format actually wipes all of the bits on a drive, and if you do that, you’re hosed!

I googled ‘data recovery mac’, and I ended up trying this program called Disk Drill. I was impressed by the great UI, and then I jumped for joy when an initial scan turned up all of my .mov files!

I hit recover to copy the files back, and BAM! I was asked to cough up 100 bucks to upgrade to the pro version, because the free version only previews files.

Sounds like blackmail to me!

There’s probably an open source solution

People have designed, programmed, and given away much more complicated stuff for free, so I figured there might be something open source out there that would do the trick.

This time, I googled ‘FREE data recovery mac.’

I discovered testdisk.

I’ll spare you a long introduction to the program — you can read it on their page. It’s an awesome industrial strength file recovery tool which recognizes metadata from 400+ file formats, managed by a guy named Christophe who has probably single handedly saved millions of idiots like me.

I ran it on an old mac (the program is 32 bit) and was stoked when it recovered all of the same files as Disk Drill. All my images, text files, and the other junk on the drive worked perfectly.

But the .mov files wouldn’t play.

Weirdly, I had about 80 .mp4 files, all 24 bytes long, which seemed to match up to the .mov files which my camera records in.

I googled again, and learned that .mov is a container format, which normally holds an ‘ftyp’ header, a ‘moov’ atom, and a ‘mdat’ blob which is the raw video stream.

The .mp4 files were supposed to be the headers, and some people on the forums had luck simply concatenating them with the .mov files, but no dice for me.

I even found a script that promised to programatically match up the .mp4 headers with the .mov files, in case they’re mixed up, but again no luck, just blank files.

I inspected the reconstructed files, the .mov files, the .mp4s, and some new video recorded from the same camera in a hex editor.

The .mp4s all contained the same text — “ftypmp42mp42isom”

The newly recorded files, though, contained a lot of information that wasn’t in either the .mp4 or the .mov files, including an apparently quite important ‘moov’ atom.

I tried copy and pasting the full header in to one of the recovered .mov files.

Quicktime played it, it showed up as the correct length, but it was all black — no video.

After several hours of this, I had basically given up hope. Grief set in a second time. I was frustrated, because it seemed like the data was still there, barely out of reach, the file sizes were almost right, I just couldn’t figure out how to reconstruct the headers.

TL;DR: Here’s the solution

And then I discovered this (free) perl script, mp4fixer.

You feed it a new 20 second recording from the same camera, and then any kind of garbage, ruined, corrupted video data of any kind, and it tries to recreate the headers using the good video.

I made a new recording, with the same camera, and fed it to the script along with the mdat containing the video blob…

perl fixer.pl new-video.mov recovered-mdat.mov new-filename

… and it worked!!!!

Got! 67315 bytes and 683 shit
testing at 2433b86c (of 26f1dc) gives us 0
recovered new-filename-out-video.h264

I was able to recover .h264 files, and preview them with ffplay.

ffplay -f h264 new-filename-out-video.h264

But there’s no audio!

To recover audio, I used the aacfixer utility included with mp4fixer . The recovery generates recovered-mdat-headers.aacand recovered-mdat-out-audio.raw

Recover audio:./aacfixer recovered-mdat-headers.aac recovered-mdat-out-audio.raw and join the recovered audio to your new-filename video from the last step ffmpeg -i recovered-mdat-out-pure-adts.aac -i new-filename -c copy new-filename-with-audio

And here’s the video we recovered!

Here’s some of that pieced together robot videos from our Good Robot pitch, for your viewing pleasure:

The whole process, step by step

1. Use testdisk to recover all the files on your drive
2. Record a 20+ second new-video.mov on the same camera (doesn’t matter what you film, you just need the headers
3. Use mp4fixer to repair the recovered stuff: perl fixer.pl new-video.mov recovered-mdat.mov new-filename
4. Recover audio:./aacfixer recovered-mdat-headers.aac recovered-mdat-out-audio.raw and join the recovered audio to your new_filename video from the last step ffmpeg -i recovered-mdat-out-pure-adts.aac -i new-filename -c copy new-filename-with-audio
5. If running in batch, use a script, something like:
   below:

#!/bin/bash
for i in recovered/*.mov
do
  perl fixer.pl new-video.mov $i $i
done

6. And for audio:

#!/bin/bash
for i in recovered/*.aac
do
 ./aacfixer $i ${i%-*}"-out-audio.raw"
 ffmpeg -i ${i%-*}-out-audio.raw-pure-adts.aac -i ${i%-*}-out-video.h264 -c copy $i.mp4
done