Working With RAW
How this flexible format can help make your matchmoving easier
In an interesting twist, the photography industry is in some ways far ahead of the film industry. They were the first to pickup and embrace the digital workflow long before it was even considered a viable acquisition option in the film industry and they were also the first to use the RAW image format.
When I started taking photographs with a DSLR back in the early 2000's I was terrified of changing any of the settings, it remained in full auto mode for the first 9 months of me using it. In my head I thought the camera might spontaneously combust at the turn of a dial. I had convinced myself auto was all I needed and that there was nothing in the settings that could possibly give me a better result. Wow, how wrong was I. Feeling brave I delved into the menu and found a tab for image quality with options for JPEG and RAW. I had used JPEG’s and TIFF files in other areas of my work before but I had never heard of, let alone used RAW files.
Setting the camera to record both RAW and JPEG, I furiously pressed the shutter button firing of test shots of anything which strayed in front of the lens. I was desperate to find out what was special about RAW. Upon opening the files in a photo editor and A/B’ing the images I found it difficult to see a difference. Apart from being much bigger in file size and somewhat less contrasty, I couldn’t see much of a difference at all. I decided to peer deeper into the images and played around with the exposure and that’s when I had that light bulb moment. Viewing the images close up I noticed a marked increase in per pixel sharpness along with vastly more detail when recovering clipped highlights and lifting under exposed shadows. We can see the differences between RAW and JPEG in the examples below…
So what is a RAW file? Without getting too technical a RAW file contains all of the data directly from the sensor without any undesirable processing applied. Sometimes it is referred to as a digital negative, like film negative it’s not meant to be displayed, it is meant to be processed. A JPEG is debayered, white balanced, sharpened, and has noise reduction applied along with being set into a specific colour space before it is stored on the memory card. RAW on the other hand has none of these things baked in and allows for maximum flexibility to make those kinds of decisions post capture.
When digital cine cameras first started moving away from using tapes the preference was to debayer and process the images in the camera, then record to to an industry standard file sequence like .dpx to a bulky hard disk recorder.
Once the record button is pressed all exposure, white balance and colourspace decisions were baked into the files with limited scope to change them after. Ironically these DPX files took up more space than RAW file equivalent on disk, but they fit the post production pipeline at the time. We had all been used to working with .dpx film scans for years in VFX and post production. It just made good sense to keep with this well established workflow. The problem came with the push for 4k.
Whilst it was easy to scan a film neg at 4k onto a static workstation with a disk array, it was extremely difficult to capture a 4K .dpx image sequence to a portable external recorder on a cine camera. The data rates and storage would be astronomical, even more so when slow motion is involved. Compression offered a solution to keeping the file sizes at a manageable level on consumer cameras but didn’t offer the quality required of digital cinema and high end television. For recording 4k on cine cameras there was a better way. A way which kept data rates down and maintained image quality, whilst giving more flexibility over traditional file sequences. The push towards capturing RAW in cine cameras had begun.
I use bread as an analogy to describe the differences between capture formats.
RAW is the ingredients to make the bread along with the recipe on how to bake it. RAW provides us with the maximum flexibility to change ingredients to taste before we bake.
Image sequences are the mixed and risen dough ready to be baked. We can adjust cooking times and while it is possible to add new ingredients it’s difficult for us to remove them.
Broadcast codecs are like a baked sliced utility loaf. Not a lot in this can be changed because it’s already been mixed, baked and sliced.
Matchmoving and RAW
An advanced matchmoving program like PFTrack is able to handle RAW files from both the RED and ARRI cameras natively which is a major benefit. Quite often in the VFX pipeline .R3D or ARRIRAW files are debayered and trans-coded in a time consuming process before work is able to begin on the selected shots. This has the potential to effect the overall image fidelity, along with the striping out of useful metadata recorded with the footage stream. The resultant effect could be a loss of key information that would aid you in achieving a camera solution. The biggest risk in trans-coding is the potential for error, the biggest of these would be cropping the image. It’s too easy to sit there whilst processing rushes and think, that shot needs re-racking I will do it now, I know this because I've done it myself. This ultimately has a knock on effect in matchmoving because by cropping image you are modifying the field of view. However not all software packages support RAW and camera manufacturers provide tools to convert the files into image sequences so that they can still be used.
The real advantage of RAW is that decisions made on set for things like colorspace, gamma and white balance are set as metadata flags and can be altered with ease in post production, things that would otherwise be lost if working with baked transcoded footage. If a different gamma setting has been used half way through a shoot then this will be picked that up and displayed automatically meaning what you are seeing is what everyone else will be seeing in the RAW pipeline. Or if an incorrect gamma setting has been selected on set then easily set it back to the desired output rather than going through the whole process of having the shot resupplied, saving huge amounts of time. Additionally when matchmoving, gamma and colourspace can be easily adjusted to reveal more trackable features in the scene. Debayering settings can also be adjusted, meaning the handling 8k footage on your laptop is not a problem, simply set the debayering resolution lower and still achieve the results that you need. In my previous article I spoke about the limitations of cameras that can effect matchmoving. By using RAW we are maximising the cameras potential, and reducing some of the issues that can be encountered when working with transcoded footage.
Perhaps one of the most interesting aspects of RAW is the information that is embedded with it. So what kind of information is stored? Basic Information such as the camera model, frame rate, frame size, white balance along with colourspace and gamma flags. Additionally more advanced bits of data can be captured such as the focal length of the lens used, the aperture and the focus position are all recorded on a frame by frame basis. For matchmoving this is pure gold…
A big plus of having key information embedded into a digital RAW file is that it limits human error at multiple stages during the post production process. It additionally presents a time saving efficiency when setting up a large project with multiple VFX shots that require matchmoving, no longer is data required to be sourced and input individually, for example in PFTrack the required data presents itself exactly at the point you need it automatically.
More With RAW
While Red and Arri led the way with RAW, other manufactures are starting to catch up implementing RAW recording in some form or another on there newer professional cameras.
Smarter RAW processing in camera, higher resolution and colour depth, reduced file sizes and faster, cheaper recording media mean RAW will soon become commonplace and accessible to all levels of production as we push forward towards higher quality content and increasingly complex VFX.
Granted RAW does currently require more processing power and there is a steep learning curve with the workflow along with a lack of standardisation, but there is so much flexibility and other benefits that can only make our jobs easier in the long run.
Most importantly get out there, shoot some stuff and experiment with matchmoving…