The Challenges of VR Post-Production
Part I: High-end 360/VR Documentaries
“There have been a lot of improvements in VR post-production workflows over the last few years. Three years ago, stitching the low resolution proxies would take an overnight render. Nowadays, we can have them almost in real time due to faster graphic cards and better stitching algorithms.”
Cristian Dominguez Rein-Loring — Head of Capture at Voyagers.
The world of post-production in the virtual reality and 360 video industry is both one that encourages a lot of creativity, but also demands incredible patience and organization. While it is an extremely technical job, what really matters is how impactful and well crafted your story is. However, those of us in this field understand that the path towards having enough time to focus on the quality of our edit is often rife with technical and organizational challenges.
We wanted to break down some of the common challenges we face in the VR post-production world, not only to foster a conversation and see how other professionals combat these challenges, but also to offer our own basic suggestions and solutions.
Challenge #1: Chaotic File Management During Production
Often times — and I’m sure many people in the post-production side can agree — the chaos in post actually starts even before editors touch a project. This is why I think it is very important for other people on the team, especially producers and shooters, to have experience with post-production.
If files are poorly organized during production, that can be the worst nightmare for editors. So much time can be wasted trying to figure out and disentangle the mess captured during VR shoots — including but not limited to: multiple video clips per shot (depending on what camera you use), spatial audio files, interview audio files, graphics, rough/fine stitched footage, plate shots (stills taken from the spot where the camera was placed in order to facilitate the tripod removal process), and sometimes even linear footage and photos that will be incorporated into your project.
Solution #1: Organize and Use SOPs
Create a neat and tidy folder structure that can be templated and easily followed by the crew throughout production, from guiding where to dump (and how to name) the files in your external drive, to where to place the final exports of your piece. Also, it’s essential that your crew create a shot list specifying what is happening in each shot, what audio files recorded separately should match with each of these shots and what the related plate shots are.
As put by Dirk Wallace, who is a Director, Executive Producer and former Head of 360/VR Post-Production at Huffpost RYOT, Standard Operating Procedures (SOPs) are incredibly helpful. “I’m a big fan of having SOPs for shooting, dumping footage, backing up, naming files etc. Included in this would be the creation of your Post-Bible, which goes over what the final deliverables are — are we shooting in 6k or 8k? Mono or Stereo? — basic things like that. Everyone needs to be on the same page, all the time.”
Challenge #2: Software Crashes
That moment when your project crashes on Premiere Pro is always a heart-stopping moment. With editing VR projects, you’re likely dealing with very heavy material, depending on what resolution the footage was filmed in (6K? 4K?), how many immersive VR effects you apply to your shots, how many 2D elements you add, etc. Not only can this greatly slow down your machine and workflow, it can also cause Adobe Premiere Pro to crash.
Solution #2: Save, save, save
A lesson that I learned especially while editing 360 documentaries (which can also be applied to any type of content you edit in Adobe Premiere Pro), is that you should always keep a copy of sequences with different versions of the cuts, and once in a while also save your project as a new version. Depending on how many different versions of the same documentary you may need to export (Facebook version, headset version, exports in different languages, etc), try not to forget to create a copy of your final sequence before you make any changes.
Challenge #3: Running Out of Space
Data management is one of the biggest challenges we face in high-end 360/VR productions. The higher the quality of your project, the larger the files will be and the more space you will need to reserve to store it. As you begin editing, and creating rough stitched files and fine stitched files, you might find that you run out of space on your hard drive. If you end up using two hard drives simultaneously to store your project, things can become tricky and messy.
Solution #3: Plan Ahead and Back Up
Make sure you start your project on a hard drive that has enough room for the entire project to be finalized. At Contrast, we usually use one master 4TB hard drive and another 4TB as a back-up.
The amount of space your project will roughly take up will depend on what camera you are using. For example, when using the GoPro Omni camera, the final project folder of our documentaries can go from 2 to 4 TB, including raw footage (a few gigs per shot), rough stitched files (usually under 1 GB per shot, if rendered in low resolution), fine stitched files (6K ProRes MOV files could go from a few GB to up to 70 GB — if rendered after tripod removal and with effects such as denoiser and stabilizer applied, for example). Also, take into consideration if you are storing any animations that are going to be applied to your documentary. More complex animations could take several gigabytes of storage.
Once your project is done, you can use the same drives to store smaller projects and archive the master folder using a cloud storage service to clean up space — if you don’t have access to a server, that is a good option. But during the post, don’t forget to backup your project every once in a while. You never know what can happen with the master drive you are using. For more information on data management, please read our post An Intro to VR Post Production, Part I.
Challenge #4: Creating the Best User Experience Without Sacrificing Your Sleep
As said by Nikita Bohdanov, a 360 video producer at New Cave Media, “The primary challenge we face in post-production is a compromise between high video quality and convenient post-production.”
Post production steps for virtual reality and 360 footage — such as stitching and stabilizing shots — can be extremely time consuming, as it always feels like there is room for improvement. If you’re good at paying attention to the details, you’ll always find something that can be perfected if there was more time. Or, depending on how the video is shot, the process of stitching and stabilizing can be extremely tedious in order to get it to the quality needed to make the documentary.
Nikita says that the challenge is to find the compromise “between spending extra working hours and delivering a comfortable user experience to a viewer. Stitching the footage from narrow spaces and stabilization can sometimes go too far and drive the whole post-production department crazy.”
“A lot of progress has been made, both in custom camera stitching software and general purpose software, but if your original camera move is too jerky, the softwares won’t work miracles because there will be too much distortion in the spherical 360 image.“ — Nathalie Mathe, VR Creator and Post Production Supervisor
Solution #4: Drawing the line
At some point, you have to draw the line. Make sure you build a realistic timetable with all of the necessary steps of post production, including stitching, stabilizing, color grading, audio mixing, tripod removal and more. Understand the limitations of time and cost. If you are faced with a particularly difficult problem with your footage, take a strategic look and gauge whether it is really worth spending your time on.
“Practically speaking, finding the fine line between negligence and overkill — the moment video is good enough — becomes key,” says Nikita.
Challenge #5: Pray not to find any glitches in your exports
We have faced a lot of issues with glitches in our final high quality exports. We might wait two to four hours for a final project to export, only to find the export unusable from various glitches. It make even take a few days for our computers to produce a usable export.
We figured a few tricks to solve the issue, in case you go through something similar, but we are still investigating and trying to reach the Adobe team to find out the secret to prevent this from happening. (If you have a suggestion/solution that is not being mentioned here, please feel free to comment below and share your experience with the community).
Solution #5: Clean, Switch or Upgrade Your Machine
One of the main reasons why this can happen is having an old or worn out graphics card — at least in Apple machines. The examples above look more like artwork, but they are actually how the videos came out after my attempt to queue them up overnight in Media Encoder. First of all, remember to clean your Premiere Pro disk cache before you export the video. The disk cache stores all your temporary files and ends up slowing down your machine and occupying space from your computer.
In some cases, rendering the timeline helps get rid of glitches. However, in other cases, if you have an adjustment layer for color correction or .png files with immersive VR effects applied, rendering can actually cause the glitches.
Another tip is to try to avoid using lower resolution shots in a larger sequence by “setting them to frame size”. That can cause your export to present a line on the back of the shot (where the footage meets on the edges of the flat image). If you absolutely need to use the lower resolution shot, convert it to a bigger size using a software like iFFmpeg, for example.
In some situations none of these tricks really work, so the best way to go is to start a completely new project and import only the final sequence that is ready to be exported. Or, try out another machine, if available. If possible, it could be an option to have a machine dedicated to exporting your projects — and reformat this machine once in a while.
As mentioned by Dirk, “get the best hardware you can afford. Time is money, and hours spent rendering, copying or stitching are hours wasted. If you can afford to upgrade, do it.” Having a machine with proper processor and graphics card can save you a lot of time in post and also save you from a lot of headaches — like the glitches we mentioned previously.
Alan Bucaria, an experienced virtual reality DP, recommends getting a machine with a minimum of “10 cores CPU and a 1080ti GPU 64GB RAM minimum for processing and editing 4K stereo footage.” He cautions, “but really, the more the better.”
Ultimately, be patient and encourage yourself to talk to other post-production folks and watching as many troubleshoot tutorials as possible. If you’re new to post production in VR, you can also check out our Intro to VR Post Production, or read part two to learn about editing social videos, or part three to learn about the final touches in post production.
If there are challenges you run into, or solutions you have found, please comment below and share your thoughts.
