Going in Circles: What You Need to Know About Virtual Reality Production
When I was a kid and we went to the movies my dad would say, “Let’s sit towards the back. I want to watch the movie not be in it!” My how times have changed. Welcome to the world of immersive entertainment and welcome to ‘Live Action VR 101’.
Producing for VR is a virtual hack-a-thon right now but hopefully this primer will give you some insights into the production process, the technology and the workflow. Imagine if you will what filmmakers did when the first motion picture cameras came out. First, they learned the technology and then they brought artistry to the medium. This is exactly what is happening with VR. As with any new technology there is a “tail wagging the dog” tendency and when that happens the result is bad VR. As any great producer knows, the technology needs to serve the story.
The greatest strides in the art of what I prefer to call “immersive narrative” (also called cinematic VR) are being made by independent filmmakers who “hack” the technology — from cameras and lenses to sound and light- in order to create the story-driven experience they know will move the immersed viewer. Today’s hacking is tomorrow’s automation and these filmmakers are informing the evolution of VR production tech. As storytellers, we want to emotionally impact the viewer and even leave a lasting impression. Virtual Reality offers an unprecedented opportunity to do exactly that, sometimes in extraordinary ways. In what I consider to be the fastest moving industry ever, we are now transitioning from the wow of an immersive experience to the compelling content and artistry of narrative. Getting there involves experimentation, hacking and learning a new language of cinema that borrows from traditional film, video games and, believe it or not, theater. Consider this brief a snap shot of where production is today.
You will need a lot more moving up to professional high quality VR filmmaking. Right now it’s the Wild West. Something new seems to come out every day. Part of my job as a producer is staying on top of all of it!
Processes that took a lot of patience, time and resources just a year ago are now taken care of with software. Examples: A year ago, synching a 16 Camera GoPro array to shoot in 360 meant a single Wi-Fi remote, a handclap for sound and a shot sync. Timecode Systems just released SyncBac at this year’s NAB Show. It allows multicam sync via a time code generator that fits on the back of GoPro Silver and Black cameras. There’s a timesaver in post! Also at NAB Show, Deep, Inc. launched Liquid Cinema, which allows for “forced perspective” allowing the content creator to direct the audience’s attention inside the VR experience at any time as well as live rendering graphics and even some visual effects. These features are big leaps in VR production. Finally, when it comes to workflow and stitching, meaning putting the output from multiple cameras together into a spherical image, more tools are coming out all the time from companies competing to be the gold standard like Video Stitch, GoPro’s Kolor, Jaunt and Ozo.
THE STORY
Story first. There are great differences in approach for production in immersive journalism, live streaming, sports, documentary and animation. For our purposes, we are going to be considering narrative drama or comedy. Remember, you are working with an entirely different medium that actually can fool the brain and the body physically and emotionally into what is called “presence.” This places the viewer in the environment that the characters are experiencing, can allow the viewer to stand in the shoes of a specific character, or even change characters. Sight, sound and light as well as superb acting will allow you to create a first person experience. The “what” of good storytelling does not change. You still need a great story with the elements of character, time and place, canon and a story arc. It is the “how” that changes and to master the currently evolving how is like working with mercury.
The best advice: really learn the medium and the tools. A good rule of thumb is that if you can tell the story in traditional 2D film then that’s what you should do. If your story can only be told in virtual reality using and pushing the limits of the medium then in my humble opinion, you’re on the right track. You’re going to learn a lot but you are also going to shape a new and exciting new storytelling medium. It’s a green field right now with lots of room for experimentation. But, as far as consumers go, we should only release our best in VR.
Writing for a 360 environment is revolutionary. For example instead of just setting a descriptor for the movie scene in front of the viewer, or describing shots, you are setting the scene and action around the viewer. You are also blocking the camera and positioning the actors in relation to the camera and each other in the round, which figures prominently into your narrative and character relationships even when nothing is being said. Depending on the number of cameras being used you’ll have multiple “stitch lines.” These are areas where the images captured by the cameras overlap. As example, if your actors are too close to the camera and on the stitch line you may have an uneven match that splits their face right down the middle in a Picassoesque way. Not so easy to fix in post.
Personally, I write to a circular blocking form. It’s literally a pie chart with each segment representing a camera lens where scene descriptions, actions and dialogue are written in the inside of each segment. You are also deciding where and when to direct the viewer’s attention in a manner that serves the immersive story. The form even allows me to create transition shots easily understood by the director.
It’s my contention however, that the writer is part writer, part director, part cinematographer and part sound designer when scripting VR. It’s exceedingly helpful if you have a background in gaming or theater. Years ago while touring with “Fiddler on the Roof,” Theodore Bikel taught me how to “act with your back” for theater in the round. Who knew that would come in so handy? As you might guess, some new professions are emerging including immersive sound designers and directors of virtual photography (DVP) as well as specializations in stitching and editing.
THE DIFFERENCE BETWEEN 360 AND VR
On the consumer distribution side, you’ll hear these terms used together or interchangeably but many purists in the industry will tell you they are not the same. While most VR is 360 not all 360 is VR. Think of 360 as something at arms length and VR as something in which the audience is immersed. Thanks to relatively inexpensive consumer cameras that shoot in 360, and the world of advertising, 360-video is surging. You can see it on Facebook, YouTube, VRideo and Littlstar and thanks to Google’s “Magic Window” technology released in the spring, anyone can iframe, i.e. embed 360 video into any web or mobile site. On desktop you use your mouse to navigate to reposition the spherical video and on mobile you use your finger to slide around the scene or you turn your body with your phone. You’ll see a lot more 360 commercials coming.
Some say 360 is the gateway to immersive experiences. But for VR you need some sort of Head Mounted Display (HMD). Even Samsung has acknowledged this migration path and now offers 360 “Samsung VR” content in a mobile app for specific Android phones compatible with their GearVR HMD. If you tap the on screen headset icon you are prompted to place your phone in the GearVR and voila!, you are in an immersive experience. With other 360 mobile vids you tap the Google Cardboard icon and drop the phone into your Google Cardboard or other stereoscopic viewer and the image splits for stereoscopic view. You can imagine how confusing this is getting for the consumer who thinks they are watching VR when they are not.
The “immersion continuum” illustrates the degrees of immersion from 360 to consumer HMD’s right on up to the fully immersive and interactive Oculus Rift, HTC Vive and Sony PlayStation VR and ultimately 3D/4D dark rides and VR domes. You can of course create content across the entire continuum or even scale what you create for one to work on all but it’s best to decide what your anchor platform will be. Here we will be talking about Virtual Reality production for HMD’s.
FORMAT
You have choices as to what shooting format you will use.
- 360 2D. This is exactly what it says. An image that surrounds, well that is really spherical but in 2D. Nothing leaps off the screen or beckons you to interact. This is not truly cinematic VR. It is 360 video in a VR HMD. Your viewer is immersed and can see all around and depending on camera rig can see in detail. This is the least expensive option depending on the cameras you choose. More on that when we get to camera rigs which by the way can contain as many as 24 cameras.
- 360 Stereoscopic VR. This is more how our eyes see, is in 3D and closer to reality. For every one camera that you would have in 2D you will now have two cameras. This creates greater presence and has more emotional impact.
- 360 Algorithmic 3D. Advanced computational photography is used here and creates superior VR and is as close to reality as we can get right now except for touch and smell.
PRODUCTION
Now that you have your story, it’s time to look at the best way to film it. Flexibility is in order as is research. Choosing the right camera and the right rig is as much of an art as the filming itself.
We do a lot of camera tests to check performance in relation to what we want to accomplish. In a recent production meeting on a new project, the director said “Wouldn’t it be great if we could do this?” He was ready to hack a camera and lenses but we also knew of a camera in development. Days later we talked to the company and we’ll have that camera by July. My motto however is “if you can’t find it, hack it!”
CAMERA RIGS
Your choice of rig depends on many factors including the format as described above, the story you want to tell, how you want to tell it and your budget. Some vendors offer end-to-end solutions including image stitching and editing. The cost range can go from say $2,000 up to S60,000+. There are also of course consumer cameras such as the Ricoh Theta, Samsung Gear 360 and forthcoming LG 360 Cam that shoot in 2D with onboard stitching for under $500. Some basic things you need to know if you’re up to shooting higher quality VR. Define your criteria for camera selection.
- Choose the camera(s) that best serves your story and the environment in which you are shooting. Decide how many cameras you want to use based on desired coverage, final resolution and of course budget.
- You’ll be shooting “oners” or long takes. While there is “framing” of shots in VR it is not shot framing in the traditional sense. Length of scene, lighting, environmental conditions- all matter.
- You will have to use natural light, sometimes low light or light from the outside in in a studio setting. You cannot light like you would a traditional movie. (Nor, can you have crew standing about- everyone has to hide!)
- You’ll need to shoot in at least 4K and at least 60fps for top quality VR.
- You’ll don’t always need to shoot in 360 sometimes 180 will serve your story.
- The lenses you choose will determine your field of view and coverage (as will the number of cameras).
- Cameras must be synched with each other and all settings must be the same. You cannot mix and match cameras in a rig.
- Battery power and cooling can be an issue. There are tech peripherals that can help you solve this.
- The higher the resolution the more data which means bigger files.
- The more cameras the more data. Monitoring your shot isn’t easy. Some cameras like the OZO have real-time monitoring. Many filmmakers are hacking solutions together using consumer VR cameras on top of the more expensive rigs. Recently the Orah4i camera launched for live 360 streaming and filmmakers are actually using it in addition to their camera rigs for previz and monitoring. Incidentally, previz is highly recommended.
IMMERSIVE SOUND
Video and image get a lot of attention in VR but sound is just as important to get right. Most onboard camera microphones will not allow you the flexibility that you’ll need to create object oriented spatial audio that captures a realistic sound field. This is where Ambisonic microphones and field set microphones come in.
Ambisonic mics record a 360 sound field that can then be decoded into 7.1, 5.1 or the binaural sound used in VR experiences. This allows the viewer to be immersed in sound as well as visual circumstances. Some VR sound designers are placing microphones not just on actors but also around the area of the sound filed and then editing into a 360 immersive environment. Startup Two Big Ears recently acquired by Facebook, has launched 3Dception for both cinematic VR and Games that is a well-reviewed end-to-end solution for spatial audio. Having an exellent 3D/VR sound designer on board is a wise move.
FROM SHOOT TO DISTRIBUTION
When producing for film you know what your delivery format will be and you leave transcoding to the distribution folks. But as mentioned before, you’ll need to take that immersion continuum into consideration. Depending upon what cameras you have decided to use your tasks may change, as may aspects of your workflow. Preparation will make all the difference and save you a lot of time in post. For our purposes let’s just say you are using a GoPro Odyssey 16 camera rig. Briefly, to prepare, you will need to number all your cameras as well as numbering the SD Cards that you will be using. This is so you can keep track of the cameras and the shots when you are stitching them together in a spherical image. With this rig you probably won’t want to run on the in-camera batteries but rather get a large battery to power all of them. And of course, use the new SyncBac product for synchronization. You’ll need storage in the field, at least 2 TB(terabytes) Think of the math. You have 16 GoPro’s each with a 64gigabyte SD card. That’s about 2 hours of high-resolution (1080/4K) high frame rate (120fps/30fps).
REHEARSE. REHEARSE. REHEARSE.
This is just not rehearsal for your actors, which is vitally important considering that they will be in a long take, but also that acting for VR is different. It’s whole body and wholly authentic. A bit like acting in theater in the round but intensely intimate. More than a few potentially great VR pieces have been destroyed by bad acting. You and your crew including camera, lighting and sound must also be well rehearsed. I suggest using a consumer 360 camera or the new Orah 4i during rehearsal so that you can use playback to work with the crew and the actors. Where you place the camera in relation to your actors is important especially if you want to maintain the correct perspective. If your viewer is looking directly into the eyes of your character you don’t want the character’s head to be twice as small as the viewer’s.
Contrary to popular belief you don’t just plunk a 360 camera down and act around it. A year ago no one wanted the camera to move; now there’s camera blocking, transition shots and moving cameras. There are single POV stories, multiple POV and moving from 20 into 30 stereoscopic. The art form is developing. Having witnessed brilliant independent VR filmmaking, I’d say we are only limited by our imaginations, not the technology.
FILE MANAGEMENT
After the shoot, it’s time to get all your ducks i.e. SD cards in a row. Fortunately you have mapped the cameras and numbered them as well as the SD cards so you know the shots and where they came from and now with SyncBac they are timecoded. And, fortunately there is some great software to help with file management. CamMan from 360Heros has a great intuitive interface and there is also ManyCam, which is a bit simpler. This may save you a bunch of renaming and reorganizing files but you still need to be vigilant and attentive to make sure you get it all right.
STITCHING
Now that your files are set you can move on to stitching the shots into a spherical video. Some consumer and professional cameras have “on-board” stitching. As a matter of fact, if you are using the GoPro Odyssey you can use it in combination with Google Jump, which automates the workflow. But, if you want to be more hands on you can use GoPro’s Kolor or VideoStitch. Now it’s time to export into editing.
EDITING
Once you’re all stitched up you’re ready to edit! There are choices here too. You want to be able to output to Spherical or Cubic Monoscopic (for the web and mobile 360) and Stereoscopic VR for specific or all HMD’s. Your choice of course depends on your distribution and with these software programs you can specify the output device. There are choices here too. One of them is Mettle’s Skybox, which has a nice suite of tools for 360/VR editing for Adobe Premier Pro. Think After Effects for VR- you can tilt, pan and roll the sphere and edit to have an even horizon line, direct the scene start field of vision, correct the stitch fine seams and much more, all key frame editable. Maxon’s Cinema 4D also has a great VR Workflow that works with Adobe Premier and After Effects. You can learn more about Maxon’s Cinema 40 workflow at cineversity.com. Finally, Dashwood’s 360VR Toolbox allows for in HMD VR editing and monitoring as a plug-in suite for Adobe Premier Pro, After Effects and Final Cut Pro.
SOUND EDITING AND MIXING
We spoke earlier about Two Big Ears end-to-end spatial audio solution 3Dception but you can also visit Ambisonic.net to learn more about the approach to spatial audio.
GETTING STARTED
This was intended as a fast overview rather than a dive into the technical weeds. But if you want to get started in the basic ABC’s of VR, my recommendation is 360heros.com. Michael Kintner, CEO and Founder of 360Heros has made getting started in VR easy with a ton of guidance and products. The “VR Starter Kit” will have you off and running and you will be joining a community of VR enthusiasts and professionals. 360Heros offers workflow solutions and hits every point on the content value chain even offering storage and distribution for your work. Working at the basic level you’ll gain a greater understanding of filming, editing and producing in VR. If you’re in Los Angeles you can even attend VR classes at Village Workspaces in West L.A. More cities are coming. Live action VR production is both challenging and rewarding. Best practices are emerging. Courses are being taught in universities. Hardware and software solutions are empowering filmmakers to be more creative.
My advice: get inside an HMD (GearVR, HTC Vive, Oculus Rift and soon Sony Playstation VR), and watch as much content as you can as any film student would. Know what can be done and then hack and push the limits.
Going In Circles: Virtual Reality Production was first published in the July 2016 issue of “Produced By” the magazine of the Producers Guild of America. It appears in CinematicVR courtesy of the author.
