Exploring Future Reality
Road Map for Experimentation
1. What editorial content makes sense? Put the viewer in the middle of the action and give them something to look at.
In many cases, inaugural virtual reality productions are collaborations between hardware companies and advertisers. For media companies, the smaller scope of an advertising project means less cost and less risk. For brands, it’s an opportunity to garner media coverage.
Rather than supplanting an older medium, VR is a new way of thinking about story, spatiality, and character exploration.
“VR can be utilized for introducing the user to all manner of subjects and stories,” said Samantha Storr, executive producer at Vrse. “From you-had-to-be-there moments to humanitarian crises, our collaborations with the UN — Clouds Over Sidra and Waves of Grace — have proved how powerful VR can be in connecting world leaders to the major issues they preside over. Rather than supplanting an older medium, VR is a new way of thinking about story, spatiality, and character exploration. There are documentary applications, narrative adventure applications, and everything in between.”
For Discovery VR, there were two important factors in selecting content. The first was to work with the production teams that were also starting to experiment with virtual reality independently. In this case, the “Mythbusters” production team was already excited about virtual reality. The second consideration was the content itself. Conal Byrne, senior vice president of digital media at Discovery Communications, noted that the challenge was in “trying to find intellectual property where the location was as big a character as the talent. We thought that would be a good place to start for virtual reality.” Discovery VR created content for “Gold Rush,” set in Yukon, Alaska and “Survivorman,” which is filmed in remote locations all over the world.
2. Video or computer graphics? Try it all.
Media companies well equipped to create video content are poised to make a smooth transition into producing 360 video for head-mounted displays. However, there are some aspects of virtual reality that can’t be tapped into with video. “360 video limits the mobility of the viewers to the point of view of the rig, even in movement on a predetermined path,” said Adrian Saravia, digital creative director at Fusion. “Computer graphics environments allow the possibility to move at will and in any direction which again, depending on the experience, can only be achieved with Unreal or Unity engines.”
Although producing interactive content with game engine software requires the greatest number of new skills for media companies, it’s worth the investment since these platforms — the Oculus, Vive and Morpheus — will use interfaces and tracking that present the interactivity that virtual reality has to offer.
3. Pre-Production and Pre-Visualization
“Great VR is immersive, first and foremost,” said Samantha Storr, executive producer at Vrse. “You have to feel like you’re within a new space. That comes down to a lot of things: art direction, story, character and environment interaction, and a robust editing process. The full 360º environment should look seamless, and the depth and dimensions of this virtual world should cohere with the real one.”
The shot unfolds as scenes such that stage direction, not editing, leads the attention of the audience.
Many producers have found that virtual reality production is somewhat like producing for theater. The shot unfolds as scenes such that stage direction, not editing, leads the attention of the audience.
The time-intensiveness of rendering, syncing and stitching virtual reality footage is an added incentive to get the footage needed in only a few takes. To address this issue, Stone Roberts at Refinery29 had the talent he was working with rehearse a scene many times before filming.
4. Setting up your scene: Production Techniques and Challenges
The whole location should be considered when capturing 360 assets. “When you’re shooting in a 3-D space, you can’t crop anything out. You’re really capturing everything,” says Matt Bell, co-founder of Matterport. “When you go in to take a photograph, you can frame exactly what you want in the photograph and outside the photograph.” Anything in the shot that can’t be cleaned up on location can be tackled in post-production, usually taking a greater amount of time than addressing it on set.
Instead of having your director, producers, grips, and electrical hiding directly behind camera, a VR crew has to hide in another room or behind a wall to be out of shot.
One major challenge is keeping lighting even as the space is captured. The lighting needs to remain somewhat consistent and not have too much contrast between light and dark in order for the 360 image of the space to appear consistent on screen.
“VR production typically consists of a smaller crew,” noted Storr. “Instead of having your director, producers, grips, and electrical hiding directly behind camera, a VR crew has to hide in another room or behind a wall to be out of shot. Tools like boom poles and lighting equipment can’t be in the shot either, so VR production typically utilizes a handful of people that can multi-task to make sure the shot is set, which cuts down on production costs.”
For cameras that capture motion, early experiments have shown that it is best for the camera to remain still or move at a very slow and steady pace. Vice had success moving the camera in its piece documenting a protest in New York City. Similarly, the North Face virtual reality experiences have shots that involve a 360 camera gradually moving while suspended from a drone.
Subjects should be at least two feet away from the camera during capture because of the wide angle of the lenses and the overlap of the images needed for stitching. When capturing stereoscopic video, the subjects need to be twice that distance, at least four feet, or even further from the camera to appear undistorted to the viewer on playback.
5. Production Costs
The production costs for virtual reality vary considerably depending on the method used to make the content and the creative direction behind the shoot. However, there are some trends emerging.
For captured content, many producers have compared the cost to a high-quality 16x9 shoot. The producers we spoke to for this report estimated a cost-per-minute of virtual reality production of anywhere from $8,000 to $100,000. The more computer generated imagery, the higher the costs tend to be. The price to create a virtual reality experience with graphics as high-quality as those in an animated Hollywood feature film could cost well above $100,000 for a minute of footage. There are opportunities to save money by reusing computer graphic assets between productions. For example, NBCUniversal took an animated model of a dinosaur from “Jurassic World” and used it to create a virtual reality experience for the Samsung Gear VR.
Many 360 video projects use only one camera rig and a few takes. This reduces the budget for camera operators. However, the degree of pre-planning for a scene means that there needs to be more time and money put in the budget for working with the talent ahead of time in rehearsal. Preparing a set for 360 video capture can require many resources because it must be lit evenly across every angle.
6. Syncing
Once the footage has been filmed, the first step in post-production is to sync the footage from the various cameras. A signal captured on each camera is used as a marker to match video clips to each other. There are three common methods of synchronization: flash synchronization, which is often the most accurate, sound synchronization, and motion synchronization.
Advances in 360 camera rigs are coming soon that will allow the camera start and stop time to be locked in the camera. Until then, syncing will continue to be a trying process. “One of the most painful things today is synchronization,” said Nicolas Burtey, CEO of virtual reality post-production software company VideoStitch. “It’s something we can do by algorithm, but if it’s done directly in the camera it’s much easier to handle.” Once the video is in post-production, stereoscopic video takes two to three times longer to sync, stitch and render than monoscopic video.
7. Narrative is essential in virtual reality and requires a new approach
To keep a viewer interested in a virtual reality content, the viewer needs to be hooked into the story and needs to know where to look within the 360 degrees to follow the next phase of the story.
Fergus Pitt, Senior Fellow at the Tow Center for Digital Journalism at Columbia Journalism School, found that 360 video without narrative can only hold someone’s attention for a short amount of time. “Narrative remains important,” said Pitt. “Humans respond to narrative in a way that they don’t respond to other types of things.”
Samsung and Refinery29 collaborated on a project called “Fashionably Bound”, which captured fashionable people from cities around the world as they shared their favorite local places. This was the first virtual reality piece that Refinery29 produced and the team found that leading the audience through a narrative required a paradigm shift from 16x9 video production. “Virtual reality storytelling as a device is very different,“ said Stone Roberts, VP of programming at Refinery29. “It’s like going from being a sculptor to being an oil painter. They are very different mediums in how the story evolves.”
One of the challenges for captured content is that figures cannot come within 6–8 feet of the camera. This puts characters at a distance and makes it harder to get the audience drawn into the moment. “You have to construct a scene and work with your on-camera talent to interact with this, 95 percent of the time, immobile camera,” said Roberts.
Traditional camera angles and film grammar don’t translate to virtual reality. Roberts thinks of every turn of the head as a cut in the film. “You’re really forced to direct with carrots every 15–20 seconds within each story,” said Roberts. “That allows the user to hang on. You need to bait them to be the editor and investigate in new ways.”
Audio is very important to delivering the experience. In the case of the Refinery29 experience, audio is used to tell people where to move their head. “Using sound to incentivize the viewer to turn and interact is the most effective method of storytelling within this medium,” said Roberts.
Dan Archer, virtual reality producer and founder of Empathetic Media, has found that the amount of guidance given to the viewer needs to be tweaked for each experience. “The beauty of each individual story is that you can tailor it. That learning curve has been really interesting for me,” said Archer. “How much leeway do you want to give the user? In the first Ferguson build for example, we didn’t include breadcrumbs along the street and people were kind of paralyzed by the amount of freedom they had. It’s about finding a sweet spot.”
8. Distribution: platforms and apps
“The big challenge right now is the distribution part of it,” said Adrian Saravia, a digital creative director at Fusion who is leading virtual reality projects. “The fact that it’s so fragmented and each platform or device has its own process and marketplace. Even for us opening accounts in the Oculus space and the Samsung marketplace, that in itself has been interesting to see all of the caveats. The low-hanging fruit for us, as a TV network and such a video-centric digital platform, is video: YouTube,” said Saravia. “The fact that it has the most commercially available experience makes it a lot easier.”
Producers can also create their own applications to host their virtual reality content. An independent application allows companies to maintain more control over how people find and playback their content. Software engineers for these applications will have to contend with the demands of virtual reality on device hardware, bandwidth and file size. “The upper limit of bandwidth and speed is always being pushed by service providers,” said Gavin Teo, principal at Comcast Ventures. Teo noted that there are limits to the CPU and GPU of mobile devices, but there are ways to make great work within those limitations. “Those minimum restraints, you can solve for them with this current generation of hardware if you are a thoughtful and experienced content creator.”
9. Measure your results
“For traditional video you look a lot at time-spent on the video, unique views or shares. In virtual reality, maybe the metric will be repeat viewings of a single clip,” said Conal Byrne, senior vice president of digital media at Discovery Communications. “You feel like you have missed the other 180° or 270° degrees of the content so you go back and watch it again.”
10. Talent and workforce development
Most of the skills of video producers will translate to virtual reality production. The medium is so new that there aren’t many experts. However, given that there is so much to be discovered about how to best use virtual reality, there is a lot at stake in hiring producers who know how to test the bounds and find the best use cases for a new medium.
“Right now, this is the birth of a new medium… When you have a shot at being the next Pixar, why work for somebody else?”
Media companies that haven’t produced animated content before may have to create new positions in order to produce some types of animated content. These animators need knowledge of 3-D modelling and experience developing content for the 3-D gaming engines Unity or Unreal. “If you want to do anything serious, you have to think about the same sort of skill set as an independent game firm,” said Roger Kenny, senior engineer at Dow Jones.
Some companies have been training their programmers to work with game development engines instead of hiring new employees with game development skills. “There are lots of conceptual leaps when you go from browser based to 3-D, but we’ve started to make that shift,” said Ray Velez, chief technology officer of Razorfish.
The opportunity to start a groundbreaking company in the new space of virtual reality has attracted visual effects artists. For many 3-D designers the choice is between taking a job and starting their own companies. “There is a talent war happening right now for the good animators and the good visual effects guys,” said Karl Krantz, who runs the Silicon Valley Virtual Reality meetup. “Right now, this is the birth of a new medium… When you have a shot at being the next Pixar, why work for somebody else?” For the time being, the supply is limited and demand is growing quickly as a wider group of firms want to produce 3-D content.
Visual effects and animation talent is mainly concentrated in the Los Angeles area. Jaunt opened their content studio in Los Angeles, Oculus has an office in Irvine, CA, and NextVR is located in Laguna Beach, CA.
Dakota Powell started the Virtual Reality Lab New York after moving back to New York from London. Powell has found that New York has a lot to offer virtual reality producers. “I think that we get different strains of virtual reality going,” said Powell. “Because it’s not a one industry town. All of the different industries that are looking to explore virtual reality are doing it here: non-fiction, advertising, all that. This is a great place to be.”
There is a steady supply of new 3-D animation talent coming out of New York City’s art schools and, in some cases, this talent can be found overseas as well.
11. Partnerships in Innovation
Companies have to decide between spending money to create the capacity for making virtual and augmented reality content in-house and working with outside production companies and hardware companies. “There’s innovation that comes from within, which is reinventing your internal process and reinventing the business,” said Francesco Marconi, corporate strategy analyst at the Associated Press. “We’re doing that, but we’re also taking a different perspective, which is to innovate through partnerships and collaborations.”
The majority of virtual and augmented reality hardware and software is still in the testing phase. The only way to start using it and learning about it is to form partnerships. “Our approach when we partner or when we invest in these organizations is not necessarily financially driven, we’re not a venture capitalist,” noted Marconi. “We identify companies or technologies that can augment or improve the Associated Press’ current capabilities. Our perspective is that the Associated Press, as a major supplier of news, should take a leadership position in identifying these new technologies that can later be used by our members, clients and partners.”
In November 2015 The New York Times announced a partnership with Google to deliver a million home delivery subscribers free Cardboard VR viewers. Subscribers were then able to access virtual reality content and immersive video from the Times via the NYT VR app.
The first film released by The Times is “The Displaced”, a documentary following the stories of children displaced by war in South Sudan, eastern Ukraine and Syria.
16x9 vs VR
“Production alone is typically less costly, depending on the shoot, than 16x9,” said Storr. “The heavy costs of VR arise from camera equipment. Shoots can require anywhere from 8 to 32 cameras or more, including corresponding lenses. Rigging is often tailored specifically to the shoot at hand. Post-production often swallows more than half of the budget.”
Many producers expect that costs will come in line with 16x9 production soon. “It may not be one-to-one in regards to a 16x9 production world, but it’s pretty close. If it’s not there yet, within the next six months to a year, it should be,” said Stone Roberts, producer at Refinery29. “It’s a different approach than you would take with a traditional production, but there’s not much additional you have to deal with.”
The Promise of Augmented Reality
Most experiments in augmented reality still remain a few steps removed from mainstream, consumer-facing applications. But researchers such as Columbia University’s Steve Feiner are developing the fundamentals of augmented reality user interfaces and experiences. In doing so they are creating the underlying systems and software necessary for augmented reality that previously have not existed.
“A lot of that work emphasizes interaction techniques and visualization techniques,” noted Feiner. “At a high level the questions really are ‘how do we communicate information to people and between people working in 3-D environments?’ and ‘How do we visualize and communicate information effectively?’”
Feiner has addressed these questions in his lab at Columbia since the early 90s by prototyping augmented reality applications that assist people performing tasks. “One very common scenario is one in which you’ve got a remote expert advising a local technician.” said Feiner. In such an application a technician using augmented reality can see instructional material overlaid on an object in need of repair.
Exploring Future Reality is a report brought to you by NYC Media Lab. Download a PDF of the full report here.
Continue to part nine of this ten part series.
Return to the Exploring Future Reality report homepage.
