How Can Journalists use Virtual Reality to tell News Stories?

Screenshot from Ferguson Firsthand, a virtual reconstruction of the scene at the Canfield Green apartments complx where Michael Brown was fatally shot.

Introduction: What is VR? How does it work? Is it the same as 360 video?
Part 1: World Building — creating character and environment assets
Part 2: The Game Engine — coding in interactions and exporting the piece


Introduction

Virtual reality experiences are disrupting the way we tell stories, but the dark art of how to create your own remains a mystery to most people outside of the video game or animation industries, especially those who would prefer to do so on a tight turnaround and conservative budget. This is a simplified guide for building an immersive story experience that could be applied to recreating a news event in virtual reality (without using 360 video). It’s meant more as a primer than a definitive, comprehensive guide — but should be enough to sate those who can’t tell their Oculus from their elbow, while providing a few tidbits to those who already can.

First things first, what is virtual reality?
A computer-generated environment that tricks the user into thinking they’re somewhere they’re not. This is largely experienced one of two ways: either through a head-mounted display (HMD) like Facebook’s Oculus, HTC’s Vive, Sony’s Project Morpheus (all of which require a desktop with enough GPU/graphics card muscle to power the experience); or via a powerful enough smartphone using Google Cardboard or custom-built hardware like Samsung’s Gear VR.

How does it work?
First, The image/video is split into two separate feeds, one for each eye. The lenses inside your VR viewing device of choice will distort the image to make it feel all encompassing. The aforementioned Google Cardboard is quite literally a piece of cunningly-designed cardboard that, when folded correctly (airfix modelling fans rejoice!), houses two lenses that your smartphone can then slot into.

Not all VR requires state-of-the-art technology: Google Cardboard, in all of its sticky-taped glory.

Secondly, the feed you’re seeing needs to correlate with your head movements (either through a motion tracker on top of your computer monitor, or using your smartphone’s built in gyroscope), which updates the video or image you’re looking at to create VR’s secret sauce: a sense of presence. Though we found while user testing our Ferguson Firsthand app that a lot of folks preferred using the single screen option and keeping their device at arm’s length while exploring the virtual space.

The white circle or dot has become the standard controller-free user interface, shrinking when it lands on trigger content in the scene.

Is that the same as 360 video?
Sort of, but with an important caveat. Admittedly, 360 video allows you to feel like you’re inside a video recording, since you can move your head around and turn your attention to more than just one single focal point. However, you’re still anchored to one spot (where the camera was set up) and the video plays along a linear, chronological timeline. So it’s kind of like becoming a CCTV camera. You tube have now incorporated spherical videos into their player, though many videos still only have a single focal point, rendering the tech a little superfluous. Why would you look at a tree behind you in this example below, you might ask, when there’s a guy wrestling with hyenas right in front of you?

So what’s the other option?
To follow the cue of video games and create a born-digital environment from 3D assets, based on photo references. Users can explore this environment and uncover different forms of media (audio, video -which could also be 360), images in whatever way or order they choose. They can also interact with elements, picking up certain objects, or examining them more closely, without being constrained by a linear, ever-ticking timeline.

How does that work?
It’s a two-part process: building the assets (the characters/buildings/vehicles/trees etc in the environement) and programming in the behaviours (so that when the user moves over one beacon, they are shown an image).

Won’t I have to learn a TON of software?
Actually, there’s never been a better (or more affordable) time to jump into this, especially with online video tutorial sites like DigitalTutors (3D focused), Lynda et al. But before you even consider wading into 3D territory, it’s far more important to bear in mind how the final experience will serve the story. What is the point of going down the VR route? Will it add a new dimension to the user experience?

Mapping the buildings on Canfield Drive to a satellite map in Google Sketchup Pro.

How can you be sure everything’s in the right place?
One distinct advantage is that this approach can be taken after reporters have left the scene, provided there is enough picture reference material to base their models on. Coupled with Google Sketchup/Earth (above), you can import satellite imagery to use as a base layer blueprint for your layout, ensuring maximum accuracy. Then you can break down what assets you have, and start organizing them by their point of origin in the scene.

Thumbnailing a hard copy layout of the scene was crucial, long before we started working on the digital version.

How you do create the character models?
We use an incredibly powerful WYSIWYG system called Fuse, made by Mixamo, which was recently acquired by Adobe. Fuse allows you to adjust the features, clothing and dimensions of your character before exporting them as a T-posed FBX file.

Working on the Darren Wilson model using photo reference as a guide.

An FBX file contains the raw form of the model you’ve designed, the textures that sit on its surface to create clothes/skin tones/details and any animations you’ve programmed in there too. Mixamo have also come up with an invaluable auto-rig script for Maya that lets you pose your FBX models using Maya’s move/rotate/scale controls.

What about the buildings and scenery?
Two options: the first, you build them yourself, using 3D modelling programs like Maya (made by Autodesk, currently offered at $123/month license) or Blender (which is free). Gimp is to Photoshop what Blender is to Maya. If it’s your first time modelling, the best way to get a feel for building 3D environments is by using Google Sketchup, which lets your construct shapes by pulling handles (you’ll see what I mean when you try it). To make sure they’re in the right place, you can spring for the Sketchup Pro license, which allows you to import satellite imagery from Google Earth into your canvas and build 3D assets on top, like a blueprint. Once you have those built (see the image in the chapter heading above) you can send that to a team of 3D artists who can produce a higher res version in Maya, 3Ds Max, Cinema 4D or their professional-grade software of choice.

View of the main scene from above, with notes and witness beacons marked

Secondly, you can use pre-made models from 3D asset stores like Turbosquid, CGTrader (which lets you barter with the creator, and reach out to them directly, if there’s an issue with the file format) or the Unity asset store. One thing to bear in mind before you purchase any models (aside from how good they look, naturally): the polygon count. 3D assets are comprised of hundreds of thousands of vertices, normals and polygons — and while the higher number of them, the higher level of detail your model will have, the higher the strain it will place on a computer that has to bring them to life in your user’s hands. Which can be a big problem during a VR experience, as any sort of processing lag will make a user feel queasy.

Speaking of queasiness, won’t VR make me nauseous?
Here the key word is latency. That’s when you turn your head faster than the image/video you’re looking at can update, resulting in a disconnect between what you’re feeling and what you’re seeing. It’s largely dependent on the hardware you’re running the VR experience on, though the efficiency of the code will also play a part. Here are some musings on the issue from John Carmack, Oculus CTO, co-founder of id games and programming legend who made the then-mindblowing graphics (we’re talking early 90s here) in games like Wolfenstein 3D and Doom.

Image courtesy of http://magixbuttons.com/article/visiting-castle-wolfenstein

In short: 20 miliseconds of latency is a great VR experience (the closer to 0, the better). Anything more than 60 miliseconds will cause the dreaded “simulator sickness”. It’s easy to get lost in the rabbit hole of technical specifics, but for those who want to dive deep, here are Oculus’s best practices.

View from inside the Unity scene, showing different beacons and 3D models of where Michael Brown was at various times.

Once you’ve made the different elements in your scene, you need to import them into a game engine. The two biggest players for VR, both of which have gone down the freeware route, are Unity and Unreal. A game engine to a 3D asset is what Indesign is to a Photoshop image. I’ll cover game engine basics in a separate post.

There’s a lot here! How can I organize it all in the scene?
Before you drop in your assets, you need to think about the narrative mechanics and the user experience — how will users interact with the environment in a way that is intuitive and enjoyable. This is a whole other science in itself called game design, but fortunately there are a ton of decent, free tutorials and resources online to learn more. Start here with this exceptionally rich post from Gamasutra.

Gamifying Your Story
Early on we came across the idea of using beacons as a way of triggering content, so that when users walk into them, they are presented with different forms of media — images, audio or video.

By walking into white beacons, users can see 3D models of the shell casings found at the scene, together with the accompanying evidence photos.

However — and this is a key learning that came late in the game — the more user testing we did, the more we saw that people felt overwhelmed by the sheer amount of choice and information they were presented with inside the experience. Turns out reconstructing a crime scene doesn’t fit the conventional model of an elegantly mapped out level, as pieces of evidence (such as shell casings) tend to cluster together, and the bulk of clues are often all found within a very concentrated space.

Steath Tweaks
However, the advantage of the game framework is that you can continue to make changes to the experience even after launch by simply exporting a new build. That way changes can be incorporated without them potentially being consigned to the last place people look — the addendum coda at the bottom of a long scroll.

You can also be flexible: after getting mixed feedback about whether people wanted the unrestrained freedom of moving around the complex, or preferred to cut to the chase, we solved the problem by giving users the choice: namely, the ability to select either the “sandbox” or “guided” version of the experience.

In game choice the user is given to tailor their experience: either freefrom (sandbox) or fixed (guided).

Exporting the “final” product
As I’ve mentioned before, you want to be mindful about which platform you’re designing for: is it mobile first and foremost (more accessible to a much larger audience as everyone’s now symbiotically chained to their smartphones) or desktop (higher specs and more immersive experience)? A few things worth noting:

i. If you go down the native app route (unavoidable, unless you’re working in a javascript library like three.js, which works directly in the browser like this very cool example from the Wall St Journal) keep it as small as possible! It’s asking a lot to get a user to devote 1/16th of their phone’s entire memory to a single 1 GB app.

ii. if you do down the browser route, be mindful of the fact that game builds often require a plug in, which requires a separate download — something that spells death in the eyes of the casual websurfer. Chrome helpfully abandoned the Unity webplayer extension a few months after our launch, which meant turning our loyalty over to the likes of Safari, Firefox and, dare I say it, Internet Explorer.

iii. Figure out a pipeline for exporting different builds at different stages of development. It’s infinitely more efficient to work on one format (eg. mobile) from a single project file all the way through to the end, and only then adapt it to Oculus/other formats once you’ve committed to that final version. Otherwise you’ll be ricocheting back and forth trying to find which was the latest version and saving multiple, superfluous copies.

But isn’t VR really expensive to produce?
It doesn’t have to be. There’s a sliding scale, typically based on the length of the experience — from 1–2 minutes up to 15. Just as there is between an editor commissioning a blog post or the next Snow Fall. One distinct advantage of going digital is the ability to reuse assets in future projects: to effectively build up a repository of models (cars, buildings, scenes) in which you can house and build multiple stories.

For 360 video, there are a host of cheaper out-of-the-box solutions (like this one from VSN mobile, $449 — other cameras are available), which Gannett have been experimenting with in their VR Stories app. However, for the full experience that companies like Chris Milk’s VRSE provide, you’ll need a full rig, comprised of half a dozen video cameras, whose feed you then will have to stitch together using a program like Kolor. The recently-announced GoPro spherical video rig runs around $3000 for the cameras alone, not including the rig or the stitching software (the 360 VR software pack offered by French outfit Kolor runs 700 euros, give or take a croissant).

However, as I said at the beginning, the storytelling possibilities for 360 video are still more limited than the freedom and narrative mechanics that born-digital experiences provide. I’ll explore what sort of experiences in a future post.

Curious about Ferguson Firsthand? Read more about the project and download the smartphone or Oculus versions from Fusion’s website. Stay up to date with Empathetic Media’s latest projects or on twitter here and here.


Dan Archer is an immersive journalist and the founder of Empathetic Media, a new media agency that provides virtual and augmented reality storytelling solutions to newsrooms. Drop him a line or tell him what you thought of this article at dan@empatheticmedia.com.