Turn up the volume
Henry, theater, volumetric capture and the future of VR stories
My last emotional relation with a Hedgehog is one of rage and frustration. It goes back to Sonic 2 and my desperate attempts to finish the game, but when Henry glanced at me, I know something would happen between us. I had the chance to be present at the Oculus Connect conference and to attend an Oculus “premiere”, Henry. Henry is the second movie produced by Oculus Studios, for lack of better word, and one of the first VR stories to be told in the right way. It’s quite difficult to qualify it as the experience is really different from the Hollywood movies we know and love, and even from the many 360 videos coming to VR. “VR story” would be my preferred wording. I’ve consumed my fair share of VR scripted experiences, and it was the first one that felt really right.
360 shades of “meh”…
I’ve worked on S180 and 360 video content for more than a year now with my project Surgevry : it’s an effective medium to capture reality in a quick and efficient way, which has been really practical for surgical training. It is also very powerful to transport you somewhere else : I’ve been to Syria with VRSE’s “Clouds over Syria” and transported into “the Wild” with Paul & Felix, but I haven’t felt a strong empathy towards my fellow humans presented in these environments. They seemed far away, strangely scaled, and somewhat distorted. They seemed 2D.
The lack of positional tracking is quite disturbing in 360 video : you’re inside a sphere and can’t change your point of view, since the cameras capture only the light at a central point. I’ve always had the strange claustrophobic feeling of being trapped inside a snow globe. It doesn’t help that, given the origins of 360 video from 360 photo (which require a static environment), the captures we see most often are dull animated panoramas, that quickly become as fun in VR as your dentist’s screensaver. There have been heated debates about the fact that spherical videos are VR or not, but it’s clear it’s not the climax of what can be achievable in VR. It lacks those precious 3 degrees of translation that convey that deep sense of presence.
…versus six degrees of freedom
Henry was different. The three degrees of translation offered by the positional tracking and the generous coverage of the Oculus Constellation system allowed for natural movement within a scene rendered in real-time (with the UE4 engine). Natural movement doesn’t mean you want to walk freely in the scene : the wire from the headset wouldn’t let you do that and if the director has done his job right, it’s not something that you will want to do. This a problem that will be familiar with any VR developper : locomotion hasn’t been solved yet in VR and there is no way to move naturally in a big environment without experiencing motion-sickness. Henry has a pragmatic answer to that :
“Please take a seat on the carpet”
This where the only indications that the Oculus hostess gave me while putting the headset on my head. The famous “So now, what do I do?” that I got from so many people I made try the headset has a simple answer when it comes to VR stories : “take a seat and enjoy”. This is a story, not a videogame, this is a passive experience that is cripted, something to contemplate and observe. The Henry environment is made for this kind of passivity : all the action is out of reach (happening beyond the tracking cam). One nice addition to the demo : you are standing in real-life on a carpet that’s a 1:1 replica of Henry’s house carpet. That contributes to the immersion and shows you clearly in which area you’re free to move . You won’t leave it for the 5 minutes of the experience.
So why is position tracking useful if I can’t really move around? One word : immersion. Research has stated numerous times that without 6 degrees of freedom, you can’t reach the mythical presence in VR and feel like you’re there. This experience made it crystal clear. The tracking volume is pretty good in the CV1, at least it lets you experience the carpet very closely. Having the correct parallax effects, even if your head isn’t moving much, is very important so that you let your conscience flow away and enjoy suspension of disbelief. You’re with Henry, in his home, and you don’t particularly feel compelled to test the limits of the tracking system and try to find out what’s on the other side of the home, as you wouldn’t get up on the stage during a play.
Oculus sets the scenes for VR stories
And it’s fine, really! Because the story is simple, compelling and emotional. The connexion to the characters is particularly interesting ; Chris Milk defines VR as the “ultimate empathy machine” and you can really feel an emotional connexion for this friendless hedgehog. The story itself doesn’t have anything to do with the codes of cinema, it’s more of a play, a very classical play, with its classical unities : unicity of action (one plot), unicity of time (real-time), unicity of place (Henry’s house). The best analogy might be the emerging field of immersive theater, with examples as the Sleep No More show in NY, when spectators move freely around actors who don’t seem to see them.
Video killed the theater stars
As in a play, the emphasis is on the actors. In VR, there are no clever shots, no nice editing, no fancy FX, you are alone with the characters. They’d better be good.
Henry copes with that nicely with a Pixar touch : the hero is an anthropomorphous creature you can relate with. Animators at Oculus Studios did an amazing job on Henry, but they must have spent months on this. Every story in VR can’t be as polished, and we will need a way to display in VR the creatures we most easily connect with, humans.
Video games traditionally use avatars, rigged models, to display characters. You can’t really do that for VR stories. Despite the marvels of technicality deployed in the latest videogames (for instance the latest MGS), avatars don’t embrace roles, avatars don’t feel human, avatars are acceptable on a small screen but creepy when you get them in your face, real scale, in a VR headset.
The only way to capture characters correctly is though “volumetric capture”, a way to record someone in full 3D -not only stereoscopic 3D-, a technology we’re exploring at my company Mimesys. We use depth cameras such as the Microsoft Kinect 2 to record humans as fully animated 3D meshes and not simply 2D pictures or rigged 3D meshes. We will release very soon a free SDK here, so that anyone with a Kinect 2 and a PC adapter can try : because the structure of compelling VR content hasn’t been discovered yet, we want everyone to come and try with low-cost solutions.
Even if our current quality is far from perfect, the results of Mimesys human captures are really surprising in VR : in front of a “holographic human”, you quickly recover all the subconscious social cues you didn’t know you had. You have to cope with the confort zones and bear with the fact you won’t have any social feedback from the recorded actor over there, which is something difficult to handle first! Of course it’s very different with live-streaming, which is something we’re also exploring for the moment.
The future of stories in VR
Stories will evolve with VR. In a great interview back in 2012, John Gaeta (Academy award FX winner, famously known for the Matrix and now working at LucasFilm) described the future of stories as passive experiences you’ll be catching in these gigantic immersive universes. You just run into them and can choose to stop by and see them or to go elsewhere. The man is working on Star wars now, so imagine the whole Star Wars universe as a passive MMORPG where you could connect to various plots whenever you’d like and see them happen in front of your eyes, possibly with other people of your choice. Immersion would be there for suspension of disbelief, not for interaction. And we are pretty that, at the center of these universes, there will be humans, holographic humans, waiting for their stories to be told.
We can’t wait for this to happen.
