Volumetric holograms in XR storytelling: next-level storytelling with real people in immersive media
By bringing real people into immersive experiences, volumetric holograms are disrupting AR & VR storytelling. With them, talented performances can be finally integrated into new media to tell stories, educate, move your audience, and spark emotions. Read this to discover:
Where others see technology and innovation, some see new ways of telling stories. If that is you, thank you! People like you have helped us humans come a long way, from prehistoric cave paintings and the first printed words, to motion pictures and, most recently, immersive media. Visionaries like you push the boundaries of communication. We say thanks because storytelling is central to human existence and society development. Stories, our collective memory, can move people to tears, warm their hearts, make them laugh, blow their mind. Stories are therefore the most powerful tool to educate, convince, entertain, inspire and transform — all in all, they are the blood that flows to keep our culture and communities alive.
“The difference, perhaps, between ‘storytelling’ and ‘story living’.” Luke Buckmaster — on immersive and interactive filmmaking
New immersive technologies like Augmented and Virtual Reality (AR & VR) are enabling audiences to move beyond the limits of flat two-dimensional content. People seek to be immersed, to be active participants of the stories they experience, so they can walk around and explore. Be embodied. This presents a great opportunity for creators, as well as a challenge: telling stories in these formats is, in practice, unlike doing so in more traditional media. One of the key differences is that bringing the human factor in is not as straightforward as it is for other types of content. Immersive storytelling has the unique ability to teleport us to a whole new world — welcome to the land of stories! But, where are the characters that bring them to life?
Humans: the central element
Check your favourite movie, video, or news app — how many faces do you see? How many films can you recall without people acting in them? How many stories do you create or share that do not have someone in them?
If stories are the nurturing blood of our culture and communities — we, humans, are the red cells that carry the oxygen within! For that reason, throughout history, we’ve developed sophisticated tools to capture real people into our evolving types of content. Photography did the trick for everything static and flat from newspapers to your Instagram grid. Video, still flat, set us in motion: from films to TikTok clips. But when it comes to AR and VR, how do we bring people into our stories? So far, instead of bringing real humans to these environments, we’ve gone for alternatives such as artificial animated characters, mostly cartoonish, created by 3D artists and developers. These alternatives may be cool, but let’s face it: this is not the same as having real people in it — be it Choi Min-shik, Morgan Freeman, Juliette Binoche or your homegrown local talent. Performers are artists in their own right and contribute something essential and unique to every story. Sadly, when actually attempted, efforts to bring real people to AR and VR often turn out uncanny at best, making it better to stay low-key (think of video cut-outs integrated virtually in the space) than to fail trying something more ambitious — see this for a more complete analysis of alternatives.
If we could bring real people into immersive experiences, the storytelling power and the possibilities of AR and VR would expand tremendously.
“I wanted people in the middle of the story, not outside in a flat way. This is just part of that trajectory. Our world is spatial, it isn’t flat.” Nonny de la Peña
Going volumetric to enable AR and VR storytelling
Luckily, a disruptive innovation has taken hold in the last few years: volumetric video. Also called volumetric capture, or simply volumetric, this evolution of video enables the capture of real people into volumetric holograms — we call them volograms. This means real people brought directly into digital, not in a flat way, but rather as full-3D content that can be seen from any perspective. In other words, with volumetric you can finally bring real people into your AR and VR stories, with their performances to be experienced in a completely immersive way. Volumetric video helps you create compelling stories like never before.
An advantage of volumetric video is that the capture process is simply the evolution of a normal green-screen video shooting. The process is very similar (finding the talent, preparing the script, make-up, and wardrobe, etc.) with the key difference that the performance is recorded from multiple angles at the same time. In general, you won’t need to think about how things will look from a specific perspective: your audience will choose.
In that way, volograms capture the acting and bring it directly into your immersive work. This ensures that you can spark nuanced emotions, capture memories with fidelity, notice subtle moves and unique clothing with natural realism, and empower your chosen talent to make your audience feel. Of course, this also means that volumetric does not work in the same way as 3D-avatars: you need to shoot the sequences you are interested in (which will be realistically reconstructed), rather than artificially configure with software what the character is going to do and say. Volumetric video enables a degree of post-production (e.g., adjustment of movements), but the more you edit it, the less life-like it will look. The power is in the performance!
“The power of actors and performers is that they are focused entirely on embodying a character, with years of training and talent, and without distraction by technology or process. Giving them the ability to explore your story, and bring something unique and surprising to the table, without the complexity of say, motion capture, is a revolution.” Anrick Bregman, Director at Studio ANRK — an Immersive XR studio
All in all, once the sequence is captured, you can integrate it within your AR and VR experiences and work with it as a normal 3D-asset. Your experience will still be compatible with your standard tools (Unity, Unreal, etc.) and the full range of devices: from the fanciest headsets to a standard smartphone with WebXR. As volumetric has roughly the file size of a high-quality video and tens of thousands of polygons, you will only need to make sure that you stay within the hard-drive and processing restrictions of the devices you are working with. Don’t worry: the first Hololens works with it, so you will manage.
Very human and immersive stories
Volumetric has been used in a great variety of content: VR filmmaking, immersive art, mobile augmented experiences, etc. To showcase this and inspire your creations, we’ve curated a list of some of the best pieces of work published internationally, which are only the tip of the iceberg. You just need to take a look at SXSW or Cannes to discover new and exciting pieces every year!
- Augmented Play & VR Play — the Trinity Centre for Beckett Studies, in collaboration with V-SENSE and Volograms, published this award-winning reinterpretation of Samuel Beckett’s masterpiece Play. Using only 5 DSLR cameras, the characters were volumetrically captured and integrated into the AR and VR experience in such a way that the audience can choose the sequence in which this piece develops, as if they were on the Director’s seat. | AR & VR | description | article
- Blade Runner 2049: Memory Lab — Magnopus created a VR sequel of a classic, the acclaimed Blade Runner. It not only impressed the world with its stunning use of technologies, but it also contributed nicely to the storyline of the big-screen sequel, Blade Runner 2049. One of the first top-level VR productions to showcase the power of volumetric. | VR | article | video | Oculus experience
- An Opera Set in Your Home — Google’s Creative Lab made it possible for Opera Queensland to share the magic of one of their grand music events for people to enjoy from their homes. This ambitious experience has set the stage for other institutions to follow. | AR | article
- Terminal 3 — This thought-provoking piece by disruptive studio 1RIC is especially interesting for two reasons. First, it uses volumetric video to make awareness-raising pieces more compelling through next-level, emotional human interactions. Second, it uses a light-weight and low-budget volumetric capture technology. | AR | article | making of
- Jeff Wayne’s The War of the Worlds: The Immersive Experience — As one of the central pieces within the theatrical experience of the same name, London-based production company Dotdotdot created this intriguing piece that uses volumetric video for the most immersive scenes. Its smart combination of this new format with 360 videos is a great storytelling option. | VR | article | video
- 7 Miracles — this ambitious feature film, Raindance award-winner, produced by VIVE Studios and shot using 360 video, integrates a few volumetric sequences, which let the audience explore freely some of its key scenes. The combination of so-called 6-degrees-of-freedom (6DOF) content within a long-format 3DOF production helps spectators engage further with the storyline. | VR | description | video | video 3DOF vs 6DOF | VIVEPORT experience | Google Play app
“The difference between the volumetric actor and the 2D actor, for me, it’s night and day. It feels real and I get a human resonance in response.” Camille Donegan, CEO and Producer at Virtual Reality Ireland
In general, the previous use-cases intend to be a source of inspiration and a summary of what has worked best in the past for ourselves and our clients. However, you are best placed to understand how to integrate volograms in your experiences to add value to your own customers.
What’s your take?
Even with all this exciting work already available, immersive storytelling is in its early days. We can barely imagine how things will look like in two years from now, let alone in the longer term. We hope this article inspires you and all creators and artists to jump behind the cameras, regardless of story types and its budget size, and unleash your creative talent. The power of human stories in AR and VR will only be realised thanks to it!
If you want to make your immersive masterpiece come to life with real people, come talk to us and we’ll help you take your storytelling to the next level. Follow us on Twitter to discover more exciting work by us and stay updated.
