Virtual Reality Beyond Games

Understanding the new medium is hard. Here’s how we tried.

One of the first virtual reality movies: Banshee Chapter: Oculus Rift Edition

Back in 2014 when we had our first virtual reality experience we were like many others, stunned from the immersion, that easy roller coaster demos revealed. So we started imagining impressive worlds experiencing all the things we ever dreamed of. Walking through walls, flying through worlds or having a spiritual kind of experience, something we are simply not able to in real life.
A year later, nothing really changed, the VR demos were still the same demos, with more or less interesting features and when we watched the first heavily announced VR Movie, we were very disappointed. Our assumptions on what should be happening just went a lot further.

The Disappointment

The first disappointment we experienced while watching a (VR)movie through goggles was that the director wasn’t aware of the function of basic methods in film making and what they were used for.
For example cutting from one persons face to another in a dialog will help a viewer watching a regular movie to follow the story. But in VR, people will intuitively move their heads and execute this step on their own.
That means that normal methods of film making don’t work anymore and moreover most of them result in constant disorientation because for example the cut and the movement in total makes you look somewhere behind you.
Disorientation is a huge problem because searching for the right thing to look at makes you miss most of the story.

But what is left if you remove almost every method of film making developed over many decades?
The most important part from our point of view is not how it’s edited, it’s the characters and how the story can deeply address to our emotions.
This seemed pretty certain to work with so we decided to focus on the technological part to find out how to implement or to combine these components as best as possible.

So we decided to set up some experiments to get familiar with the technology and to observe how we can prepare a viewer for new impressions, if physical stimuli can help to keep the immersion and what components are important.

The Zentrifuge

In our first experiment, we tried to explore the effect of not matching physical and visual impressions on the viewer.

The experiment consisted of a prototype of a rotating platform with the test person laying on it equipped with the oculus.
When we started to rotate the platform, the oculus showed a recording of the same sealing but rotating in the opposite direction.

The first experiment: Zentrifuge

It turned out people had several methods to (subconsciously) deal with the situation, that the visual and physical impressions didn’t correspond to each other.
One group of people experienced motion sickness while others simply closed their eyes which means that they refused to believe what they were seeing. The last group at first had a weird feeling, but after seconds it seemed, that the brain just reversed the balance. This people of course had an idea that something was strange, but they couldn’t determine. So their brain just tricked them into believing their eyes, not their balance.
We think this was just possible because they were laying and didn’t need their balance, otherwise they had to (sub)consciously focus on the balance to be sure.

Reality Exists Only in Our Minds

That fact the human brain tricks itself to keep working correctly was very impressive to us, so we started getting deeper into the psychology of exception and perception with three major findings. One was from Dr. Thomas Kieke, a psychologist who is working in a closed psychiatric clinic, one from the book “Hypnose in der Psychotherapie, Psychosomatik und Medizin” and an other from a TED Talk by David Eagleman.

When we met Thomas and talked with him about our first experiment, he was curious about the VR technology and how people can accelerate their imagination through it. He stated that “the biggest simulation of all is the brain itself, but having the visual impression supporting it could lead to phenomenal results to the treatment of patients.”

Let’s look deeper on hypnosis as it is described in the book. Hypnosis as a trigger for the imaginary power of the brain needs the surrounding and all impressions you experience before the induction and will have an impact on the strength and deepness of the trance — or in our case the immersion of the VR.

David Eagleman gets to the heart of it: “Your brain is locked in a vault of silence and darkness inside your skull. […] So that it takes this inner cosmos and puts together a story of this, your subjective world. But here‘s the key point: Your brain doesn‘t know, and it doesn‘t care, where it gets the data from. Whatever information comes in, it just figures out what to do with it.”

So what would happen if you try to maximize the imagination while being in a virtual reality?
We assume, that people won’t be able to determine what exists in reality and what not. And now having virtual reality as a part of our world, we have to renew our look at this correlation. 
Normally we rate something as real if we can detect it through at least three different senses, since than it gets incredibly hard not to accept the entity as actually there or in an abstract way, if it has a measurable impact or consequence on our subjective reality.

In other words, we have to identify where people draw the line between their subjective real and virtual world and it seems that identification with the scene plays a huge role.

Screenshot of the Arachnophobia Oculus Demo

The Arachnophobia Oculus demonstration for example shows this in an impressive way. Even with low graphical quality and a minimum of interaction possibilities, it exposes the true power by activating your instincts and trapping your mind with spiders walking beside your arms.
In short, instincts do not distinguish between real and virtual and that is why they become a strong tool for keeping the immersion alive.

Therefore we thought testing the brains affection and rejection to deep digital stimuli is the way to start.

Hypnotized by Apple Speech

We chose to put somebody into a light trance by hypnotizing the subject through the voice of our computer.

The trance was supposed to be a simple arm levitation and supported by a live link from two web cams attached to the ceiling streaming the footage directly to the oculus of the subject so the person saw him/herself lying on the floor. At the strongest part of the trance, the computer advised to open the eyes after your mind floated up under the ceiling looking out of a tiny hole down at your body, yourself.
People reported that they felt scared trusting a computer and let it in a way, control their minds. But it worked and people reported furthermore that they had kind of an out-of-body-experience when they where in trance and saw themselves from above raising their arms for several minutes.

Because this really scared us, we decided to stop scamping around in peoples minds and leave it to real professionals. Instead we focused again on preparation and guiding to create a strong identification.

The Black Box

The Black Box aims to test how a viewer has to be influenced before and during a VR experience to create the immersion as deep as possible and furthermore to see if the VR can help the brain hiding the real physical circumstances, in the same time.

The Black Box from Outside

The experiment starts with the participant walking towards the open box forced to expect a different situation than watching a movie in cinema or in a couch like surrounding.

What happend inside the Black Box.

As shown, the viewer has to go inside the box before wearing the oculus. Sitting in a tiny box, with no sensory stimuli it got pretty fast pretty creepy for them. The cool thing is, that people seem to hide directly into the VR, like addicted to gain any kind of impression.

Because we gave the box in the scene such huge attention, it became some kind of a reference object to the real world to the viewer. It was the feeling of having a gate from one reality to another and entering the gate through getting into the box.

People reported that they thought they had to find the box again to exit the VR, so they were kind of shocked seeing them destroyed. This idea and having impulsive music, triggered their instincts and again kept them from verifying. Following our own rules, the viewer is forced to quickly verify the threatening rising sand in the real world, which stops the experience immediately.
In this case we refused to have a static walk through the desert, instead a dynamically rendered and triggered flexible story. So we controlled everything by watching the viewer and accelerating his head movements and triggering the parts of the scene in the right moment.

But How Does it Affect Storytelling?

Looking on the transportation of filmed/captured content for virtual reality the problem is that the director doesn’t have the control anymore. So the content has to be experienced even if the viewer misses parts of the story and even more. The story has to be understandable with a minimum of reached content. In reverse an extended and flexible story with a lot of small hints and multiple ways of the narration could create always a unique story or experience and ensure, that the viewer gets most out of it.
Another big problem for VR is that it has to take place somewhere. You have to stand, sit or lay to concentrate on the visual impression which makes you unable to move and or interact properly with your surroundings. So basically the biggest advantage of the technology, of emerging yourself to anywhere else, can also reveal the most weakest point in terms of storytelling.

Is There a Virtual Reality Without Reality?

In our understanding reality is the state where things can influence your surrounding and decisions have consequences which affect (your) life. Moreover people tend to rate things as real if they don’t have to verify that their not. Imagine a small elephant slowly walking on your hand, if you can see, feel the feet with every step, hear and maybe smell it then intuitively It would be very hard for you not to believe that it’s there. Sure it’s not possible, however three or four sensory impressions would be hard to hide and belief your own logic.

This is not yet happening in pure VR, so technically there is no “reality” part because it does not have any of these characteristics and we think, that’s the biggest contradiction in the naming and meaning of the medium itself.

Anomalies Are Something in Between

If you look on the advantages of the technology the most important part for now is that there is always something to see, no matter where you are looking at (like in reality). For us, this is the most overlapping characteristic. So it’s obvious: Let the characters and story happen in your real environment. Ideally a mixed reality that does not depend on tracking technology and just takes everything real and transforms it into parts of the story, would have the strongest impact on the participant. Imagine that walls are just where walls are, but there can things emerge out of them or come around the corner. Just think about Nemo waiting in your aquarium or the famous scene from Psycho happening in your bathroom.

ANOMALIE is our proposal, how virtual content can be as real as possible for the viewer to have the strongest influence and immersion.

ANOMALIE took place in July 2015 in Würzurg, Germany