Virtual Fictions: a 4 week exploration of 360° filmmaking and VR
Nicholas Mortimer and Joseph Popper
Intro
In August 2016 we lead Virtual Fictions at the School of Machines, Making and Make-Believe: a four-week programme that explored ideas around virtual reality through 360° filmmaking. Our interest in VR was on two main fronts: as a storytelling medium and as a wider technology full of history, promise and complications. The course was hosted in the School’s studio at ACUD MACHT NEU in Berlin.
We set our students a brief to produce an engaging and compelling narrative experience. In the main we were working with entry-level consumer 360° cameras and cardboard headsets. Some students did go out to source Oculus Rift and HTC Vives to accomplish more technical outcomes in terms of digital interaction. However from the outset our emphasis was in the play and experimentation with what were fresh, new concerns and considerations for nearly all of us — tutors included. We were not out to compete with big production houses — we put ideas and stories first.
We were very lucky to be able to work together with Anrick Bregman of Unit 9 — an inspiring and successful VR film director with a wealth of knowledge, experience and stories about VR media production to complement our more wide-eyed wonder and curiosity.
It was exciting to step into the world of VR storytelling because it feels VR is a medium yet to be anchored or established in any particular field — the rules of what you can and cannot do are yet to be truly written. It seems the term ‘virtual reality’ itself remains contestable. With cameras, cardboard headsets and curiosity we explored our own ideas of what it could be.
The Course
The programme began with a total bombardment of the students and ourselves with different sources of information and inspiration. These varied from hands-on making workshops to guest speakers and studio visits. We then lead a more steady process of roundtable and individual tutorials as the students found their individual interests and developed their ideas. Production of their projects then geared up towards a School of Machines’ VR expo to conclude the month.
Our aim was to introduce a methodology to the programme rooted in a critical design approach, sharing our interests in the consequences of technological advancement. This was at the core of example VR-related projects we discussed with the students, featuring works of contemporary artists, designers and filmmakers. We also referred to Carl Smith’s research in Context Engineering, where perception of reality itself can alter the nature of our relationship to it. We also referenced science fiction films such as WestWorld (1973) to emphasise that a virtual reality experience need not only be inside a headset — the boundaries of a virtual experience being a key interest of ours.
Throughout the four weeks Anrick shared valuable advice and feedback about how to approach VR film production from his own practice as a VR director. His ideas and lessons about the VR medium ranged from the technical to the more conceptual, and also introduced unresolved questions and issues that are specific to VR storytelling. Anrick also impressed upon the group to think beyond the ‘hype’ of emerging VR experiences and his interest in ‘future-proofing’ the work made in the medium.
One example Anrick discussed is how current VR media content only considers and uses 1.5 of the senses in sight and sound. Nicholas lead a Phantomatics workshop in week 1, which was a fantastic and playful way to emphasise this — where students blindfolded us and created immersive disaster experiences without visual stimuli. We were thrown into a plane crash, a sinking submarine, a beach under shark attack and a fantasy jungle. The level of detail and reality achieved by the groups in a matter of hours was remarkable. In removing sight from the equation, students were able to be innovative and creative with how they shape a vivid experience. Memorable details include take-away-coffee-cup ear-lids to simulate a loss in cabin pressure, and barbecue-flavoured crisps to waft the scent of the barbecue on the beach: complete with feet in a plastic bag of sand.
VR artist Kim-Leigh Pontin gave a talk which introduced the concept of priming the viewer: to consider where a virtual experience begins and ends, and what happens to the viewer before and around when the VR headset is put on for the first time. These ideas were interpreted by one of our students, Sophie Braziunas, who designed her film and experience around a specific space. Sophie situated her narrative inside a closet room at the back of the studio, where the audience were invited to physically step inside with Sophie and then watch her and the interior space transform as she lead you through different corners of her mind and memory.
We also made a visit to a dedicated VR laboratory at the Hans-Plattner institute, guided by resident researcher Pedro Lopes. Pedro lead us on a thrilling journey through a series of haptic feedback experiments which explore how we can further our experience virtual reality though electro-muscular stimulations responding to artificial environments. It was also fascinating to see experiments into how virtual reality can be a shared experience — both by a group collaborating for an individual’s experience and — even more interestingly — from the mutual influence of connected yet separate experiences, whose actions in different worlds can directly affect each other.
What did we discover, and start to question?
We pursued many possible avenues for creative experiments in filmmaking and the production of new, immersive experiences. Three main areas emerged as key questions and spaces to explore for the students and teachers alike. These areas balanced both the technical aspects of creating such experiences on a reasonable budget (time and money) and the more conceptual concerns with regards to the wider issue of realities and their representation. Using these areas as starting points we can elaborate and question the next chapter of VR in its role as a tool for experiencing new realities.
1. What happens to narrative roles in VR? Does the viewer become the camera or an observer in a 360° film scenario?
2. The potential and constraints of VR hardware and software, and the effect of ‘stitching’ on time and space.
3. Priming: How to create an experience before the headset goes on using designed spaces and real-time actors.
These issues were all grappled with by the students, fueling a truly experimental feel to projects that were remarkable in their ambition and realisation within three weeks of production. The following three paragraphs are a summary of what we have learnt so far:
1. Camera or Observer?
Perspective is key to the experience. When designing experiences for VR, the camera is often imagined either as a character or an observer, and in some cases a ‘ghost’ invisible presence.
Using the 360° video tools at our disposal — there is no responsive interaction with any of the action seen in the virtual environment. This means, as a viewer, your given role in the scenario is passive — the action comes to you.
There is a need for constraints of a fixed perspective to be overcome in an immersive environment. The traditional cinematic syntax — based on the free shifting of camera angles to weave together compelling, emotive situations — is lost in translation to a 360° film setup.
However, the key here is in creating situations where the act of inspecting can become a narrative experience in its own right — by immersing the viewer in a situation a level of direction can be designed into their experience of the story.
By fixing the camera and directing action to focus around a central point, an effective narrative structure can be achieved. For example, by keeping the viewer fixed to a chair inside a scenario you can give the illusion of freedom: distracting them into an experience where their level of attention can be significantly increased.
2. Hardware, software and the effect of ‘stitching’ on time and space.
By playing around with the limits of our technology we can learn a lot about the potential, effects and metaphysics of VR.
One filmmaker in our group, Sarah Riazati, discovered that injecting any video with metadata can make it spherical — not only footage from 360° cameras. This simple observation transformed what possible kinds of imagery could be made — we were no longer dependent on only what can be filmed in camera. Archive footage, animations and other imagery were now valid forms and resources for storytelling when run through a super simple piece of software. In hindsight incredibly simple but, in the context of an experimental workshop, this discovery was a game changer. Furthermore, this discovery returned our attention to the VR video stitch line: what is obscured can be just as important to the 360 experience consists of.
The resolution of current cameras and headsets will always interfere with our response to certain VR environments. However when making 360° films, these aspects can be exploited to our advantage. This can be achieved by using a variety of effects that can blur, interfere or distract from the reality of the hardware used in the making.
What happens if stitch lines are welcomed as glitches in reality, rather than being laboriously hidden and corrected? Could the overlapping, hidden spaces and invisible states become a far greater source of narrative counterpoint? Just as the language of cuts evolved in early cinema: with swipes, dissolves, blurs or ripples, could we see the stitch line become a tool to describe the transition between different states of mind, layers of internet or moments in multi-faceted time warps?
3. Priming the viewer
The inclusion of ’priming’ or preparing the viewer for the context of the VR experience greatly enhances both the overall appreciation of the narrative and the danger of the headset becoming a distraction.
By bridging the gap between the real and the fictional, there is a far more unique and personal experience gained by the viewer, who will be able to shift their perspective into a greater sense of suspension of disbelief and immersion. This can be achieved in a sliding scale of designed moments, either as a basic scripted interaction with the person who provides the headset, or as a more elaborate scenario that extends to the space and place the viewing takes place in.
What could the extent of priming be in a VR situation? Is it possible to imagine multiple encounters of priming that occur both inside and outside of the digital experience, meaning that the act of removing the headset becomes one aspect of the wider narrative, meaning it is no longer the dependable divider between the real and the fictional space of the story. Is this an acceptable form of VR, and if not, why not?
The utilisation of pre-emptive narrative tools should look to the rich and complex history of the theatre as a way to understand the importance of the layering of expectation. Suspense or excitement can begin the moment a viewer is aware of a story, and VR experiences can become more holistic immersive experiences if they appreciate these nuanced and often hidden tactics.
What were the outcomes of their projects?
The outcomes by the group were diverse in terms of aesthetic, approach and content — all were inventive in utilising the tools at their disposal to achieve different narratives or experiences. The following is a summary of a few selected projects:
Filmmaker Sarah Riazati weaved and warped found footage from her personal Youtube channel to create a trio of video textures which were both immersive and atmospheric.
Illustrator Deborah Rodriguez overlaid hand drawn animations with live action for a 360° music video — following a group of friends around Berlin to a musician’s soundtrack.
Game designer Marie Le Blanc Flanagan coupled an Oculus Rift with a tattoo artist to — through gameplay — offer real world consequences of virtual reality. She designed a game where if you squash a fly in VR, your prize can be an immediate, permanent tattoo of the insect on your arm.
VR Director Elli Raynai deployed DepthKit to capture a 1:1 scale set in his interactive film ‘The Key’ — which housed the story of a boy’s encounter with his father, a reinterpretation of a childhood memory.
VR filmmaker Herman Fredlund also accomplished a 360° music video — where he adapted more traditional cinema tropes (such as cuts and changing perspective mid-action) to frameless film as the video follows a couple dancing tango through different scenes of Berlin.
What was the final show like?
We concluded the programme with an opening which showcased the projects developed by the students. The feeling was informal, and treated as an opportunity for the students to share their experiments and ideas about virtual reality in an open and honest manner.
It felt more of an expo rather than an exhibition, with every student occupying a certain floor space with a dedicated work to view. The setup also enabled the students to mediate how the experience of their ideas. There was a playful atmosphere of testing and leading the audience through individual experiences.
The dynamic was interesting with the viewing of works based on a single interaction at a time. This also encouraged visitors to engage in conversations about what they experienced, comparing and contrasting their thoughts and feelings about each work.
The atmosphere made for a very discursive and enjoyable evening, and also provided an invaluable chance for the students to test their works on a wider audience outside of the group.
Conclusion or Why is what we did valuable and interesting?
Virtual Fictions introduced a curious and experimental attitude to virtual reality and 360° storytelling. We provided a platform for practical and conceptual approaches to immersive and filmic experiences. Working through different exercises we lead an atmosphere of exploration, prototyping and iteration.
Four weeks is clearly not enough to discover the full potential of Virtual Reality as a medium for storytelling. There seem to be infinite ideas to unpack, and realities to test and immerse within.
The emphasis of the programme was in the ideas and we readily embraced an honest, playful aesthetic that ran through many of the students outcomes. We emphasised that virtual reality is not just a solely visual experience and consciously attempted to expand ideas out from more industry-led, image-heavy predilections.
There was also a constant desire throughout to push the potential of the basic, consumer technologies available to us, and to be inventive around the constraints. In a short timeframe, the studio achieved a broad and exciting range of outcomes which each explored how virtual reality mediums can transform ideas and narratives in different ways. The programme has ultimately taught us all that when designing experiences, imagination and ambition triumphs over the high budget and highly polished. The best works don’t come with the best kit — a notion we endeavoured to share throughout the course.
Like this story? Get more in the weekly newsletter from CinematicVR.
