working in co-located VR
Virtual reality is typically a solitary experience. One person sits alone with their headset on, looking around at who knows what while everyone else watches them, thinking about how foolish they’ll look when or if they ever get a turn. Festivals run long with lines of frustrated people, wondering if they may have made a mistake with their choice.
Flock is a shared, immersive experience made for a continuously cycling audience of three to thirty participants which recently premiered at the Future of Storytelling Festival in New York City. Flock is a dance party, a game, a sandbox and a participatory story. As far as we know it is the largest simultaneous VR experience ever attempted and the highest volume VR experience put on at a festival (over seven hundred participants in three days).
I have discovered that the difference between trying a Holojam experience and hearing about the experience is vast and I think it’s important to try to bridge that gap and make it clear how very deeply affecting and revolutionary this art form will be.
What I’ve learned so far
When I started working with the DK1 my initial instincts were to extend my knowledge of animation into the virtual realm. I am a strong believer that tight control of pacing is necessary to immerse an audience in a narrative and I never had an ambition to create interactive art or gamified narratives. Rather, I wanted to create atmospheres to float through or highly structured narrative shorts. My work at NYU has shifted my ideas considerably.
Rather than approach VR with traditional storytelling in mind, I have started to think in terms of curated sandboxes. A sandbox allows for play and expression in a way that doesn’t really exists outside of video games and preschool. Video games don’t allow for full physical immersion and you might get arrested if you try to nose in on activities at a preschool.
Holojam
Holojam is a completely transformative experience. Physical, tracked objects co-exist with interactive, virtual objects. Participants see each other as avatars and can interact with each other.
Ken Perlin invented Holojam. He is an Oscar winning computer scientist and artist whose work graces the guts of practically every software package that is designed to generate images. I began working with him in 2013 and my first experience sharing a walk-around virtual space with other people was a mind expanding experience.
A combination of techniques in motion capture, networking and game design are employed to make a Holojam happen. Virtual objects may be controlled by physics simulations or behavior engines. Their state can be saved and reloaded allowing for persistent worlds to be shared. Avatars are built with procedural animation techniques and do not require full-body motion capture suits. Feet and legs can be filled in if heads and hands are being tracked. Entire bodies can be made to animate underneath a user with nothing more than a tracked headset. Avatars can be mapped in indirect ways allowing participants to embody non-bipeds with exaggerated proportions.
We have been using Holojam to study how people behave in small groups when immersed in a shared virtual space. As an artist, the most inspiring thing to me about the medium is how excited and uninhibited people get when interacting with each other in VR. Novelty is clearly an important ingredient in the creation of this excitement but I think it also speaks to the desire we all have to shed our individuality, see past our differences and become part of something larger than ourselves. The experience offers a glimpse into what could be a new and as yet unnamed art medium — a co-located, room-scale, mixed reality sandbox event. In lieu of a less verbose description I will refer to this particular collection of elements simply as a Holojam.
Flock was born out of a desire to create a space for people to move together. The particulars of the experience are not as important as how they inspire interaction. The initial thinking led me to consider using a flocking simulation to encourage individuals to move along pre-set flight paths. Flocking behavior has been a staple of creative code in computer graphics for twenty years. Craig Reynolds defined a set of simple rules which lead to surprising and complex emergent behavior.
From Craig Reynolds website:
Separation: steer to avoid crowding local flockmates.
Alignment: Steer towards the average heading of local fockmates.
Cohesion: Steer towards the average heading of local flockmates.
These rules can be applied to human behavior and even our neural machinery. Birds cannot conceive of the flock structure that they are participating in any more than a neuron knows what an idea is. Feeling good is nature’s way of guiding us and as groups make us stronger and more resilient so it feels good to move together and participate in things which are larger than us.
Similar experiences can be found in ancient rites where individuals would shed their identity and embody the spirits of animals. Daily life does not invite this kind of role playing, and for good reason, but given the opportunity and a safe space individuals are more than ready to transform themselves. Holojam is a costume for individuals but also the entire world. This creates an apparent separation from the confines of one’s own physique and leads to a surprisingly non-judgmental space where individuals are free to express themselves in the moment.
The UX
Flock is an outgrowth of a desire to learn how groups of people can be encouraged to move through space in coordinated ways using a combination of visual and aural cues. The idea is to use the best of VR’s techniques in the simplest way to create the greatest possible impact on the viewer. Rather than starting with content and working out from there to the audience, I started with what I wanted the audience to do and how I wanted them to move and developed backwards into content.
The very initial concept to encourage movement involved running a flocking simulation on the computer and then guiding individuals along those pre-set paths. Each user would be presented with a tight tunnel of light that they would need to fly through and keep up with in order to achieve a flocking effect. While I still believe this could lead to some interesting results, it would require experienced users with a certain level of trust and daring for it to work. Instead we chose to focus on UX design which would allow for a high tolerance for inexperienced participants.
The first idea we tried to implement involved a mother bird flying through the space and dropping food pellets that users would need to collect in order to power up. Part of the attraction to the mother bird would be the bright spirals of light that would follow behind her. Successfully flocking would mean forming a sort of branching conga line out from the mother bird. As each individual in front of you eats food they will deposit it out of the backs of their heads and into your waiting mouth.
Another idea involved turning the most accomplished (in terms of bugs eaten) birds into mother birds that could lead their own flocks around. In this scenario there would be tiers of bugs. Bugs that exist in space that only grown up birds could eat and bugs that would come out of other birds heads that only younger birds could eat.
Recreating the entire lifecycle of a bird also meant exploring the possibility of reproduction. Flock has a birth nest, and a death nest and in one version it also had a nuptial nest, In this case one other bird would become your mate and you would need to seek that mate out. Rewards would come when users faced each other and mirrored each other’s movements. The rewards would involve the bird masks becoming larger and more ornate, pointing more closely in the direction of Balinese demon masks with peacock feathers emanating radially.
The most absolutely simple way we could imagine to encourage people to move through space was to use a concept akin to a ‘space filling algorithm’ — which does exactly what you think it would. We filled space with bugs that would appear anywhere that birds weren’t. That meant that anyone that wanted to eat would not need to wait until the bugs landed on them. We ultimately wound up using this for our first user test and it proved to be forgiving and effective.
People
It was common for users to initially be very tentative in the space. There is no floor in the virtual space, and it is not immediately clear what the safety features are in terms of bumping into things (these features quickly become apparent).
As you would expect, different groups had different dynamics. For the most part as people warmed up to the slightly disorienting space they became more bold in their foraging. Squawkers found it to be the most fun to make bird calls at each other and didn’t necessarily care about bugs. Competitors treated bug eating as a competitive sport and endeavored to get as many as possible. Movers just enjoyed the process of moving through the space and once some confidence was achieved they would begin to stride through the world. Interactors were mostly focused on other birds. They would follow others and were always drawn in by beak to beak interaction.
There is some part of me that feels redeemed when I can create enjoyable experiences for people. I fail more often than I succeed but if the smiles on the faces of our audience were any indication I will be able to sin blissfully for some time.
The Future
Just a week after we presented this project I performed a live 3D drawing on the Vive using a tool created by Scott Garner with the phenomenal Jacob Marshall of the band MAE and Oak who is presently starring in the broadway show Hamilton. The drawing was ‘projected’ in WebVR to a room full of over seven hundred people watching on cardboards. This model presents the possibility for a new kind of performance or festival experience where users would enter with their own phones, visit a website, and share a space inside a world being created and modified in real-time.
There is much more to explore with this small subset of the Holojam medium. A future version may be thirty to forty minutes instead of just seven, it may enable live DJs, or combine many different kinds of animals. Ken’s next vision involves drastically reducing the price of room-scale tracking so that these events can happen spontaneously in any venue or even at home. Insofar as art is meant to heal and connect, Holojam is a powerful medium for facilitating such experiences.
Thank You
None of this would have been possible without the vision of Ken Perlin and the support of NYU’s Media Research Lab. Melcher Media and everyone involved with the Future of Storytelling festival provided the venue and the means and the vision and confidence to take a risk with relatively untested technology and totally untested creative. Tim Fain and Julien Mier brought a powerful creative energy to the process, inspired the final vision and are very much responsible for the hypnotic and soaring feeling of the experience. The long list of volunteers made everything run smoothly and my key collaborators brought much needed energy to a laborious process.
Produced by: Object Normal and Holojam with support from NYU/MRL
Created by: David Lobser
Music by: Tim Fain and Julien Mier
Lead 3D artist: Lily Fang
Code/UX: Christopher Romero
Sets and Costumes: Corinne Brenner
Project Management: Jessica Fiorini
Holojam:
Principals: Ken Perlin, Michael Gold, David Lobser
Team: Sebastian Herscher, Aaron Gaudette, Wenbo Lan, Daniel Zhang, Scott Garner
Special Thanks: Peter Kellogg, Jess Bass, Katy Yudin, Laura Roumanos, Charles Melcher