Symphony of Conclusions
Sequence of thoughts on volumetric interactive webVR during TZINA: symphony of longing. Get lost with me, I think I’m getting somewhere.
When was the last time you let yourself get lost? Do you remember the feeling of not knowing where the path leads to? What will the next move bring?
In a way, the web gives us a sense of control lost; swiping from tab to tab, delving deeper into the matrix — losing time, but still in control of space. 3D spatial stories are now asking something new of us. In order to benefit from them, to truly see them and understand what is happening around us — we need to release control over time and space. To truly get lost. To give up the sense of control. In spatial VR humans have more opportunities to expose themselves and to be vulnerable. We can let ourselves get aimlessly lost in a virtual space, like children, as well as to play and to explore — things that have almost vanished from our routine as Western adults. Technology is in a stage of reminding us what it means to be human.
Tzina is a song of lost love and of longing for things that are no more.
This is one of the main principles that guided us in the making of Tzina: symphony of longing, an interactive webVR documentary that we’ve premiered this year at IDFA DocLab. Tzina is a project that is set in a small public space — a square — in which it is impossible to get lost in, in physical reality. It is a space that is entirely visible from every viewing angle.This square is filled with regular people who frequent it every day, at regular hours. They sit on the same benches, and find comfort in being alone, together. This is a setting, a feeling, a place, that I wanted to capture and preserve. But in order to do so I need you, the viewer, to be in it. Not just pass by it, but to actually get lost inside of it, alongside the people who come there. Once you are able to do that, maybe then, you might feel connected.
From day one, we thought of Tzina as a 3D spatial documentary. The thought of letting the viewers walk around the protagonists and to offer them the chance of sitting next to them was magical to me. We knew that we were going to use Kinect with DepthKit to capture live action 3D characters.
Work using this filming technique on documentary subjects in open field conditions was difficult and bordered on creative sadism. We pulled out an 18-meter power cable, two computers, a Kinect, Black Magic, a tripod and portable sound equipment, and waited. Sometimes we waited for a whole day, sometimes two. Sometimes a character just wasn’t in the mood to be filmed, another was sick and one got cold feet and disappeared for a while, so we just packed and came back the next day.
The project’s characters were not committed to or responsible for it and creating the complex filming infrastructure on a daily basis, when in some days no documentary subject showed up — not to mention the effects of the surroundings, both in terms of noise and light — was difficult and often frustrating. Despite this, I feel these were necessary steps to take in ensuring the authenticity of our documentary content as well as our desire to keep the characters in the documentary space which Tzina attempts to imitate. Furthermore, the gap between the intimate and sensitive nature of the content (love, loneliness) and the technological challenge was an interesting one for me as an artist.
One of the first references for the style of the project was the short animation film Ryan, directed by Chris Landreth.
The idea of remaking a vivid portrait of memories and feelings influenced our creation and was a source of inspiration.
Using one kinect gave us ~180 ° of depth. We looked at our 3D interview materials as bodies that needed to be filled and as an opportunity to give life to our protagonists’ emotions. By this action the animation affects both the viewer and the character’s body, with the hopes of creating a connection between them in an intimate space.
One of the project’s cornerstones was our approach to work within technological constraints. We did not act to hide the limitations we worked within, but rather adopted them as an ideology, as they represent the point in time in which the project was made and may have the potential to be chronologically unique.
We also saw a lot of creative potential in completing the characters’ bodies using animation. We saw that animation within the 3D world had the ability to bridge the experiences of the protagonists and the viewers and to momentarily create a shared experience which reveals the uniqueness in the “everyday” story of a stranger. It was another dimension for the viewer to get lost in.
The 3D animation allowed us to create sub-worlds within the bodies of the virtual characters themselves, should the viewer choose to literally stick their head inside of them. I wanted to create the feeling that as you let your curiosity about a specific story grows, the character telling it gives you more.
This action raised multiple moral dilemmas for me as a creator, because the animations were my and Laura Chen’s personal interpretation of the characters’ stories, accentuating externalized elements in the character and treating their body as raw material to be manipulated. In addition to classical film editing, we were adding other forms of interpretation: weaving virtual extensions of the interview subjects’ bodies, distortions of the body by means of effects and shaders, positioning in the virtual environment according to the story’s orientation and not the person’s, and so on. I struggled with these questions, debating them back and forth between myself, committed on the one hand to give these people’s stories a noticeable emotional effect, and on the other to respect their bodies.
The second thing we knew for sure when we started to work on Tzina was that the platform that we will be using would be webVR for Chromium browser and HTC Vive. As creators, we sometimes felt lost in the complexities of this platform. This platform was chosen for several reasons: public accessibility, open source infrastructure, the tension between a technological platform that is still being constructed and established against a physical space that is fated to be demolished, the ability to integrate multiple viewers at one time in order to invoke a feeling of being “alone together” in the digital space, the desire to be part of the movement to rebuild and reimagine the web — a world which is beginning to transcend the borders of the screen — and a strong artistic desire to push the technological envelope as far as possible in order to examine our own place within it.
In retrospect, while I would not have chosen a different platform to work with, the biggest challenge we faced in this project was maintaining 90 FPS consistently, whilst still rendering a complex 3D setting in real time in a web environment. The lead developer, Avner Peled, and the technical director, Or Fleisher, ended up developing many shaders for our 3D characters and environmental elements in order to meet these requirements.
The choice of a 3D spatial experience was also made in order to distance the project from everyday web/digital experiences like browsing and accessing email as far as we could. Tzina is a fully immersive digital project, in which no clicks are made and no interfaces outside of the viewer’s body are utilized. I believe that other than the mouse click’s association with mundane web activities, it also serves as a reminder of the viewer’s existence outside of the movie. Instead of allowing them to immerse themselves and forget about the outside world, clicks and artificial body extensions, such as the Vive’s controllers, do not challenge one’s conception of reality enough. This decision made for a constant struggle throughout the work process and encouraged the use of unconventional solutions (as well as fights within the team). For example, because of the need to prevent a situation in which the viewer teleport themselves into another space within the square (because of the Vive’s movement limitations), coupled with our desire to create a constant state of motion (like we have in cinema), we created a mechanism by which movement between episodes is symbolized by rotating the characters sitting on the square’s benches towards the viewer. This mechanic created an interesting spatial dynamic which received positive reactions.
We decided that we would design the user experience in identical ways for both the web and VR platforms, and that we would treat these two separate platforms as one hybrid platform during development. As a digital native creator, I was challenged by thinking about UX as identical across two disparate sensory platforms. Naturally, these seem like two platforms that do not “like” each other and which cannot easily interface. The Web platform is not yet ready — or may even actively be resisting — the basic technical elements required for storytelling in a fleshed out 3D world, and the VR space is very limited in comparison to that which could be accomplished using other tools (arranging eleven characters and sixteen “extras” in a 3X3 area was exceedingly difficult.)
We’ve tried to turn the project’s world into an internal one by creating a surplus of “is” using symbols, metaphors and decorations in order to process the ‘seen’ as originating and driven by the internal worlds of the characters. This decision was very interesting from a documentary point of view because it led to a form of generic hybridity between the genres of fantasy and documentary. There were moments when we knocked our heads against the wall because everything was too heavy on the platform. Since we needed to come up with “light” solutions to handle the technical challenges we came up with the idea of representing “extras” in the square as Point-Clouds people. Ziv Schneider stopped random people and circled them, capturing over a 40 photos per person. The result was mesmerizing — dancing pixels shaped like holes and colorful humans.
Tzina is a story about lost love and loneliness. I was drawn to these people since I, as a person, often feel lonely, and never really succeed in communicating this feeling to the people I love. It’s like a non-feeling that can be challenged only by an honest encounter. Since the results of the volumetric captures were full of noise, we positioned the characters in a way that forced the viewers to look for the angles in which the digital characters look more human. I wanted to affect their physical body as well and to encourage them to go down on their knees when they interacted with the characters. To be equal, as humans. I wasn’t sure what effect this would have, but I witnessed connections that became more tangible and sensitive. I also witnessed how difficult it was for people to release their body. This is the engine which drives the drama in Tzina — how much can you let go? The more you let go, the more the project will give back.
It surprised me to discover that people in public spaces are open and willing to talk. It surprised me to discover how much comfort I found in a familiar public space, surrounded by strangers that somehow shared my fate. This is an element that I felt was critical for Tzina to work. The comfort in shared loneliness. The pigeons in the square are all, in fact, avatars of viewers that are online and watching Tzina at the same time. If one focuses on a pigeon, one can see where it is from in the world. This is a relatively simple multi-user mechanism, but the emotional effect it has on the square is immense, and fitting when discussing the emotional effect of partnership in loneliness with strangers. When you realize that the pigeons in the square are people too, the space’s proportions can suddenly change. You may suddenly feel small again in comparison to the square, and the experience of following a pigeon around, letting it act as your guide and knowing that it is another user, offers an additional layer of meaning.
Tzina: symphony of longing is still locked with password on http://tzina.space/ — -> feel free to ask for it .
- link for Tzina’s Press-Kit.
I was lucky to work with such an amazing humans, thanks you!
❤ lead developer Avner Peled // design director Ziv Schneider // technical director Or Fleisher // creative coder Laura Chen // script editor Udi Ben-Arie❤
and thanks to the depth kit team for your incredible development and all your support.
