Storytellers United is a diverse and inclusive community of creatives working on innovative forms of storytelling. We connect from across many different time zones and even more backgrounds to share ideas, projects and inspiration. We felt it would be nice to provide space beyond chat channels to have a deeper look at what people in the community are working on. This interview is a first try at a longer format to do just that!
Inhabiting the image: “Captured” — an interactive cinema installation
“Captured is an interactive installation, which captures your face and creates a new identity for you in a collective scenario taking place in the virtual world. Captured is an ongoing research project about inhabiting the image, extending our physical likeness in virtual worlds and simulations.” (Hannah Haaslahti)
I interviewed the team behind Captured, Hanna Haaslahti, Alap Parikh, and Tyler Henry to learn more about storytelling in VR, the technical challenges of real-time 3D face reconstruction and how it is to collaborate across three different continents.
Alap, can you tell us a bit about your recent collaboration with Hanna Haaslahti?
Alap: Sure! Hanna and I met for the first time at CPH:Lab in Copenhagen, after being introduced to each other digitally through a rather extended sequence of connections. We had been in touch over Skype for a few weeks before the meeting, figuring whether this could become a meaningful collaboration — it’s on the right path so far!
The main reason for me to come onboard was that she needed someone capable of understanding the possibilities of the technology she’s interested in using for this project (Captured); these include computer vision and artificial intelligence, mainly, but also the entire range of components it takes to create a virtual world: 3d modeling, animation and rigging, sound score and design, script/story, visual style and so on, and I find myself in a position where my diverse interests and inability to focus and spend all my time on any one area is turning out to be beneficial :)
Tyler and Alap, you are based in New York and Hanna in Helsinki. How did you and Hanna meet and decide to collaborate?
Alap: Actually, I am based in Mumbai now. I studied and worked in New York for three years before deciding to travel and then relocate back to India earlier this year, primarily with the aim of carving out time and space to explore my own creative projects. Since then, I’ve spent my time working on a few different projects (all roughly within the narrative immersive media space), and this lead me to forming a relationship with an experimental media studio in Goa called Quicksand Games Lab.
Here’s where the good sides of living in a globalized world come come in: Hanna reached out to the organizers of CPH:Lab, one of whom had worked with the co-founder of Quicksand, who then reached out to a couple of us who were working with him. I was excited from the beginning: I’m a huge fan of collaborating across cultures, nations, and genders; the project itself piqued my curiosity; and Hanna was just a lovely person to speak to and share ideas with. Things were moving forward before I knew it.
Tyler: I’m located in New York myself, but was in residency in Helsinki collaborating with Hanna on the project through much of 2016–7. Hanna and I first connected in the Spring of 2016, while I was working on my MFA thesis at Parsons in Manhattan — we were introduced by my thesis advisor, in fact, the wonderful Finnish polymath Marko Tandefelt. Hanna and I began the first of our many transatlantic video chats, and we discussed her project ideas involving body scanning, the politics of avatar creation, and generative cinema.
The project was still in concept stage at the time, but Hanna already had a clear installation design, and a narrative in mind — a cyclical reenactment of historical trauma. We looked at historical photographs showing Jews forced to scrub the streets of Vienna by mobs of onlookers and Nazi brownshirts. We talked about the resurgence of fascist politics, and the rituals of discipline and control that permeate our digital and physical bodies and identities. Hanna’s vision was to build an interactive cinema installation, where the visitors would become digital actors, enacting the film that they are watching. Our discussions struck a deep chord with my own thesis work exploring feedback loops of gesture and gaze in cinema through surveillance and machine vision.
I was really excited by the project ideas, and Hanna invited me to work with her in Finland in an initial residency, to develop the avatar generation system and interaction design for real-time body scanning. We found some funding and I ended up flying into a snowy Helsinki on the eve of Trump’s election, which made for a surreal first day to say the least. We spent the winter exploring 3D scanning and interaction algorithms, and developed a first prototype system of a face scanning booth, while working on character rigging and avatar interaction. I flew back to New York in January, in another strange synchronicity, on the eve of Trump’s inauguration. That initial trip was followed by a second residency in Summer 2017 in Helsinki, where we fleshed out the interaction design of the scanning system, eventually building our own high resolution textured face meshes using a stereo machine vision system and morphable face modeling.
Hanna, you started the “Captured” project back in 2016. Can you describe what “Captured” is?
Hanna: Captured is an interactive installation, which captures your face and creates a new identity for you in a collective scenario taking place in the virtual world. Captured is an ongoing research project about inhabiting the image, extending our physical likeness in virtual worlds and simulations. The project explores new narrative formats, hyper-realistic capturing and 3D modeling technologies and their effects and social implications on human relationships.
It seems that the project turned into some sort of platform or framework which lends itself for different narratives. Can you tell us a little bit about the narrative you are currently working on?
Hanna: In a way, this project started for me a long time ago. I saw this documentary photograph in the local newspaper long long time ago depicting so called scrubbing patrols. I cut it out and kept it, because I simply could not understand it. Growing up in the 80´s in a safe and liberal Finnish town, the image conveyed a message from another time, which was reverberating behind the rebuilt facades of postwar Europe. Later, when I became professional artist, this clipping would reappear every now and then and I never knew how to approach it, what to make out of it. In 2015, I was chatting with my friend Marko Tandefelt about depth sensing, SmartBody, ReconstructMe etc. It suddenly became clear to me that scrubbing patrols should be re-enacted as a collective action with 3D Avatars reconstructed from people visiting the installation. The scenario is about collective instincts taking over individual freedom. You are captured as part of virtual community as sometimes we are taken over by mob rule in real life, with or without our consent.
Can you describe the technologies being used in the project?
Hanna: Some of the key collaborators in the project were found by Marko, the first and foremost Tyler Henry, creative coder and artist based in New York. We invited him to Helsinki and started working together the very same day Trump was elected in 2016! At first, we tested all possible depth sensing solutions, from Kinect to Intel RealSense, Orbbec etc. and the results were not good enough for agile, realtime installation environment. We almost gave up, but then Tyler came across 4Dface, an open source 3D face reconstruction and tracking library. It can instantly model quite good faces just from a single 2D image. We decided to focus on the quality of the texture and extended the capture into a stereo camera rig with two 2K computer vision cameras. Tyler developed our application in openFrameworks using the 4Dface/eos library and now we have what we wanted in the beginning, a robust realtime face reconstruction platform. The pipeline was continued in Unity, with an unique face implementation code for avatars created by Thomas Carlsson. All the animations were motion captured in Remedy studios in Espoo by Henri Blåfield and the current fully functional Unity prototype was carried out by Virtual Dawn, a VR/FX company based in Tampere.
Tyler: The development of the scanning and avatar generation system for Captured happened over the course of two collaborative residencies in Helsinki. Hanna and I spent the first exploring the latest 3D reconstruction and depth sensing algorithms and hardware. We built a face scanner prototype system using the new Intel RealSense depth camera, and were able to create some beautiful face scans with custom detail and shading, but we had trouble with the speed of the process.
We began looking at the design solutions of the early portrait photographers, who had the same trouble with exposure times. This archival research led us to explore the history of tintamarresque photo sets (“face in the hole” photo backdrops), and we built one in Hanna’s studio to start our second residency. At the same time, I was researching the latest open source software in the field of 3D morphable modelling, the technology behind many new commercial face recognition and animation applications. I began experimenting with the “eos” C++ face modeling library from Patrik Huber at the University of Surrey, which uses statistical analysis to estimate face shapes and expressions in real time from a stream of 2D images.
From this research we evolved our prototype, focusing on capturing high resolution video textures, and 3D face landmark data through a stereo pair of machine vision cameras. We went through many iterations and fine tuned the algorithms and interaction design through constant user testing. While developing the scanner, I was working closely with Thomas Carlsson who was simultaneously developing the character rigging and choreography for the cinematic Unity scene. Together we integrated our code to connect the scanner as an avatar/character creation engine in the overall system. The stream of face scans became like a collection of dramatic masks, worn and interpreted by the avatars in the scene.
I saw some scenes on video. The aesthetics are enigmatic! The digital double of the participant has a face that is hyper realistic in texture and shape but also totally stoical and motionless (no blinking, no facial expressions), while the body is active and moving around following the script of the narrative. This creates the impression that the participant passively watches herself “getting played”. What does that mean for the “agency” of the participant in the installation?
Hanna: What is your relationship to yourself, where are the boundaries between you and your community, what is communality in individuality? Those are the central questions posed by the installation. The agency is taken away from you and to me that is interaction in the most contemporary sense, since we are living in the era of hyperobjects, neural networks, the third mass extinction, offshore plastic accumulation zones… “The user” is not anymore in the center of perspective and the projections are not moving magically according to his will. But his image is traveling, mutating and multiplying with other images in the cyberspace, like a butterfly wing stroke is causing storm in the other side of the planet. Or perhaps more timely analogy would be a plastic water bottle thrown out of the car window in Lapland and ending up in the Great Pacific garbage patch!
For the motion capture I worked with two great finnish actors, Marc Gassot and Vilma Melasniemi. Our aim was to create expressive body language to compensate for the absence of spoken language. The bodies and the gestures do the talking, while the faces are like masks, mute and motionless.
What is special about storytelling in VR/AR?
Hanna: It is more about creating a world and modeling a visit to that particular story world than telling a story. I’m especially interested in how you are embodied in VR, are you just eyes in the void or are you also seen, having some kind of a presence or agency. Having worked many years with various forms of direct interaction, I feel that those ideas have not yet reached VR/AR.
As a technology, contemporary VR/AR reminds me about a particular momentum in the history of photograph, how it suddenly brought exotic and distant places right under your nose, allowing you to travel around the world and further with your gaze. A mind’s eye went exploring inside a photograph, but with VR/AR you literally step inside virtual reality and be there, it´s a full body experience. Hopefully the invading, colonial gaze unleashed by photograph would be tamed in powerful political gestures in VR/AR. We need biospherical thinking, worlds that see you, connect you and make you responsible.
Alap: It struck me recently — and I hope I’m able to phrase it well — that these mediums are the manifestation of what we have learned about the world through modern physics in recent decades. Just as the observer/scientist affects the object being observed by the very act of observing, VR/AR allows us the opportunity, in a very direct manner, to allow the participant to form a truly unique bond with the artwork and change it by the very act of engaging with it. And so what’s intriguing to me here is the need to step away from traditional narratives and create a world that is alive to and aware of the participant in the same way that the world around us is.
I noticed that all of you are artists who have a strong creative vision for telling stories with immersive technologies, and at the same time don’t shy away from doing the actual technical development. I find this really exciting, as most of the time I notice a division of labour between the “designer/visionary” and the “techie”. Did you have those distinct roles or task areas in the project, and if yes, how did you decide upon these roles?
Hanna: As an artist, I come up with an idea and then I’m motivated to share and realize that idea. If my idea is good, people get interested about it and want to collaborate with me. Also if my idea is challenging and crazy enough I get to work with really brilliant people like Remedy/Henri, Thomas, Tyler, Alap…allthough they of course have a particular job/task to complete in the project, they participate as people, not just as programmers waiting for exact tech specs to complete.
Realtime 3D face reconstruction is extremely difficult computing challenge, few years back it was literally impossible to do. Now there are few companies, who develop the tech, but with very uninteresting content. On top of that, we have a narrative simulation with AI controlled agents. So there many exciting and fun emerging technologies to work with! Since I approach the tech from the necessities dictated by a story, which needs to be articulated, I’m not just motivated by technical challenges. To achieve this, you need people with different backgrounds and expertise to come together and share their world.
Alap: We do have distinct roles, but open dialogue. I agree with Hanna when she says that these immersive artworks require people to come together, inspire each other, and share all of their knowledge such that it becomes a dialogue, and such that the project itself grows and shapes itself via this dialogue. This flowing communication requires more of us to step out and really listen to and understand what is being said, letting go of our perspectives temporarily. The ability to have this communication is invaluable and so difficult because we all think in our own languages — cultural, technical, etc.
Unfortunately, I think a lot of how the world is structured right now encourages this paradigm of specialization and division, where the “techie” does not play enough, and the “visionary” tends not to be grounded enough. Although it is definitely important — and efficient in most cases — particularly within the realm of storytelling the contribution needs to come from every involved member. That, I think, makes for true collaboration. I think the future lies in more holistic collaboration and the blurring of more boundaries in the creative process.
The project is in active development. I am excited about the next steps! Can you tell us about your next milestone?
Hanna: The next step is to take the prototype to the final work together with Tyler, Alap and Quicksand! That includes transforming the scrubbing scene into full fledged simulation with AI controlled agents by implementing a FSM (Finite State Machine) as their brain. By combining narrative with simulation, we are hoping to create a story world that evolves by itself, but with some degree of determination.
I’m also preparing about the next narrative scenario for Captured. I will spend couple of months next year in New York, researching the swing dancing history of Harlem, especially social dancing scene of 1920´s and 1930´s. That will be the world of Captured II.
Thank you all for the interview!
Hanna Haaslahti is a media artist working with image and interaction based in Helsinki, Finland. Her tool is computer vision and she’s interested in how machine shapes social relations. Her artworks generate new narrative interaction between people, she explores possibilities of human-machine-human collaboration. Scientific research on perception, vision and AI inspires her, likewise everyday observations about human life in the age of machines.
Alap Parikh is an artist and developer focusing on immersive experiences. He works extensively in augmented and virtual reality, dissolving gaps between the orthodox categories of art and expertise. He’s fascinated by stories that explore the peculiarities of reality and human perception, and is currently interested in the shape of narrative in participatory mediums of art.
Tyler Henry is an artist and creative coder based in New York, working in interactive installation and performance. His practice involves building multisensory spaces that integrate people, narrative and technology. He is interested by the interfaces between bodies, and environments, and by cultural archaeologies of images and myth. His work bridges disciplines of art, design and engineering and is naturally collaborative — so working with Hanna and now Alap has been a dream.
Project Blog — http://cosmeticspace.tumblr.com/
Website of CPH:DOX — https://cphdox.dk/en/industry/training/cphlab/projects/
Storytellers United — http://storytellers.link/
