Inhabiting the image: “Captured” — an interactive cinema installation

Alap, can you tell us a bit about your recent collaboration with Hanna Haaslahti?

Alap: Sure! Hanna and I met for the first time at CPH:Lab in Copenhagen, after being introduced to each other digitally through a rather extended sequence of connections. We had been in touch over Skype for a few weeks before the meeting, figuring whether this could become a meaningful collaboration — it’s on the right path so far!

Tyler and Alap, you are based in New York and Hanna in Helsinki. How did you and Hanna meet and decide to collaborate?

Alap: Actually, I am based in Mumbai now. I studied and worked in New York for three years before deciding to travel and then relocate back to India earlier this year, primarily with the aim of carving out time and space to explore my own creative projects. Since then, I’ve spent my time working on a few different projects (all roughly within the narrative immersive media space), and this lead me to forming a relationship with an experimental media studio in Goa called Quicksand Games Lab.

Hanna, you started the “Captured” project back in 2016. Can you describe what “Captured” is?

Hanna: Captured is an interactive installation, which captures your face and creates a new identity for you in a collective scenario taking place in the virtual world. Captured is an ongoing research project about inhabiting the image, extending our physical likeness in virtual worlds and simulations. The project explores new narrative formats, hyper-realistic capturing and 3D modeling technologies and their effects and social implications on human relationships.

It seems that the project turned into some sort of platform or framework which lends itself for different narratives. Can you tell us a little bit about the narrative you are currently working on?

Hanna: In a way, this project started for me a long time ago. I saw this documentary photograph in the local newspaper long long time ago depicting so called scrubbing patrols. I cut it out and kept it, because I simply could not understand it. Growing up in the 80´s in a safe and liberal Finnish town, the image conveyed a message from another time, which was reverberating behind the rebuilt facades of postwar Europe. Later, when I became professional artist, this clipping would reappear every now and then and I never knew how to approach it, what to make out of it. In 2015, I was chatting with my friend Marko Tandefelt about depth sensing, SmartBody, ReconstructMe etc. It suddenly became clear to me that scrubbing patrols should be re-enacted as a collective action with 3D Avatars reconstructed from people visiting the installation. The scenario is about collective instincts taking over individual freedom. You are captured as part of virtual community as sometimes we are taken over by mob rule in real life, with or without our consent.

Jews in Vienna forced to scrub street pavements, directly after Austria’s annexation to the German Reich. Photographed on March 13 or 14, 1938. Votava Foto.

Can you describe the technologies being used in the project?

Hanna: Some of the key collaborators in the project were found by Marko, the first and foremost Tyler Henry, creative coder and artist based in New York. We invited him to Helsinki and started working together the very same day Trump was elected in 2016! At first, we tested all possible depth sensing solutions, from Kinect to Intel RealSense, Orbbec etc. and the results were not good enough for agile, realtime installation environment. We almost gave up, but then Tyler came across 4Dface, an open source 3D face reconstruction and tracking library. It can instantly model quite good faces just from a single 2D image. We decided to focus on the quality of the texture and extended the capture into a stereo camera rig with two 2K computer vision cameras. Tyler developed our application in openFrameworks using the 4Dface/eos library and now we have what we wanted in the beginning, a robust realtime face reconstruction platform. The pipeline was continued in Unity, with an unique face implementation code for avatars created by Thomas Carlsson. All the animations were motion captured in Remedy studios in Espoo by Henri Blåfield and the current fully functional Unity prototype was carried out by Virtual Dawn, a VR/FX company based in Tampere.

Screenshot form 4Dface, 3D reconstruction mesh on 2D image

I saw some scenes on video. The aesthetics are enigmatic! The digital double of the participant has a face that is hyper realistic in texture and shape but also totally stoical and motionless (no blinking, no facial expressions), while the body is active and moving around following the script of the narrative. This creates the impression that the participant passively watches herself “getting played”. What does that mean for the “agency” of the participant in the installation?

Hanna: What is your relationship to yourself, where are the boundaries between you and your community, what is communality in individuality? Those are the central questions posed by the installation. The agency is taken away from you and to me that is interaction in the most contemporary sense, since we are living in the era of hyperobjects, neural networks, the third mass extinction, offshore plastic accumulation zones… “The user” is not anymore in the center of perspective and the projections are not moving magically according to his will. But his image is traveling, mutating and multiplying with other images in the cyberspace, like a butterfly wing stroke is causing storm in the other side of the planet. Or perhaps more timely analogy would be a plastic water bottle thrown out of the car window in Lapland and ending up in the Great Pacific garbage patch!

Screenshot from Optitrack, motion capture at Remedy Studios with Marc and Vilma.

What is special about storytelling in VR/AR?

Hanna: It is more about creating a world and modeling a visit to that particular story world than telling a story. I’m especially interested in how you are embodied in VR, are you just eyes in the void or are you also seen, having some kind of a presence or agency. Having worked many years with various forms of direct interaction, I feel that those ideas have not yet reached VR/AR.

I noticed that all of you are artists who have a strong creative vision for telling stories with immersive technologies, and at the same time don’t shy away from doing the actual technical development. I find this really exciting, as most of the time I notice a division of labour between the “designer/visionary” and the “techie”. Did you have those distinct roles or task areas in the project, and if yes, how did you decide upon these roles?

Hanna: As an artist, I come up with an idea and then I’m motivated to share and realize that idea. If my idea is good, people get interested about it and want to collaborate with me. Also if my idea is challenging and crazy enough I get to work with really brilliant people like Remedy/Henri, Thomas, Tyler, Alap…allthough they of course have a particular job/task to complete in the project, they participate as people, not just as programmers waiting for exact tech specs to complete.

4K stereo camera rig at the studio, 2018.

The project is in active development. I am excited about the next steps! Can you tell us about your next milestone?

Hanna: The next step is to take the prototype to the final work together with Tyler, Alap and Quicksand! That includes transforming the scrubbing scene into full fledged simulation with AI controlled agents by implementing a FSM (Finite State Machine) as their brain. By combining narrative with simulation, we are hoping to create a story world that evolves by itself, but with some degree of determination.

Thank you all for the interview!

Hanna Haaslahti is a media artist working with image and interaction based in Helsinki, Finland. Her tool is computer vision and she’s interested in how machine shapes social relations. Her artworks generate new narrative interaction between people, she explores possibilities of human-machine-human collaboration. Scientific research on perception, vision and AI inspires her, likewise everyday observations about human life in the age of machines.

Join the conversation via http://storytellers.link/

--

--

--

A creative community interested in innovative forms of storytelling

Recommended from Medium

Art of Creativity Awards 2017 — AOC

AOC Awards – hand made kaleidoscope, photography by Kathleen Collins.

The Makers Exhibition: A MakersPlace Ambassador Showcase

Space GOATs on Acid travelling through the SOL-ar system

[Artist Buzzed] Yvette Yunyoung Choi

José Delbo Virtual Scavenger Hunt — Rules & Hints

READ/DOWNLOAD#) Landscapes in Oil: A Contemporary

Upcoming Drops — Andrew Ferrer, Vache Morte and Javier Arrés x Katy Arrington

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alexa Steinbrück

Alexa Steinbrück

A mix of Frontend Development, Machine Learning, Musings about Creative AI and more

More from Medium

Augmented Reality with OpenCV

3D Object Detection for Mobile AR using MediaPipe and WebXR

Custom TensorFlow Lite vs FaceNet

Faces: AI Blitz XIII with Team GLaDOS