Flickers of Light in Dark Times — Motion Capture Live Performance

Man-Ping Wu
Feb 7 · 12 min read

A mocap beginner’s guide for putting on a live performance. 🤔🤔🤔

Performance at NYU Tandon — Photo taken by Jude Zhu

Overview

We were a team of four of New York University (NYU) Integrated Digital Media (IDM) and Interactive Telecommunications Program (ITP) students grouped together as Team Ono to showcase a live motion dance capture performance.

The Premise

Flickers of Light in Dark Times is a live motion capture dance performance about getting crushed by the adversity in urban life, and regaining strength by accepting kindness and support.

The piece plays with notions of the visible and invisible structures that surrounds us in urban life. The tracked dancer navigates the spaces in between the audience members, vexing from completely ignoring them, and acknowledging their presence as a source of strength. The audience members takes an active role as they offer her support through motion tracked wisps that rattle and sprout lights in the darkness.

With Flickers of Light in Dark Times, the artists create an immersive live experience through movement, projected interactive 3D graphics and music, crafting a cohesive narrative with cutting edge technology.

Film created by Louise Lessel

Team Ono and Timeline

Once we were assigned into groups as Team Ono, we had 1.5 months time span working together until the performance on December 14th. The team was named after Yoko Ono a multimedia artist who is known for her artist performance. The first half of the semester was focused on understanding the basics of motion capture and integrating three different softwares — OptiTrack Motive, Unreal engine(UE4) and Maya. The second half was focused on a the class final assignment, which is to create a live motion capture performance.

In our first meeting, we brainstormed ideas and discussed how our skill sets can contribute to this piece in UE4. We realised there were obstacles of assigning roles when everyone has diverse backgrounds and working together— Nancy and I (Man-Ping) are IDM graduate students with UX and graphic design backgrounds, Chester is an IDM undergraduate student who is a 3D modeller and Louise, an ITP graduate student, is an experienced interaction designer who is proficient in programming, installations, and Unity. We naturally leaned towards Louise as the leader as she has the most experience in the field, however we were all determined to take on the challenge and gather all the needed hands for the project.

Nancy Chiu — Technical Director

Louise Lessel — Director of Mocap and Tech Integration

Chester Ma — Director of Graphics and Model Design

Man-Ping Wu — Art Director.

In addition, we partnered up with Aaron Moreno, a Sound Designer from ITP, and Katie Brady, the Performer from Barnard College.

In preparation for the show — Photo taken by Chester Ma

Tasks at hand

With a big task at hand, we split our responsibilities into different aspects of the piece. Louise was in charge of being the project manager with handling the macro tasks with the technology, world integration and communicating with Aaron and Katie. Chester was in charge of applying the particles on to the dancer. Nancy was responsible for three wall interactions and reaction setups. My tasks (Man-Ping) involved being responsible for the promotion materials, rigid bodies and wand particles.

Aaron helped by incorporating sound design that allows the performance space to become an immersive environment and Katie was responsible for the choreography.

Team Ono Slack Channel Conversation

Storyboard

Our piece’s premise is to demonstrate one getting crushed by the adversity in urban life, and regaining strength by accepting kindness and support. The storyboard went through multiple alterations after gaining feedback from our professors and industry professors tweaking the piece into something more cohesive and fascinating to the audience members. Little details were asked and explained to test whether or not our core message would come across.

Final Storyboard — Drawn by Louise Lessel

Mood board and Ideation

During the process of storyboarding, we gathered and compared our individual mood boards to see if we all had a collective art direction for our projection design. We all collectively agreed to work towards a soft floating-like dim water particle effect to outline for the particles of the dancer and wand particles. In addition we wanted other visual elements to be simple like the white vector outlined walls and black background, giving a better contrast between the different elements and avoiding complicated visuals.

Pinterest boards in the works

Process

The setup and the process for the live performance through UE4 has its ups and downs. Our professors, Todd and Kat, even nicknamed us as the Team Oh No! as there were many moments when one variable was working and the next moment it crashed. We kept in mind about the potentials of both software and human error, so we have backups and saved often. We updated each other by the minute with how things were going with the task at hand and what we needed to complete.

Every little fix is a BIG achievement in the world in UE4

Environmental Setup

Louise was in charge of the full integration for the digital and physical space, and making sure details like the map level environment, audio setup (with Aaron), sequencing and equipment had an orderly set up. During the performance, she was on the monitors making sure the system was debugged and virtual cameras were working.

Set up of the Map Level with the Dancer, Cameras and Walls.
Integration of all the Events using Open Level Blueprint — Like a linking visual code with nodes and objects.
Louise had the sequencer ready, so elements are ready to be perform at the right time.
Making sure the floor projection is working— Photo taken by Chester Ma

Wall Setup

Nancy was in charge of the walls with trial and error. Todd was readily available on slack and office hours when she needed the help. Factors to consider as a successful interaction are the wall collision, live motion capture effects and ray casting. The walls interaction requirement were collision and the dancer’s ability to interact through ray casting.

Sequencing required wall actions

  1. Bricks of the wall spawn one at a time, and one row at a time, from the left wall to the middle, then to the right wall.
  2. Once all 3 walls are formed, bricks start moving in and out organically
    Bricks should “ripple“ as dancer pushes the walls.
  3. Rows of bricks will start pushing in towards the center (dancer) from above.
  4. Walls fall after dancer gained strength and particles from the audience members Particle Dancer Setup.
Failed attempt wall transitioned from her local file to shared team UE4 scene

Dancer Particles

Chester was responsible for the dancer particles with the Niagara VFX system. It’s a tool where users can create and adjust particle systems and effects on objects. We were taught about this system in class, whereas in previous versions particles were created and edited using Cascade. We wanted our particles to be consistent of soft floating minimal particles to both the dancer and the wand particles and used Matt DesLausier’s reflective series as a reference point.

We spent a lot of time tweaking the softness of the particles with the challenges of extreme colors and high opacity. With some tweaks by changing the color and decreasing the opacity, we were able to capture the delicate aesthetic we wanted to portray.

Final particle effect shown on screen — Photo taken by Chester Ma

Wand Particles Setup

My primary responsibilities were the rigid bodies (wands), wand particles and the promotional materials for the show.

Rigid Bodies
Rigid bodies are objects with three markers that can create movement or be another camera in the digital scenery. We were interested in creating wands to give the audience members an opportunity to interact in the scene and connect with the dancer. It was an obstacle coming up with a plan to make it as we wanted to create an organic magical like object. There were discussions with using a 3D printer or wood to create, but the problem came around to spend the time and money when there were ready available rigid bodies in the stock room.

After some thinking, I realized from my graphic design background that thick 100lb paper stock will do the trick. It is easier to manipulate and strong enough to withstand the performance. Many mockups later we created three circular wands and a backup. We placed three velcro pieces on the wands for the markers to be placed on.

Final “Wand” Rigid Bodies

Wand Particles
The particles were created with Niagara system plugin. There were some issues with the plugin being a bit unpredictable and how I handled it.

Wand Rigid Bodies with Point Force Attraction in Open Level Blueprint
Niagara System — with the emitters and systems

Promotional Materials
In addition to creating the live performance, we had to create promotional materials with a teaser video and poster to promote our live performances.

It was a tight deadline within 2 weeks into the project to have a teaser video ready. Despite not having a setup ready, it did helped speed the creative direction.

Final Poster Design

Sound Design and Choreography Setup

Louise brought along Aaron and Katie to help out for the piece, as we realized elements of sound and performance are the missing key components for this piece. We may not see each other in every meeting, but when we meet up for the occasional rehearsal at the black box, it was a magical experience bringing all the elements together. We provided feedback to one another and discussed how it could be further improved.

Katie our dancer — Photo taken by Chester

Integration

Since this big project, involving two instructors and four team members working on the project, we decided to use SVN for version control.That way everyone had access UE4 project and could work simultaneously. We gave each other heads up if one of us check out one of the levels to avoid conflict files. Thanks to our professor Todd for lending us his system. 🙏🏻🙏🏻🙏🏻

Using SVN as a Google-Drive-Alike but for big projects

Preparation

Stage Setup

The IDM department was moved into a new building and we had to set up in one of the working NYU IDM labs. Setting up the stage was a monumental task that required the whole class on a Thursday night. All of equipment setup like floor padding, monitors, projection screens, cables and more was quite daunting as the equipment’s value ranges in the thousands of dollars. Everyone had to be careful, while working very quickly, or be known as the person who broke expensive equipment. The macro tasks with setting up the big pieces was done quicker than expected by micro tasks like organizing the cables, safety check, camera setup and sensors, which took up the majority of the time.

The setup was Thursday night and the live performance was on Saturday. We had to juggle with the time limitations for rehearsals and testing any technical issues. In addition, everyone’s main project was still a work in progress.

For our performance setup, thanks to Louise, we had projection of interactive 3D graphics and 4-channel audio.

Original Space
Stage setup- Photo taken by Kiki Wu

We did an entire system integration with a spout plugin to connect monitors to the projectors and Optitrack cameras to UE4.

Rehearsals and Performances

For each group performance, we all had two runs through to give everyone a chance to fix any minor mishaps and document the process. We had three spots for rehearsals in specific time slots. During those times, we realized the main project was giving us some hiccups. One of them was when we imported the project through spout into another computer that lead to some potential bugs. It created some inefficiency with our rehearsal time as we had to delay it in order to fix these issues. Despite all of that these rehearsals helped us being adjusted to the system and catch some details before we even realized it.

Minor Mishap

During the performance there was a minor mishap with the wand particles not showing up in the UE4 play through and being reflected on screen. We did notice the trouble through rehearsals as we were attempting to have the particles show up and the rigid bodies create some movement for the cameras to capture. Prior to the performance, we attempted to fix it through our local individual laptops with multiple playbacks and added key press when the worse comes to worst. This was happening while the other groups were rehearsing on the main stage. Despite the two run-throughs and all attempts to fix, sadly it didn’t showed up.

Response

Despite the minor mishap, the audience members response was extremely positive and supportive, which was encouraging to see. It showed despite some parts may not be working, the audience members will not really realize the difference. The performance was thought-provoking and combined our ideas of the physical and digital realities. Both elements being a critical for the piece.

Performance at NYU Tandon — Photo taken by Jude Zhu

Next Step

Where would you take this project if you had more time/resources?

If we had more opportunities to rehearse and debugging the new setup it will be fully appreciated, as we were limited in time and space to allow other groups to rehearse. The time crunch for the space being set up two days before the final performance was intimidating and daunting, while handling our projects at the same time. The new setup can easily cause some errors by human error, since Optitrack and UE4 are sensitive software. It would had been better if we had performed in a space where the system was already setup for a period of time, instead of building a new one from scratch. More time also means prevention of the wand particles mishap, if we had an extra week setting the stage and do more testing with the system.

Despite the errors that were caused and only having 1.5 months to create the entire piece, I would say it was a success for a group of mocap beginners. Creatives can be one of the harshest critics of their work, but not everything can be spotless and goes without complications. The reception itself is already proof of our accomplishment and the experience behind it made me appreciate process, the people I’d worked with and solutions behind making the piece come to life.

Photo taken by Jude Zhu

I want to first give thanks to our professors Todd Bryant and Kat Sullivan for the endless help and assistance during the late nights and early mornings. Next Aaron Moreno, and Katie Brady whose contributions were extremely valuable and lending their time to bring this piece to life. Lastly to Team Ono with Nancy Chiu, Louise Lessel and Chester Ma who made sure everyone was putting their best foot forward and motivating each other to make the best motion capture collaboration. Those all nighters and encouragements made it all worth it in the end.

Man-Ping Wu

Written by

A NYU IDM grad student, who happens to be an interdisciplinary designer.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade