Behind the scenes of a real-time animated VR music video
By Flipside XR and DEL Records
I was at VidCon sitting on the couches on the fourth floor of the Anaheim Convention Center with a group of strangers talking about work. Due to some cosmic error with VidCon’s mobile app, we all ended up meeting by chance. That’s where I met Bruce, Eddie, and Alejandro from DEL Records, an entertainment company focused on regional Mexican music.
I showed Bruce our app and what our users were making with it. He and his team immediately wanted to make something with us. Serendipity wins again!
We make Flipside Studio. Think of it as a television studio for live animated content all within virtual reality. DEL Records wanted to create an animated music video for the release of T3R Elemento’s new single En Menos De Un Minuto.
We had four weeks to complete this project from concept to delivery. DEL Records provided us with high-level direction and trusted us to come up with a concept and execution of the video. Once we completed our portion of the project, DEL Records finished the process with final edits and the addition of visual effects to bring things in line with their other marketing materials. We collaborated throughout the four weeks of productions.
The process was very much like any video production except that instead of real locations we built virtual ones, and instead of filming real actors we created 3D models that were animated by puppeteers in Flipside Studio.
This process is called motion capture and Flipside Studio uses the Oculus Rift and HTC Vive as motion capture hardware.
In certain cases, our technology did not provide the capture quality we were looking for in the character animations. To solve this, we invented new methods to improve the capture process that allow character animations that are common in video games to be combined with the movements the puppeteers were making. We call these hybrid animations.
The top half of the character was controlled by a puppeteer and the bottom half was controlled by a walk cycle animation. This improved the quality substantially. In some cases, we needed one of the characters to be controlled by a pre-recorded animation that we downloaded from Mixamo while the other characters in the scene were controlled by a puppeteer.
Jim Henson and all the puppeteer pioneers before him figured out that hand controlling a puppets mouth can be a very effective way to perform lip syncing. We adapted these ideas to Flipside to allow the mouths of our characters to be controlled by the joysticks on the virtual reality controllers. Even though Flipside has an excellent automatic lip sync system, controlling the mouth with the joystick works better for lip syncing to music.
There were many other steps in our process. We used a combination of Unity3D, Flipside Studio and After Effects to produce the video. The process pushed us to solve new problems and think about virtual production in a new way.
After all is said and done we are proud of the video we produced with DEL Records and it’s a satisfying feeling knowing that millions of people have viewed our work that we produced with an app that we built.
Thanks you to our team of amazing programmers, puppeteers, and artists who made it all happen and thank you to DEL Records for believing in us and Flipside Studio.
