Marionettes in VR
Over the holidays I finally got around to playing with a UE4 plugin I’ve been eyeing for some time: VICO Dynamics. It’s a plugin for all your rope physics needs and is way more powerful and flexible than Unreal’s built-in cable component.
For whatever reason, the first use case that came to mind was playing with marionettes in VR. A day or two later, this is what I had:
Turns out puppet shows in VR are surprisingly fun and hilarious. If you own an Oculus Rift, you need to give this a try. It also works with the Vive, but the controls are not as intuitive.
I’ve also put up the Unreal 4.14 project on Github. There’s a README in there with all the information you’ll need for running/tweaking it.
Creating the experience
Completing the project took way less time than expected. It pretty much came down to attaching a rope component from each limb of the puppet to a part of the stick you hold. While that seems like the obvious thing to do, I never had much luck doing similar things in the past using UE4’s constraint system and cable component. VICO Dynamics worked great though.
The longest/most annoying part of the process was actually wiring up the hand controllers to make gestures. Since the Oculus Avatar SDK is still not integrated into UE4, I had to cobble together a blueprint and some animations to fake it.
Oculus gestures are rad
I chose the Rift and Touch because I was looking for an opportunity to play around with gestures. The Vive could have provided a more natural feeling of holding the puppet stick, but the Touch gestures ended up adding a next level of fun.
Being able to poke the marionettes and move their limbs with your virtual index finger was entertaining… but I wanted more. Jon Wade talks about creating natural interactions in his motion controller deep-dive, so I thought I’d try a pinching gesture for grabbing and moving the individual strings.
In the initial version, you needed to bring the string tabs back to their original position. Being able to leave them anywhere led to more interesting poses.
More marionettes!
The default UE4 mannequin that I used for the marionette was easy to set up because it had all the proper physics collisions and constraints in place.
I imported a few Mixamo models to test how different skeletal assets would react. Unfortunately, they didn’t come with their own physics assets. I did a quick a job of creating my own but didn’t finesse every single joint.
“Playing with myself”
Har har.
I’ve been looking for ways to use my 3D scan in VR. This seemed like the perfect opportunity, and the results were… interesting.
Accidental creepiness
All the assets for the marionettes are actually life-size. I scaled them down a bunch so they become little toys.
When I was testing it out, I ran into some performance issues with having more than one marionette in the scene. I thought this might be due to physics issues caused by scaling. To check that possibility, I returned all the marionettes to their full-size. This is what I encountered:
In case you are wondering, the performance problem was that rope collisions were on by default. Turning them off did the trick.
Zero-gravity
Pretty much every interaction in VR is made better when gravity is turned off. This was no different.
Although really cool, the effect wasn’t quite what I hoped for. When you poked the marionette it only moved by a little bit instead of keeping its momentum. Instead of fixing it though, I was drawn to making that effect more pronounced.
I wanted to see what would happen if I increased the linear and angular damping on the marionette so that it would stay rigid unless touched. The idea was to create a new way of posing characters, and it worked surprisingly well:
Shadow puppets!
I left dynamic shadows on because the scene’s complexity was low and I was getting a solid 90fps at all times. While the shadows from the marionettes were pretty cool, it was actually the shadows from my hands that caught my eyes the most.
I kept thinking how fun it would be to use gestures with the Touch controllers to put on a silly shadow puppet show.
What’s next?
My mind is racing with ideas for all of the awesome things I can now do with strings in VR. I’ll be sure to take a stab at a few in the coming months. It’s really nice knowing that there’s a performant rope physics plugin out there that acts as you expect it to.
If you like these types of posts, be sure to hit the 💚 as it gives me 🔋 to explore and share more quirky VR interactions like these.
If you’re scratching your head at how this relates to Shopify and commerce in VR, don’t worry. You can read about those things over here.
