Marionettes in VR

Over the holidays I finally got around to playing with a UE4 plugin I’ve been eyeing for some time: VICO Dynamics. It’s a plugin for all your rope physics needs and is way more powerful and flexible than Unreal’s built-in cable component.

For whatever reason, the first use case that came to mind was playing with marionettes in VR. A day or two later, this is what I had:

I’m still trying to find the best soundtrack to go with this.

Turns out puppet shows in VR are surprisingly fun and hilarious. If you own an Oculus Rift, you need to give this a try. It also works with the Vive, but the controls are not as intuitive.

Download the app here.

I’ve also put up the Unreal 4.14 project on Github. There’s a README in there with all the information you’ll need for running/tweaking it.

Creating the experience

Completing the project took way less time than expected. It pretty much came down to attaching a rope component from each limb of the puppet to a part of the stick you hold. While that seems like the obvious thing to do, I never had much luck doing similar things in the past using UE4’s constraint system and cable component. VICO Dynamics worked great though.

The longest/most annoying part of the process was actually wiring up the hand controllers to make gestures. Since the Oculus Avatar SDK is still not integrated into UE4, I had to cobble together a blueprint and some animations to fake it.

Oculus gestures are rad

I chose the Rift and Touch because I was looking for an opportunity to play around with gestures. The Vive could have provided a more natural feeling of holding the puppet stick, but the Touch gestures ended up adding a next level of fun.

Playing with the furniture is almost more fun than the marionettes.

Being able to poke the marionettes and move their limbs with your virtual index finger was entertaining… but I wanted more. Jon Wade talks about creating natural interactions in his motion controller deep-dive, so I thought I’d try a pinching gesture for grabbing and moving the individual strings.

The pinch gesture is not super obvious, but feels natural when you discover it.

In the initial version, you needed to bring the string tabs back to their original position. Being able to leave them anywhere led to more interesting poses.

More marionettes!

The default UE4 mannequin that I used for the marionette was easy to set up because it had all the proper physics collisions and constraints in place.

I imported a few Mixamo models to test how different skeletal assets would react. Unfortunately, they didn’t come with their own physics assets. I did a quick a job of creating my own but didn’t finesse every single joint.

The results are decent but could definitely be improved further.

“Playing with myself”

Har har.

I’ve been looking for ways to use my 3D scan in VR. This seemed like the perfect opportunity, and the results were… interesting.

I have not included my avatar in the source or downloadable versions. Sorry ;)

Accidental creepiness

All the assets for the marionettes are actually life-size. I scaled them down a bunch so they become little toys.

When I was testing it out, I ran into some performance issues with having more than one marionette in the scene. I thought this might be due to physics issues caused by scaling. To check that possibility, I returned all the marionettes to their full-size. This is what I encountered:

Thankfully I didn’t try this with my own avatar.

In case you are wondering, the performance problem was that rope collisions were on by default. Turning them off did the trick.

Zero-gravity

Pretty much every interaction in VR is made better when gravity is turned off. This was no different.

Lost in space.

Although really cool, the effect wasn’t quite what I hoped for. When you poked the marionette it only moved by a little bit instead of keeping its momentum. Instead of fixing it though, I was drawn to making that effect more pronounced.

I wanted to see what would happen if I increased the linear and angular damping on the marionette so that it would stay rigid unless touched. The idea was to create a new way of posing characters, and it worked surprisingly well:

This technique could have been used in our Thread Studio app.

Shadow puppets!

I left dynamic shadows on because the scene’s complexity was low and I was getting a solid 90fps at all times. While the shadows from the marionettes were pretty cool, it was actually the shadows from my hands that caught my eyes the most.

I kept thinking how fun it would be to use gestures with the Touch controllers to put on a silly shadow puppet show.

Brb while I add networking support…

I’m convinced there’s a way the Touch controllers can generate this pose.

What’s next?

My mind is racing with ideas for all of the awesome things I can now do with strings in VR. I’ll be sure to take a stab at a few in the coming months. It’s really nice knowing that there’s a performant rope physics plugin out there that acts as you expect it to.

If you like these types of posts, be sure to hit the 💚 as it gives me 🔋 to explore and share more quirky VR interactions like these.

If you’re scratching your head at how this relates to Shopify and commerce in VR, don’t worry. You can read about those things over here.