How to do Cloth Simulation on Virtual Clothing in Lens Studio
Hello Everyone, This is My First Blog Ever. Haven’t been a writer but I want to document this process so that I can provide the help I needed a while back. The Cloth Simulation Template from Lens Studio doesn’t have many resources to refer to when it comes to Virtual Fashion. When I stumbled upon this problem I was kind of noobing out and looking for some tutorial or something I could use for help. But Sadly, No Luck. So After several hours of hit and trial, I started to feel I was getting somewhere in the right direction. And This is what I managed to make.
Well right after this I was feeling like a hot air balloon on top of the sky (very clear with that insomniac celebration) but I had to test this breakthrough once again. Honestly, I was afraid at first because if it didn’t work it’d bring me back to square one. And I would be lying if I said I just got it working easily with my second try 🥲. It took me another couple of hours to get it working again on this new cloth I modelled. Now that it worked I could document the process with an experience of 2 whole lenses, lol. Enough with chatter let’s get started.
In this tutorial, we will use the Jacket Example Object as our base to build our piece of clothing in place of that.
1. Clear Your Workspace:
Open the cloth simulation template in LensStudio. Clear the Object Hierarchy as shown below. I will explain why did we do this later in the tutorial.
2. Modelling:
There are plenty of ways in which you can model or design a piece of clothing so I will just cover what one should take care of while working with Cloth Simulation. When modelling for cloth simulation template, Lens Studio documentation recommends a mesh of a maximum of 3000 triangles. But the best-looking cloth mesh I could come up with low poly modelling was 9k+ triangles XD. To my surprise, it just worked fine. But yes it does have a frame drop in the experience if the mesh has too many faces to calculate the simulation on.
Well to say briefly about my model when I was modelling a kurta for the tutorial I eventually chose to go with the Legendary Akatsuki Cloak from Naruto. So pulled a reference as an image and then used faces of the Reference Model as a base and modelled a decent cloak using Blender. Here‘s how it looks.
3. Vertex painting:
Vertex painting is simply as its name suggests assigning colors to each vertex in the mesh. This step in our tutorial is the most important and more than that it is the biggest pain in the A :’). Well for me it was more than a task to actually nail a vertex painting that does what I expect of it. In the Cloth Simulation template, vertex painting is mainly responsible for Holding parts of the mesh, like you would hold a hanging cloth in your hand. It keeps the vertices stable concerning the bind points you assign the color to. To be more clear on my previous statements you can watch a short tutorial by Kavin Kumar on vertex painting.
In the Cloth Simulation template (on LensStudio v.4.10+) you are allowed to have multiple vertex colors following different objects. These objects are called bind points. If you see your object hierarchy there are a few more sub-hierarchies of objects, one of which is BindPoints Object.
We will look into this object hierarchy in more detail in the next step. We shall use these bind points as references while painting our vertices. Look into your resource panel under 3d Body Tracking Resources you will find an fbx called bindPoints you can use this object in your preferred 3d program as a reference. You should save your project to locate this bindPoints object from Blender (my preferred 3d program). Open the file location of your project and import the bindPoints.fbx
Now we can refer to the bind points while vertex painting. Vertex Colors used by lens studio may not strictly follow gradients of the color given to regions because it works for the exact assigned color for the vertices that connect to make a polygonal face :) (You can see that I tried gradients but didn’t work any better).
Use different combinations of RGB colors to assign for different bindPoints. Using contrasting colors in nearby regions may help in distinguishing better. Note down your colors and their RGB values with each color in a range of 0–255. If your program shows RGB is in the 0–1 range convert to 0–255. For eg. i.e. (0.5, 0, 1) is converted to (128, 0, 255). Note: You can choose to remember hex codes as well but I preferred RGB values as I assigned colors using simple fractions of 1.
The bindPoints each handle the cloth simulation by holding the colored vertices with the specified offset from itself. My preferred way of vertex painting is painting the whole mesh black and then over black I put the rest vertex colors on that as required and use the color black again to erase. I chose black as it’s easiest to choose the value(0,0,0). The Main Point of coloring vertices is deciding which parts of the mesh you want to be less affected by the simulation physics. In simple terms think the parts I left black will be the part that will have most of the cloth simulation physics.
This is the vertex painting of The Akatsuki Cloak. There is no specific pattern you need to paint in. Just think like the parts that need cloth physics should be kept black. For my case I wanted cloth physics under the belt to behave like a frock and top-shoulder/underarms to belly for any bends I do with my arms.
Once you’re done with the vertex painting exporting the mesh has no special steps. Just select your cloth mesh and export the selected mesh as .fbx or .gltf. Keep your clothing as one single mesh because one script can handle only one mesh data.
4. Lens Studio Setup:
Now that we have vertex painted our model it’s time to import it into your Cloth Simulation Project. Once it’s in your resources panel you just have to plug things into the script and configure the cloth simulation script according to your vertex colors.
To start with it, click on the Cloth Simulation Jacket [EDIT_ME] object in the jacket example. In the Resources panel, expand the fbx you imported and drag the Mesh from the Meshes Folder to the Cloth Simulation Script.
The next step is to plug your vertex colors and the bindpoints that hold them.
💭 Remember we cleared our workspace in the 0th step here’s why it’s important. If you connect your mesh under jacket example and a bindPoint of parts from some other cloth example it will show you weird problems like your clothing gets hung to some distant point or gets stretched and broken. To eliminate these problems caused by a mismatch of inputs, we keep only those bind points and Collider Guides that are under the example we are going to use. This exact issue of mismatch took me hours to figure out. This problem was 85% of my problem in working on this project. Even after being aware of this problem, I tried to be careful during my second try, and yet I faced it, even after plugging inputs carefully. I cleared my workspace and I could make it work sooner without silly mistakes.
The Jacket example contains only 3 bind points with vertex colors but you can add the rest bind points and vertex colors as per your requirements. Keep an eye on the preview panel as you plug the right things together your result should be as you expected of it. Refer to the vertex painting you did as you plug respective bind points.
Add Follow Object fields by ticking them and adding the color and bind point.
The Simulation is calculated on the colors, that you don't add (like in my case I didn’t add black and also the gradual shades near arms get excluded), with respect to other vertex colors. Once you are done with adding all the colors and bind points check if all the colors are working around their bind points. If you see any weird spikes in your clothing where you had some vertex color you probably need to color that region again. This problem occurs if any vertex in the region received uneven colors. After you’re done with Vertex Painting, Now it’s time to configure colliders!
What are colliders? These are shaped meshes that represent your underlying object and react to physics (specifically collision physics, i.e. reaction when something comes in contact with it). In the Digital 3D world, colliders and gravity are very important components for physical interactions to work and have a response as governed by Newton’s third law of motion. The Simplest example of collision is a sphere on a ground plane, if the sphere or the plane either of them is missing colliders then if you drop the sphere it will pass through the plane without being stopped by it. You Learn More About Colliders online.
Back to our project, In the template you can see an object named Collider Guides, these collider guides are used to have collisions with your clothes. This collision of the body with clothes happens in real life too which is how the clothes interact with your body. The Collider guides have a guide collider object under them where each guide object acts as a part of your body and works along with your bind points.
If you check your Cloth Simulation Script, you will see a section where there is a list of Colliders that interact with your clothes. Our base jacket example had only some upper body colliders but my cloak needs to react to my spine, hips, legs too! So I will simply click Add Values and add the colliders I want to include.
Now if everything looks good, our clothing is almost ready. You can experiment and tweak values in the script to understand how it behaves and what suits the best for you. You can swap the material in the script with your own material. This is what I made for The Akatsuki Cloak.
Test your clothing on yourself and you can test its performance on your device.
If you still face problems. May God Help You! *Just kidding you can connect with me and if I can help you solve your problem I will try my best.*
THAT IS THE END OF THIS SMALL TUTORIAL! Please feel free to share your feedback and let me know if this helped you.