The Building of Reallusion’s NVIDIA Omniverse Connector
Learn how Reallusion’s Connector was built from scene and character animation to material and light conversion
Weien Chen, Senior Software Engineer, Reallusion
Character creation and animation are two distinct disciplines that demand the skill of well-trained artists with specialized background knowledge. These domains can be difficult and frustrating for artists who come from non-related backgrounds with different skill sets, and it’s a pain point that Character Creator and iClone were specifically created to resolve.
Furthermore, Character Creator is well-positioned as the go-to solution for creating realistic and stylized characters with robust pipelines for mainstream tools like ZBrush, Substance, and Blender. Combined with the ability to export FBX with LODs (levels of detail), digital human shaders, and a rich collection of motion assets, iClone stands out as an animation editor without a steep learning curve.
The launch of NVIDIA Omniverse in 2020 was a momentous occasion that attracted our attention. Omniverse represents the next-generation 3D virtual collaboration and real-time simulation platform that connects people and applications for broad-based collaboration. Reallusion software suite combined with a massive library of high-quality character assets and motions would play a crucial role in this ecosystem, while Omniverse provides the path-traced rendering and AI technology that makes for a powerful synergy.
Where to start?
To build the connector, we start by referencing the “Connector Sample code” available for download on the NVIDIA launcher application.
On the Omniverse Youtube channel, there is a great beginner tutorial on how to “Create an Omniverse USD App from the Connect Sample”.
Scene and Character Animation
iClone and Character Creator’s 3D scene consists of nodes with basic transforms consisting of translation, rotation, and scale values. Characters, meshes, lights and cameras are all attached under these nodes. Furthermore, the bones of a character skeleton are also represented by these nodes. Nodes that only represent transforms are exported as USD Xform and nodes that represent the body and facial bones are exported as USD skeleton joints. Additional bones and skinning are added to accessory nodes that are attached to bones before converting to USD format.
USD Xform scaling works in fundamentally different ways to iClone. In iClone, a node can be made to inherit or ignore the parent scale, whereas, in Omniverse, the node scale is always inherited from its parent node. Under such conditions, bone node scale inheritance must be removed with its values reset before exporting to Omniverse, in order for the scale values to match.
Most of iClone’s facial expressions are composed of morph animations which are exported as USD blend-shapes. In contrast to FBX blend-shapes which are stored in local-space positions, USD blend-shapes store positional offsets. Since iClone also stores positional offsets (in combination with strength multipliers), it is completely compatible with Omniverse, allowing for direct conversion to USD format. It should be noted that Omniverse requires a skeleton root for props with blend-shapes attached and additional processing may be required.
Material Conversion
The following section contains MDL code excerpts for iClone’s USD exporter. Refer to https://developer.nvidia.com/usd/mdlschema for more information.
Within the USD file, MDLs are specified using info:mdl:sourceAsset with info:mdl:sourceAsset:subIdentifier as the entry function. The new subIndentifier attribute was introduced by NVIDIA and PIXAR. input: is later called to feed the texture and material parameters. OmniPBR.mdl and OmniSurface(Base).mdl provided with Omniverse were used as starting points.
OmniPBR.mdl was chosen from the start because it works well in both RTX Real-time and RTX Path-traced mode, on the other hand, OmniSurface and OmniHair are photo-realistic MDLs for RTX Path-traced mode. The existing PBR, Skin, Head, and SSS shaders were then rewritten from HLSL to MDL.
Flowing water in the swimming pool is another good example, it’s simply:
float2 inputs:texture_translate.timeSamples = {
0: (0, 0),
4000: (4, 8),
}
Besides the above mentioned built-in MDLs, there is also a base.mdl on GITHUB with some reusable functions that can be deployed in a jiffy.
Light Conversion
Point lights and spotlights use UsdLuxSphereLight with adjusted cone angles. Tube and rectangle lights use UsdLuxCylinderLight and UsdLuxRectLight respectively. Light IES profile file is also one of the shaping attributes. Light intensity in USD is similar to Luminous intensity per unit surface area. The USD intensity of a spherical light with radius (r in meters) can be approximated with the following formula:
USD intensity = candela * 1000 * / (4PI r*r)
The following formula is used when radius is in centimeters:
USD intensity = candela * 1000 / (4PI(0.01r)*(0.01r))
Radius is a significant attribute in Omniverse Renderer. We impose a minimum radius of 2 cm as suggested by NVIDIA. See our lighting tutorial for more examples.
What’s next
A prototype of a one-way live sync connector is under development. Since iClone’s undo/redo system is similar to the memento pattern, a table for keeping track of live objects with universal IDs is used. This table will update after performing an undo and redo function.
Check out our character creation and character animation solution, or you can watch our 2021 GTC Fall session video recording to get an overview of how this solution works.
For more information and to join the discussion, visit the NVIDIA Omniverse forum and NVIDIA Omniverse Discord Channel. For more resources on USD, refer to USD GITHUB and prebuilt Lib/Tool by NVIDIA
__________________________________________________
About the Author
Weien Chen is a senior software engineer on the Graphics team at Reallusion. He participated in shipping iClone 7/Character Creator 3 and their Omniverse Connector. He also worked on Reallusion PopVideo enhancement, VXGI and PopcornFX and Reallusion skin, SSS, and hair shader development.
