What You Need to Know About Universal Scene Description — From One of Its Founding Developers
USD is the HTML of 3D Worlds and will be key to unlocking the metaverse
By: Aaron Luk, Senior Engineering Manager, USD Ecosystem
Can the metaverse be a better ’verse? As the hype around the metaverse as a metaphor for all things futuristic coalesces into concrete efforts to enable the 3D evolution of the internet, it’s a good moment to reassess how both human beings and machine learning can reason about the scale and complexity of the data models and pipelines needed to represent virtual worlds in full fidelity.
A concept from the 1990s, the metaverse was first coined by science fiction author Neal Stephenson as a set of connected virtual worlds that extend the physical world. Like the 2D web, there are consumer and industrial uses of the metaverse. The 3D worlds of the metaverse will usher in a new era of design and simulation that unlocks new possibilities for AI and global industries.
So how is the metaverse evolving from speculative fiction to reality? The metaverse will require a common standard upon which to describe, compose, populate, simulate, and collaborate within 3D worlds. Again like the 2D web, the success of the metaverse’s 3D spatial overlay of the 2D internet will depend on its universal interoperability as governed by open standards and protocols.
Universal Scene Description (USD) will unlock the metaverse
Originally invented and open-sourced by Pixar Animation Studios to support large-scale visual effects of feature films, USD is an open scene description with APIs for creating, editing, querying, rendering, collaborating, and simulating in virtual worlds. Today, its highly scalable and interoperable qualities are enabling large, high fidelity scenes that accurately simulate the real world. We call these simulations digital twins.
A typical use of USD assembling scene elements to create a ground-truth view of a composed world is in automated factory assembly lines. With USD, numerous layers that describe properties of a robot in the factory can be combined together with sparse overrides to non-destructively modify source data. The geometry, shading, lighting, and non-graphical properties such as electrical info and sensor data are composed in a USD layer stack and presented as an autonomous robot.
The assembly line on which the robot sits can also have its own layer stack describing its properties. With USD, these can be easily stitched together into a composed factory world that brings together all the layer stacks in a USD stage. This stage can then be fed into any number of subsystems including rendering and physics to simulate real-world behaviors.
Digital twins for manufacturing and other use cases are already a reality. Companies like Siemens, Amazon Robotics, and BMW Group are leveraging USD to bring their virtual factories to life, finding new levels of operational efficiency for large-scale workloads.
Unique properties of USD
USD is much more than a file format. It’s a framework, ecosystem, and interchange paradigm that supports a wide variety of properties to define and render objects, while making scene structuring and editing more efficient and collaborative with sparse, non-destructive authoring.
There are four unique properties of USD that enable it to be highly extensible and meet the demands of virtual worlds:
- Composition Engine: The composition engine enables the assembly of data from numerous sources as individual layers and the authoring of sparse, non-destructive overrides on top of those layers. This preserves the integrity of source data ingested in the USD ecosystem.
2. Custom Schemas: The data model of USD is fully extensible with custom schemas. USD itself comes bundled with core schemas such as geometry and shading. NVIDIA has worked with Pixar and Apple to create physics schemas for rigid bodies. Other custom schemas are being explored to further expand the ecosystem for digital twins and virtual worlds.
3. Asset Resolver & Data Storage: USD is entirely filesystem agnostic. The data is not tied to filesystems or any other persistent storage. It can even be procedurally generated. This is made possible by USD’s plugin system for asset resolvers and file formats.
4. Hydra: USD can feed different renderers out of the box like Pixar Storm and Renderman with Hydra, a generalized pipeline for custom renderers. NVIDIA has also integrated real-time ray tracing and physically accurate rendering support with Hydra.
Pushing the limits of USD with real-world testing
NVIDIA and our partners continue to push the boundaries of Universal Scene Description. The latest developments in USD are driven by real-world testing in applications for autonomous driving and robots and industrial and scientific digital twins.
The increased scale and complexity of the data models needed to mirror the real world leverages the relative scale and complexity of world-building for feature film animation. But the requirements for digital twins sprawl into entire families of use cases beyond media and entertainment. Here are some areas where digital twins require core enhancements to the USD ecosystem.
Unicode Identifier Syntax (UTF-8)
NVIDIA is developing Unicode Identifier Syntax for USD to bring international character support to USD workflows. USD supports the UTF-8 standard for content, but identifiers such as prim names are currently limited to ASCII, presenting challenges when the source data is encoded in non-Roman languages. UTF-8 identifiers will be encoded and validated in USD core and may allow leading digits, unlocking many use cases such as interchange of source data from Autodesk Revit with UTF-8 identifiers for manufacturing in Germany or architecture in Japan.
Geospatial Coordinates (WGS84)
NVIDIA is developing USD data models for geospatial posing with WGS84, a geodetic datum describing the curvature of the earth. This allows USD content, for use cases like railroad design or city mapping, to georeference real world geometry using geospatial coordinates. NVIDIA has prototyped an API schema for WGS84 reference frames and local offsets.
The USD development roadmap includes plugin points for geospatial kernels such that arbitrary coordinate reference systems can be plugged into the system.
The future of USD
NVIDIA is continuing to invest in USD, with numerous projects on the roadmap. Recent contributions from NVIDIA to the USD ecosystem include:
- Python 3 support: Updating from Python 2.7 (sunsetted in January 2020)
- Adding usd-core to PyPI: Easy installation of USD core for Python programmers
- Material Definition Language (MDL) materials support: Encoding physically accurate MDL materials in USD data models
- Schema for rigid body dynamics simulation (with Pixar and Apple): Standardizing the representation of mass distribution, collision behavior, and other data necessary for rigid body dynamics.
In the medium term, in addition to the UTF-8 and WGS84 support described above, NVIDIA is developing:
- glTF interoperability: A glTF file format plugin will allow glTF assets to be referenced natively as USD content. This means that users who are already using glTF can take advantage of the composition and collaboration features of USD without having to alter their existing assets or pipeline, just like users of Alembic and Draco. We will collaborate with other industry partners to productize this glTF file format plugin in the open-source USD GitHub repository.
- USD compatibility testing and certification suite: To further accelerate USD development and adoption, NVIDIA is building an open-source suite for USD compatibility testing and certification. Developers will be able to test their builds of USD and certify that their custom USD components produce an expected result. This is critical to maintaining an open ecosystem on top of the rapidly evolving nature of the USD software stack.
In the longer term, NVIDIA is working with partners to fill some of the larger remaining gaps in USD:
- High-speed incremental updates: USD was designed more for offline import/export interchange rather than large-scale real-time updates needed by digital twin simulations. NVIDIA is developing additional libraries on top of USD that optimize data layout and change tracking to support real-time simulation at scale.
- Real-time proceduralism: NVIDIA has begun to augment USD’s declarative content with a procedural graph-based execution engine called OmniGraph, as well as Warp, a framework for writing simulation kernels in Python that are JIT converted to to C++/CUDA and compiled at runtime and run directly on the GPU.
- Compatibility with browsers: To be accessible by everyone, everywhere, virtual worlds will need to be capable of running inside web browsers on any device. NVIDIA, alongside other USD stakeholders, is exploring client-side approaches such as WebAssembly builds with JavaScript bindings, as well as server-side approaches such as pixel streaming from Omniverse Cloud.
- Real-time streaming of IoT data: Industrial virtual worlds and live digital twins require real-time streaming of IoT data. NVIDIA is working on building USD connections to IoT data streaming protocols.
More information can be found in the recording of the GTC session that covered earlier iterations of the developments described in this article.
Getting Started with USD
You can access a wealth of USD resources from NVIDIA, available online for free. A good place to start is with NVIDIA’s hub of USD resources, which includes tutorials and pre-built libraries and tools, such as turnkey distribution of usdview in Omniverse Launcher. To learn the basics of USD with examples in USDA and Python in a step-by-step web tutorial, sign up for the USD DLI course.
NVIDIA Omniverse is a scalable computing platform for full-design-fidelity 3D simulation workflows and a toolkit for building USD-based metaverse applications. Omniverse was built from the ground up as a USD engine and open toolkit for building custom, interoperable 3D pipelines.
Learn more about USD at NVIDIA GTC
NVIDIA GTC, the conference for the era of AI and the metaverse taking place March 20–23, will feature numerous sessions on USD and developing with NVIDIA Omniverse.
Be sure to check out my session, An Overview of Universal Scene Description for Building Virtual Worlds, and a panel with leaders from Cesium, Open Geospatial Consortium, Autodesk, Adobe, Pixar, and NVIDIA about how USD can become the “HTML” of the Metaverse.
You can also access a hands-on training lab to experiment with the fundamental concepts of USD utilizing the USD Python API.
Subscribe to the Omniverse newsletter to receive updates about Omniverse Enterprise. Follow us on Instagram, Twitter, YouTube, and Medium to stay up to date with the next releases and use cases.
Visit the Omniverse Developer Resource Center for additional resources, view the latest tutorials on Omniverse, and check out the forums for support. Join the Omniverse community, Discord server, and Twitch Channel to chat with the community, and subscribe to get the latest Omniverse news.
About the Author
Aaron Luk, Senior Engineering Manager, Omniverse, NVIDIA
Aaron Luk is senior engineering manager leading Universal Scene Description (USD) ecosystem engineering, including oversight, integration, and optimization of USD and Hydra paradigms across all components of NVIDIA Omniverse. Previously, Aaron co-developed and deployed USD into production pipelines at Pixar.
