USD(Z) Deep Dive for ARKit: Part One
At WWDC 2018 Apple announced various improvements to its support for Augmented Reality experiences: one of these was the support of a new file format, USDZ. USDZ enables new experiences and adds new ways in which users can interact with Augmented Reality content.
I’ve written more about WWDC 2018 here:
This year’s WWDC was packed with AR-related content. Apple singlehandedly pulled the rug from under some of the main AR…medium.com
In this article, I dispel some myths about the USD(Z) format, talk about how it works with respect to ARKit and point out some nuances which I found while working with it and writing conversion tools for it. We will slowly dissect one of the USDZ examples provided by Apple to understand the format when applied to ARKit and QuickLook.
There’s a ton to cover, so in the next articles of this series I will explain the nuts and bolts of the USD specification and file format from a more generic point of view to enable you to work with it and create USD scenes programmatically via either Python or C++ from any platform, not necessarily a Mac.
What is USD(Z)?
Universal Scene Descriptor (a.k.a. USD) is:
an efficient, scalable system for authoring, reading, and streaming time-sampled scene description for interchange between graphics applications.
Let’s translate that to English.
Very simply put, USD is what you would call a 3D asset file format with some special additions. One of these is the ability to package a USDC file with its textures into a USDZ file for consumption by ARKit and QuickLook.
What does all this mean for the AR industry and ARKit?
USDZ files can, for example, be shared via iMessage or you can open them directly from a webpage with Safari:
The cool thing about these two examples is that they show how QuickLook allows users to experience AR content without having to install any new app.
This is huge: imagine being able to see in 3D, in the real world, any product before you buy it from your favourite online shop, all of this without having to install any new app! This would not be possible without USDZ’s support from Apple.
Another use I am experimenting with is to showcase / share medical 3D models easily: send them to your friends with an iOS device and they can immediately check out your work, no apps needed. The usage cases for USDZ and QuickLook are many.
USD is developed by Pixar and is used in their software, from Hydra to Presto. Pixar made its implementation open source some years ago on Github, so now anyone can use it to implement conversion tools or to integrate it in their engines and software:
Universal Scene Description. Contribute to PixarAnimationStudios/USD development by creating an account on GitHub.github.com
A single USD file can be thought of as representing a scene, and a USD scene is a hierarchy of what Pixar calls Prims (which intutively can be thought of as nodes.)
USD performs almost the same function that, say, the gLTF or COLLADA file formats do, but compared to these two other example formats it has both more and fewer features.
For example, one of the main differences is that gLTF is starting to become widely supported, whereas USDZ is currently only supported officially on iOS’ QuickLook.
Getting the sample USDZ asset
For the purposes of this article, I will assume that you are on MacOS and that you have setup your USDPython tools to be used from the command line in your terminal as explained here after downloading them from here.
Visit https://developer.apple.com/arkit/gallery/ from your browser, click on the guitar 3D model and save it somewhere sensible.
Extracting a USDZ asset without renaming it
I’ve read online that most people when they want to access the files contained in a
.usdz file, they first rename it to have a
.zip extension and then extract it from Finder.
Instead, we can use the command line tool
unzip to extract an USDZ file directly as such:
unzip stratocaster.usdz -d stratocaster
You can now navigate to your extracted folder and directly open the
.usdc file to check that it can be opened like before:
Before we continue, let’s understand what all these different file formats mean.
What’s the difference between USD, USDC, USDA and USDZ?
Files with the
.usda extension all represent a USD scene file, but the way they store this data is different.
Recall that a USD file can be thought of as representing a scene, and a USD scene is a hierarchy of nodes. Most of the time you will have a 1-to-1 correspondence between a USD file and a 3D model, instead of a scene; i.e. you will have one single mesh in the file.
.usda files are USD scenes stored on file in ASCII format. Hence these files are text files containing the scene decription in a manner that is human-readable. For example, the following is the
.usda file for the representation of a USD scene containing a simple cube:
def Mesh "cube"
float3 extent = [(-0.5, -0.5, -0.5), (0.5, 0.5, 0.5)]
int faceVertexCounts = [4, 4, 4, 4, 4, 4]
int faceVertexIndices = [0, 1, 3, 2, 2, 3, 5, 4, 4, 5, 7, 6, 6, 7, 1, 0, 1, 7, 5, 3, 6, 0, 2, 4]
normal3f normals = [(-0.15186, 0.988312, 0.013374), (-0.206543, 0.977774, -0.036018), (0.026661, 0.999633, 0.004783), (0, -1, 0), (1, 0, 0), (-1, 0, 0)] (
interpolation = "uniform"
point3f points = [(-0.5, -0.5, 0.5), (0.5, -0.5, 0.5), (-0.5, 0.5, 0.5), (0.5, 0.5, 0.5), (-0.5, 0.5, -0.5), (0.5, 0.5, -0.5), (-0.5, -0.5, -0.5), (0.5, -0.5, -0.5)]
uniform token subdivisionScheme = "none"
As you can see the format is quite readable and makes it easy to understand how the scene is organised.
However this format is not efficient, since for large files parsing a text file takes a long time. For this reason Pixar also specifies the
.usdc file format, which allows software to read USD scenes much more quickly.
.usdc files are the binary representation of USD scenes. Hence they are stored in binary form, and opening them with a text editor is not of much help. Here is the cube scene example again, this time after opening its
.usdc equivalent with a text editor:
��� ��� �����������������¿���¿���¿���?���?���?����������������������������������������Ô���EEEUU�ÿ�úÿþþú����������������������?������?��������������¿������¿������?����������¿������������������¿���¿���?���?���¿���?���¿���?���?���?���?���?���¿���?���¿���?���?���¿���¿���¿���¿���?���¿���¿�������Ú�������Ö��������ó�primChildren�cube�specifier�typeName�Mesh�properties�subdivisionScheme�points�extent�normals�faceVertexCounts�faceVertexIndices�float3�default�intB�ò 3f�interpolation�uniformq� �ðtoken�variability�none������������������������ð
���UEQþö�öóöùk�������� X��)�a���*@�1 @d�1)�
�1 @�1�1 @¨�2È�! �1 @ø�1�3 @� H �0�1� @�°�,@����� @���������������ð
��������°���QA�õþÿ ������������@ÿþ������� ��������p����� �������� ���� �������������ÿû�������TOKENS����������°������î�������STRINGS����������������������FIELDS����������¦�������������FIELDSETS�������B������/�������PATHS�����������q������I�������SPECS�����������º������@�������
However these files can be read by software much more quickly than their
.usda equivalents. In fact USDZ packages for QuickLook must use
.usdc so that the scenes can be loaded quickly on mobile devices.
.usd files can be both ASCII-based or binary files. Their extension doesn’t give away what format they are from the outside, but systems reading them can determine their format from the first bytes of the file. Hence,
.usd files can be both text-based or binary based, i.e. you can rename either a
.usda file to
.usd and they will work like before you renamed them.
This format is the one which is used by QuickLook to render scenes and can be imported and used in iOS projects developed in Xcode.
In its current form it’s more or less a zipped folder containing at most (and at least) one
.usdc file and any textures which that scene uses.
You can learn how to create USDZ files from OBJs here:
With iOS 12, ARKit includes a built-in viewer for displaying and sharing high-quality 3D content using Pixar's usdz…developer.apple.com
Exploring the Stratocaster USDZ asset
Now that you have extracted the Strato, you can convert its
.usdc file to
.usda as such:
usdcat Stratocaster.usdc -o Stratocaster.usda
Stratocaster.usda file with your favourite text editor, and boom:
We can now read the contents of the file to understand how USDZ assets are organised for consumption by QuickLook.
Open the newly created
Stratocaster.usda by double clicking on it so that the changes we make to it are shown in real-time.
Just at the top you can see the definition of the various materials for the asset. Some of these use constant properties, whereas others, such as this one, use a texture map:
For usage with QuickLook, the names of the attributes (a.k.a the material properties) in the various
pbrMat1s need to be exactly as shown in the example above. Changing the name of
diffuse in the material “StratocasterGuitar_SynthString_Material” (not the one shown above) results in the loss of albedo colour for the components using that material:
For every material attribute that uses a texture instead of a constant value, QuickLook expects a material Prim like this one:
def Shader "color_map"
uniform token info:id = "UsdUVTexture"
float4 inputs:default = (0, 0, 0, 1)
asset inputs:file = @StratocasterGuitar_NeckWood_Albedo.jpg@
float2 inputs:st.connect = </Materials/StratocasterGuitar_NeckWood_Material/Primvar.outputs:result>
token inputs:wrapS = "repeat"
token inputs:wrapT = "repeat"
and you also need to provide a texture sampler:
def Shader "Primvar"
uniform token info:id = "UsdPrimvarReader_float2"
float2 inputs:default = (0, 0)
token inputs:varname = "Texture_uv"
This is what the material definition looks like when taken as a whole:
Scene Graph and Meshes
Looking at the expanded Scene Graph for the Stratocaster model, one thing is immediately noticeable: it seems to repeat elements.
What’s the difference between the “splitContainer” section and the elements ending with “Material?
tl;dr: They are essentially the same thing, and the nodes without anything attached, the ones ending with “Material”, are simply a by-product of how USD assets are parsed by the viewer and SceneKit.
If we go back to our
.usda file, we can look for the text “”StratocasterGuitar_ButtonTone_Material”, and here it is:
Submeshes are represented by GeomSubset Prims in USD; they are used to specify subsets of the whole model to which to apply a material exclusively.
If we were to look for any of the “splitContainer” elements, we wouldn’t find them in the file, because they are auto-generated from the indices of a given submesh, and use the material they are assigned via the
rel material:binding relationship.
The indices listed above for the ButtonTone element are indices into faces, not vertices. They refer to the global face indices for the whole guitar mesh, which can be found further up in the file:
The “StratocasterGuitar” Prim also defines the elements composing a vertex: positions, normals, texture coordinates, etc, which are then indexed via vertex indices.
Materials in USD
The USD specification doesn’t explicitly forbid the usage of non-PBR materials, but iOS’ QuickLook doesn’t work with non-PBR materials, so make sure that your USDZ file uses PBR materials or QuickLook will not render them properly.
Why is PBR important?
PBR stands for “Physically Based Rendering”. PBR enables a variety of enhancements to how realistically objects can be rendered, one of these being the ability to use image-based lighting, which provides more accurate ambient diffuse and specular reflections.
The differences between “traditional” shading and PBR shading can be stark:
In Augmented Reality, PBR is especially relevant as it allows the 3D models being rendered to be shaded in a way which is more aware of its surroundings. For example, if you were to render a USDZ model of a shiny metallic cube, with PBR materials and QuickLook the cube would reflect the environment around it as if it were a real one:
Current issues and limitations with USDZ
One of the biggest pains by far is the lack of good tools for artists to export their creations from their asset creation software to USDZ directly and without having to use the command line tools. Command line tools are not for everyone, and most of all the provided ones work with only a handful and outdated intermediate asset formats, making it hard to really exploit the functionalities of the USD format.
In their current form, neither the USD or USDZ specifications talk about the ability for the user to interact with the 3D scene described by a USDZ file. Hence all that’s possible to do at the moment when opening a USDZ file in QuickLook is to look at it, and nothing else: tapping on it won’t trigger an action and such action has no way to be coded into the format anyway.
You can bake animations into the asset, but those are pre-baked and don’t make it interactive.
You can’t use non-PBR materials when combining USDZ with QuickLook. Mind you, nothing stops you from declaring Attributes (in USD-speak) on your Prims which have names resembling the material properties you’d find on, say, a Lambert material, but they won’t work with Pixar’s renderer or Apple’s neither.
This is a limitation especially when you want to display widgets/abstract 3D objects rather than 3D models of realistic objects.
Apple products only
So far USDZ hasn’t been adopted by Android or other platforms, and hence it can only be consumed by users with iOS and MacOS devices. On these other platforms gLTF and gLB seem to be the main candidate to become the USDZ equivalent.
Moreover, QuickLook is only available on iOS and although third-party solutions exist, there isn’t an equivalent native solution on Android, but you can be sure that if and when it arrives, it will use the gLTF format and will display the content slighly differently from QuickLook.
AR is inherently averse to the concept of apps: you don’t want to have to switch to a different app every time you want to access a different AR layer or service, let alone download and install an app which you’re going to use only when at a specific location and never open again.
For this reason, the fact that USDZ allows users to interact with AR content without having to install an app is a step forward in that direction, albeit a small one. To really move in the direction of an AR cloud, we need to be able to, at least, add interactions to experiences such as the ones offered by USDZ.
This calls for a solution closer to a web browser than an app store; Mozilla and others are working towards that solution with WebXR.
Given what the USD format is intended as a format to represent 3D data, but not interactions, I don’t think that Apple or Pixar will modify the specifications to introduce interactions and scripting.
Instead, Apple will either have to bite the bullet and allow users to work with something similar to the WebXR/AR or extend QuickLook/Safari to integrate scripting of the USDZ scenes available.
Apple will have to gradually introduce better tools to author USD assets if it doesn’t want to officially support gLTF, so we might see some announcements with respect to that at WWDC 2019.
USDZ, when combined with ARKit and QuickLook allows you to:
- Let users try out 3D models of realistic objects without having to install apps; great for e-commerce applications
- Send 3D models to friends via iMessage
- Package USD scenes into a portable format
- Render objects in 3D realistically thanks to PBR
By far the most promising usage for USDZ files is the first one on the list: users can try out products in real-life scale and with a lighting environment that makes them look as much as possible as the final product after purchase:
Follow me on Twitter: twitter.com/albtaiuti