Conduit : ProdFS
Previous: Conduit : Deploying Portal and PVC
Back for more.
Every studio has its own flavor of resolving a path to a versioned file — environment variables, DCC plugins, a scrappy set of tools to manage symlinks, etc.
Pixar provides a robust cross platform mechanism within USD to programatically resolve filepaths in the ArResolver class. Many DCCs provide the ability to insert pipeline logic into filepath resolution. Unfortunately, not all DCCs have a robust plugin interface. So what to do?
We decided to do symlink management through a virtual file system. And we’re definitely not the first — Rhythm & Hues had the Production Tracking System Filesystem (ptsfs) written by Walid Harmoush and Gautham Krishnamurti back in the early 2000s. Lucky for us, in 2019 we can leverage the “Filesystem in Userspace” (FUSE). We call our virtual file system the Product Filesystem, or ProdFS.
Products and Workspaces
A couple of basic pipeline concepts when working with our Conduit pipeline:
- An artist does their work in a workspace.
- An artist must declare their dependencies as input products.
- An artist must declare what they want to share as output products.
Product Version Control (PVC) manages the versioning of files in Conduit. We also track the relationships between workspaces, input products, and output products. For each output product version, Conduit registers the source workspace. This tells Conduit the workspace from which each output product version was created. If an artist wishes to edit an input product, they switch the input product to an output. By doing so, the artist can edit the former input product and create new versions. There’s no locking of products within Conduit. However, by declaring a product as an output — meaning, an artist wants to write to it — the artist lets others know that they are editing that product. There is no restriction that prevents multiple artists from editing the same product and we do provide tools to help resolve file conflicts. But with proper layering in USD, this should be less of an issue.
Version resolution through ProdFS & the PRI Table
Once an artist creates a workspace and declares the input and output products, they are ready to work. And by work we mean write out new files to share. Within the Conduit UI, an artist can launch a shell or a DCC that runs within a workspace context.
There are 3 key file locations, or mounts, managed by PVC:
Working (/w)
PVC stores local working mutable copies of the workspace and output products under the working mount:
Repository (/r)
PVC stores immutable versions of products (including workspaces which are a special type of product) under the repository mount.
ProdFS (/p)
ProdFS presents to an artist all the directories they will read and write from. The directories are versionless symlinks to locations under either the working mount (for workspace and output products) or the repository (for readonly input products).
It’s important to note that the directory structure for either the repository or workspace mounts is irrelevant — at least from an artist’s point of view. The structure of these mounts can be whatever best suits the needs of the infrastructure. The only artist facing directories will be from the ProdFS /p mount.
PVC stores with each workspace the desired version for input products. In general, the artist simply chooses latest or published from within Conduit UI. Conduit UI then queries PVC/Conduit to determine the correct versions. PVC/Conduit records resolved locations in a JSON file called the PRI table. ProdFS looks to the PRI table to determine what directories to present to the artist and how to symlink those directories appropriately.
We chose to use a JSON file as an intermediary between PVC/Conduit and ProdFS to minimize calls to the database. Additionally, it allows for a disconnected workflow (no connection to the Conduit services) as long as all input products from the repository are also available locally. PVC/Conduit is only required to initially create the workspace and to copy the working data to immutable versions in the repository when published. PVC/Conduit does not serve up the data — we’re simply letting NFS and the symlinks do that. The ProdFS service is distributed to each workstation. The current version of ProdFS relies on release 1 of python-llfuse. We found this particular library to be the most stable with our version of CentOS. Importantly, it does not require python 3 — still a bridge too far at this point. Someday…
It’s also important to note that with ProdFS, an artist can simultaneously have n number of unique active workspace contexts on a single machine. Conduit UI initializes the context when a shell is launched. Processes on the farm must also initiate a context for each render task. The ProdFS /p mount will present the appropriate content for each context. For debugging an issue, TD’s can launch a shell or shell command through the Conduit UI in a specific context. Standard linux terminal commands, like ls and cd, allow an artist to inspect the state of ProdFS.
All the examples above are obviously not representative of the complexity of a typical production asset or shot. And there is quite a bit that happens during publishes to stage the data and deduplicate files. But we wanted to introduce the usage of workspaces and ProdFS. Artists only ever read, write, or access files from directories that don’t exist without the ProdFS service running. The PRI table can be parsed and used to repath the /p (virtual) file references to /r (repository) in asset or shot data. But that’s no fun as you lose the ability to dynamically update version references.
Who’s using it and what’s the catch?
So… who’s using ProdFS?
ProdFS has been in development since 2016 and production testing began in the summer of 2018 with deployment to a subset of our workstations. On Monday, February 11th, ProdFS was deployed to all machines at the studio as production adoption of Conduit continues.
Downsides?
FUSE has shown itself to be an elegant solution for resolving versions through ProdFS — on linux. However, recording linux paths rather than relying on programatic/plugin resolvers (like ArResolver) means filepaths won’t “just work” on Windows. We don’t currently have a complimentary ProdFS solution for Windows which is a drag for our Immersive Technology (AR/VR) team. A python script can parse the PRI table and repath references to a windows friendly repository. While there is a FUSE for macOS and a FUSE for Windows, we haven’t explored how to replicate the functionality of ProdFS on those platforms yet. That’s on our to-do list. In the meantime, ProdFS allowed us to simply skip the step of writing resolvers for every DCC and USD.
For now.
