Conduit : Deploying Portal and PVC
Prev: Conduit : Services Archetype
Hello, friends. It’s been a busy 5 months since our last update.
As a refresher, there are a number of components to what we call Conduit, our next generation pipeline. The basic services and database layer has been in use since the delivery of our 2017 feature film, Ferdinand. This allowed us to track and version media generated by the artists (imagery and movies). For this first deployment, we were able to integrate versioning of media into the existing production processes so there was little impact to artist’s day-to-day workflow. This was intentional: we could test performance and scalability of our stack — and continually improve as needed — with minimal impact to production. Artists and production could continue to use the existing proprietary playback tools while we enticed them to our new media search interface called bssPlay. bssPlay is a suite of Python/Qt tools and libraries that can be accessed as a plugin in Autodesk/Shotgun’s RV playback tool. Its real power comes from leveraging Conduit services to quickly find media from any artist on any show — i.e., the basic functionality we all now take for granted when searching the internet.
Portal With a Side of Sangria
Building bssPlay directly into the playback tool as a plugin had the distinct advantage of being able to manipulate the RV session (playlists) natively. However, bssPlay had the disadvantage of being linux-based. This meant that many coordinators and production staff couldn’t search for and manipulate media in a browser on a variety of platforms and operating systems. The obvious solution was to build a web based application. While the initial application would be to manage media and review sessions, we knew that this would ultimately become the primary interface for both artists and production management to interface with Conduit.
Our Media Tools team spent the early part of 2018 conducting rigorous user story mapping sessions with key management staff across the studio. The team walked through each coordinator or supervisor’s day as it pertains to reviewing media to find inefficiencies and areas where we might be able to help. Building off of what we learned in developing the Health web application (demonstrated at the Achieving and Maintaining Real-Time Rigs talk at SIGGRAPH 2018), the Media Tools team designed and developed a web interface to manage media and review sessions. After a bit of research— which included scouring the musings of our friends across the tech industry on Medium — we settled on building this application using Material UI and reusable React components. Because this would ultimately be the interface for artists and management to access the pipeline, we called it Portal.
In November of 2018, we demonstrated the beta version of Portal to production at one of Blue Sky’s Technology Happy Hours. Started last year by Karyn Monschein, our studio’s Head of Technical Direction, we gather once a month in Blue Sky’s main theater to demonstrate to the whole studio the latest tools and innovations from across the Technology organization. And while we might entice people to come with cookies & sangria, they tend to stay to see the cool tech.
Portal gathered positive feedback and in a future post, we’ll walk you through how it works. Our intention all along was for Portal to go beyond an interface for finding media or managing reviews. We expect Portal to become the primary interface into the pipeline itself. After all, Conduit looks at media as simply another type of Pipeline Resource Identifier (PRI).
For the most part, deploying bssPlay and Portal required minimal change for artists or production management — at least compared to what we were about to do.
Yet Another Version Control System
In a previous post, we described how the heart of Conduit is the PRI. Media and other items in Conduit are versioned using the Product Version Control (PVC) system and are addressed by their PRI. When we first began designing Conduit, we had two primary goals for our versioning system that ultimately led us to write our own rather than use git or countless other battle tested systems.
The first goal was to keep the granularity of version references at the product level. Artists could think of working on and sharing products rather than focusing on the granularity of files. They would talk in terms of the collection of files that made up a component of the pipeline.
The second goal of PVC was not simply to track versions of products, but to also track the relationships of dependencies between products as they traveled through the pipeline.
In August of 2018, we began production testing PVC in our new USD-based pipeline. The design team expanded to include fresh eyes and several Technical Directors from production. While some of the choices we made in the original design were arguably “correct”, they introduced complexity that became clear during production testing.
When the Correct Choice Isn’t the Right Choice
In developing Conduit, we had to make some key design decisions that were knowingly “incorrect” from a technical perspective. Let’s discuss a few of those choices.
#branching
spies:100:10-lgt:walter@1.2.5
spies:100:10-lgt:walter@1.latest#noBranching
spies:100:10-lgt:walter@13
spies:100:10-lgt:walter@latest
One particular choice, the ability to branch a product, seemed like a necessary feature as well as an important concept shared with code revisioning systems like git. We originally assumed branching would be as essential for an artist’s workflow as it is in code development. In practice, however, when translating the ability to branch into specifying the latest or published version of a product, the complexity it introduced didn’t seem worth it to the artists. Thus we decided to hide branching. The underlying services and core of PVC still have the ability to track and store branching products if we revisit the choice in the future. But for now, products will version up using a simple integer-based version number.
#includePRIinPath
/p/spies:100:10-lgt:walter/walter.usd#hashedPath
/p/9A90F99E93DFF9C3EDD52CC72B1879F0C15ECD79/walter.usd
A related example of our intentionally making an “incorrect” choice was in how we access paths to products on the filesystem. In a future post, we’ll go into ProdFS (Product File System) which is the custom virtual file system we developed on top of Fuse. Essentially, ProdFS provides a context that resolves versionless file paths as they are accessed. Ideally, artists never have to think of the filesystem just as we don’t think of files we save on Google drive or any other modern file system, but we didn’t think we were quite there yet. To help with the transition, we chose for PVC to use readable(ish) filepaths. These filepaths would incorporate the PRI. The advantage is that when an artist opens a DCC or inspects a USD file, they can infer information about the contents of a file based on the path. The huge disadvantage to having the PRI as a part of the filepath is that renaming a PRI means that filepaths become invalid (without also being updated). A more correct solution would be to encode the filepath into a hash that would remain immutable even if the name of the PRI was changed. From a PVC engineering perspective, there’s no additional work to have an encoded file path. But because we don’t control all of the applications artists use, and because having a helpful hint in the filepath seemed important for adoption, we purposely made an “incorrect” design choice.
Acceptance of the Diamond Dilemma
Another “incorrect” design choice requires a bit of background in a versioning system that tracks data dependencies. Specifically, what we refer to as the diamond dilemma. Let’s start with this drawing depicting the dependencies between 4 products:
In this drawing, both p2 and p3 depend on p1. p4 depends on both p2 and p3, and by proxy p1. In Conduit, we use the “@” symbol to signify a version. Now, if p2@2 depends on p1@1 and p3@2 depends on p1@2, for p4 to correctly reference both p2@2 and p3@2 simultaneously, it must also reference both p1@1 and p1@2. This diamond-shaped dependency problem is easily handled by a computer, but when translated into production data for an artist it can be deathly confusing. In a production scene, an animator working on two characters that share a component may have multiple versions of that component in their shot. While this, in theory, ensures the integrity of the production assets, trying to debug or even explain a shot where the same component can appear multiple times at different versions… well, it stinks.
Combine that with the fact that most of the time, artists want to see the latest of everything. (Until they don’t — usually to isolate data for a production deadline or for an overnight render.)
Thus we made the decision in the design of ProdFS to automatically resolve a labeled product to a single product version. Meaning, for versionless file references, ProdFS will only resolve the path to a single version. While this may break the integrity of the production data graph, after discussing the alternatives we decided that for the sanity of our artists, this was the right choice. Technically, Conduit keeps track of what is correct and can inform artists when there may be incompatible versions. There is also a mechanism to explicitly reference a specific version of a product through ProdFS. But typically, artists will work with versionless paths and would need to explicitly change the path to reference a specific version if desired.
Ultimately, we chose to knowingly accept the diamond dilemma — at least in the case of resolving versionless references. To handle these diamonds, we abstracted the problem into a configurable set of dependency resolution strategies to allow TDs and developers to determine how best to handle these situations. We’ll go into more detail next month when we talk though ProdFS.
What’s Next?
So who’s using this thing again, you ask? Portal and bssPlay are deployed and now in use on both Spies in Disguise and Nimona. Starting next month, the first team of artists will use ProdFS exclusively for production data. Stay tuned for upcoming posts that provide deeper dives into Portal, ProdFS, and Workspaces which allow for multi-asset and multi-shot workflows. Also, we’ll be posting a glossary of Conduit-related terms since we try to take naming seriously.
