Introducing Achievements into QC

As soon as an errant publish breaks a nightly render, TDs usually introduce some sort of quality control process into an artist’s workflow to prevent future breakages.

We experience significantly fewer render-stopping breakages thanks to USD and the ability to layer asset functionality. But even with USD, an artist can inadvertently forget a necessary pipeline task or downstream requirement without some sort of QC step. We faced a proper QC challenge: How could we flag artists about incomplete assets without being so intrusive that they’d turn off the QC steps altogether? We decided to introduce gamification into the process and use Achievements to better inform artists and production management about the state of a given asset.

Quality Control and EZ-Pass

What are some examples of QC steps? These can be anything from making sure a model has proper UVs and scaling, checking for the proper setup of materials and fur, to running a full-blown render to ensure it generates a non-black frame that falls within memory and rendering limits.

We identified three basic types of QC:

  • Preflight Checks — tests that are typically interactive wherein a user is provided visual confirmation within their working application; these are quick and can be run while an artist continues working without having to publish their work.
  • Publish QC — QC tests run during a publish process.
  • Post-Publish — QC tests that require processing and occur after a publish has been complete. These can be renders, simulations, etc. A published asset may be available to downstream departments before these QC tests are complete.

The results for the Publish and Post-Publish QC tests are stored in an entity we call a certificate. Essentially, this is an immutable QC log of the test results that is stored with the published version.

EZ-Pass checks used on Ferdinand

At SIGGRAPH 2015, we presented our pipeline execution framework called TaskProcessor (TaskProcessor: A Pipeline Execution Framework and IDE). Built on top of the TaskProcessor is a tool called EZ-Pass. EZ-Pass allows TDs to manage and execute all three of these QC checks. It provides visual feedback to the artists as it runs through the tests, and in some cases, it can run automated fixes.

One challenge is that many of these checks are relevant only to downstream artists — not the artist actually running the tests. For example, a simulation artist may require a particular hierarchy or attribute that is meaningless to an upstream animator. When the animator runs this test and it fails, she has no context for how to fix the issue and may simply turn that test off. This is particularly burdensome for QC checks that take processing time (e.g. running a render). Artists don’t want to wait while running time consuming checks for downstream departments. EZ-Pass allows longer running Post-Publish tasks to be run offline.

Rebecca Hallac and Christopher Moore discussed Blue Sky’s Asset Health initiative in the 2018 SIGGRAPH Talk, Achieving and Maintaining Real-Time Rigs. For Spies in Disguise (2019), we wanted to ensure that each publish of a character maintained real-time rig performance for the animators. But in reality, as a character is developed, the performance would vary greatly until it was optimized by rigging. Preventing un-optimized rigs from being published to animation was simply not an option during character development — an animator needs to continue to work even if the rig is un-optimized. This is where Achievements help.

QC : Achievements Unlocked

Supervisors and production management can define expected/desired results of a publish in the form of Achievements. Achievements can be earned by passing one (or a set) of QC tests for a publish. Upon creation, an asset has a default set of basic Achievements that are marked as “Achievable”. What is Achievable can be adjusted throughout the life of an asset. After both the publish and post-publish process, the certificate is parsed to determine what Achievements have been earned. With the introduction of the health service, these results are stored in Conduit (see Conduit : Introduction). Artists and supervisors can see what Achievements have been earned in a variety of web interfaces and custom UIs in the artist’s working application.

This asset earned 7/7 achievements!

While asset versions and certificates are immutable, Achievements can be revoked if the production determines a new set of criteria for that Achievement. For example, say an asset version earned the “Real-Time Achievement” when the criteria for “Real-Time” was 24 frames per second. If the show suddenly requires 48 fps performance, the show’s Real Time Achievement criteria can be modified and past versions that no longer meet that requirement have that Achievement revoked. This new status, Achievement Revoked, indicates that a version no longer meets those requirements. This is why we record the certificates (which are the results of the QC tests, immutable and not typically artist facing) and achievements (which can be revoked) separately.

Lastly, Achievements can be assigned a point value. This is purely a subjective number to try and impose a relative importance for each Achievement. The original concept was to be able to chart Achievement progress throughout the life of the asset. However, the Health Service became a far more useful tool than the point system, as it provides detailed historical charts on relevant health data.

Conclusion

Providing a QC step for assets and shots is simply a requirement that artists begrudgingly accept. While still a work in progress, we’ve found that a focus on what has been earned (an Achievement!) rather than what is broken ultimately results in significantly cleaner inventory.