Distributed Rendering with Pachyderm

The Why

You have a bunch of friends who’ve gotten together in a hacker house ready to challenge the likes of Pixar with their indie Blender modeling and animation skills, everyone contributes some compute power to create a heterogenous makeshift render farm, but it’s on you to make a seamless queuing system that can render frames from everyone to craft a masterpiece…

Seriously guys, “Frames win games” ~ Nvidia

The How

If this is your first time trying out Pachyderm, make sure to read through my first post too (it helps!)

First, we need a bucket to contain everyone’s blender files that is yet to be rendered, for this we create a Repo

pachctl create repo blends

Next, let’s add a blender file to test our queue,

pachctl put file blends@master:/gallardo.blend -f gallardo.blend

To check if it got uploaded successfully,

pachctl list file blends@master

You should get something like:

NAME            TYPE SIZE     
/gallardo.blend file 794.8KiB

You can follow along on this by checking out my repo, under simple-blender-renderer

First we need to split the blender file into managable chunks, say each frame, so we can independently render them, we can use a Pipeline to do this…

It’s definetly neccasary to break out emacs and modify some py scripts

The Splitter

Essentially, we want to run blend_splitter.py on each file that’s submitted on the pipeline (from Blender’s context), the following is the wrapper that calls blend_splitter.py the reason we need two python scripts is because this one is called from the shell context — splitter.py

And this one is run from within blender’s context (indicated by the blend_ in the name of the script) — blend_splitter.py

What do we do here exactly?
For each frame in the blend file, we create metadata about the individual frame job and store that in the splitter pipeline as a list of json files, each file contains frame numbers that need to be rendered

To run this we’d need blender binaries, special thanks to The New York Times for maintaining containerized blender images, without which most of my time would have gone into compiling Blender from source!

(Psst! if you spotted opencv in this, and are wondering why we need it, stay tuned, it’s covered later)

Lastly, the pipeline specification,

What we convey in this spec is that we want to (from shell context) run splitter.py on every file that gets submitted to the blends repo this in turn runs blend_splitter.py (from Blender context) and generates frame numbers that we want to render and stores this metadata in the splitter repo

Putting it all together, from the folder that contains the four files above (simple-blender-renderer/blender-framesplitter in my repo), run:

pachctl create pipeline -f splitter.json --build --username <YourDockerUsernameHere>

Wait a few seconds, and run:
pachctl list file splitter@master:/gallardo

You should see something like:

NAME                TYPE SIZE
/gallardo/000000000 file 118B
/gallardo/000000001 file 118B
/gallardo/000000002 file 118B
/gallardo/000000003 file 118B
/gallardo/000000004 file 118B
/gallardo/000000005 file 118B
/gallardo/000000006 file 118B
/gallardo/000000007 file 118B
/gallardo/000000008 file 118B
/gallardo/000000009 file 119B

If you looked at one of these files, they’d look something like:

{
    "jobId": "gallardo",
    "frame": "1",
    "outFilename": "f-######",
    "file": "/pfs/blends/gallardo.blend"
}

Notice file here? We’ll use it in the next step

And now…

One does not simply count the number of times I say “render” in this article

The Renderer

First, let’s start out with the specification,

Notice the use of cross in the input spec? what this means is that would want to take [each frame — from each blend file] and cross join it with [each blend file]

Another interesting instruction is the use of both the repos to carry out the rendering task, the metadata that provides the frame to be rendered, and the blend file itself, we pass both to the renderer.py (with some handy debugging regexes to read the current state so we can track render completion metrics)

Which in turn triggers the blend_render.py that actually does the rendering, with the help of both the metadata from splitter as well as the actual blend file from blends

To run this, let’s create the rendering pipeline, from the folder that contains the renderer (simple-blender-renderer/blender-framerenderer in my repo), run:

pachctl create pipeline -f renderer.json --build --username <YourDockerUsernameHere>

Wait a few minutes so it can finish rendering everything and run
pachctl list file renderer@master:/gallardo

You should see something like:

NAME                   TYPE SIZE
/gallardo/f-000001.png file 947.3KiB
/gallardo/f-000002.png file 962.5KiB
/gallardo/f-000003.png file 932.9KiB
/gallardo/f-000004.png file 1.016MiB
/gallardo/f-000005.png file 1.03MiB
/gallardo/f-000006.png file 915.5KiB
/gallardo/f-000007.png file 937.6KiB
/gallardo/f-000008.png file 1.027MiB
/gallardo/f-000009.png file 1011KiB
/gallardo/f-000010.png file 1015KiB

You can also download all the frames by using the recursion -r flag on get file with this syntax:

pachctl get file -r renderer@master:/gallardo -o .

Here’s a digram of what happens behind the scenes

Simple frame rendering

Kudos! You have now built yourself a “upload blend files, get rendered frames solution” and are well on your way to becoming an indie animation studio! Sweet!

But…

Sensing a pattern with my articles now, aren’t we?

The Here We Go Again

We could do better than this to maximize our utilization of our heterogenous cluster, by splitting each frame into 16x16 pixel tile(s) and rendering each tile independently we’ll be able to better distribute our chunks so that slower nodes do not end up becoming a blocker for the entire job

(Psst! You can test this out on blender with Ctrl + B when you’re in the Camera view, which lets you render out a portion of the entire scene, in the next set of scripts we’ll programatically achieve the same outcome)

Step 1: The Map

To split each file into tiled frames, we can modify our blend_splitter.py to factor in a preferred tile size, in our example let’s take tiles that are 16x16 pixels, if you’re rendering with a GPU, you might want to increase this to something like 256x256 or 512x512 based on Blender Guru’s recommendation

You can follow along on this by checking out my repo, under split-blender-renderer

What do we do here exactly?
For each frame in the blend file, we create metadata about the induvidual frame job as well split each frame into tiles of 16x16 pixels and store that in the splitter pipeline as a list of json files, this is done with the startX, startY, endX and endY values and to finally merge them back together we’ll also need to know where the tiles need to be relative to each other, so we also stash locX and locY values in the same file

To run this, from the folder that contains the required files above (split-blender-renderer/blender-framesplitter in my repo), run:

pachctl create pipeline -f splitter.json --build --username <YourDockerUsernameHere>

Wait a few seconds, and run:
pachctl list file splitter@master:/gallardo

You should see something like:

NAME                TYPE SIZE 
/gallardo/000000000 file 222B 
/gallardo/000000001 file 224B
... more files here ...
/gallardo/000000398 file 231B 
/gallardo/000000399 file 231B

Step 2: The Renderer

The renderer is pretty similar, except we now need to plumb the startX, startY, endX and endY values to both the render.py as well as to blend_render.py (just the interesting bits henceforth)

We’ll also need to provide the same context in blend_render.py so Blender can use it to render just the tiles (we do this by specifiying the dimensions in Blender’s border min/max x/y properties)

We also need to provide the locX and locY values as part of the filename, so we can use it to merge the tiles back together in the next step, think of it as Prop Drilling in React

You could ignore lines 25 and 26, the tile_x and tile_y is what Blender uses to do tiled rendering of the subset of the tile that we have configured, think of this as: frame (that we specify) — tile (that we specify) — tile (that blender requires internally to intermittently render out the tile that we specify) if you don’t provide these two values, then Blender resorts to rendering the entire tile in one go, which depending on if you’re rendering on the GPU, might be exactly what you want

To run the renderer, from the folder that contains the required files above (split-blender-renderer/blender-framerenderer in my repo), run:

pachctl create pipeline -f renderer.json --build --username <YourDockerUsernameHere>

Wait a few seconds, and run:
pachctl list file renderer@master:/gallardo

You should see something like:

NAME                           TYPE SIZE
/gallardo/f-000001-x-0-y-0.png file 32.71KiB
/gallardo/f-000001-x-0-y-1.png file 32.71KiB
/gallardo/f-000001-x-0-y-2.png file 32.71KiB
... more files here ...
/gallardo/f-000010-x-7-y-2.png file 17.16KiB
/gallardo/f-000010-x-7-y-3.png file 17.16KiB
/gallardo/f-000010-x-7-y-4.png file 4.189KiB

You’ve now rendered the tiles, it’s time to merge them back into frames

Step 3: The Reduce

Using regex, we can collect the frame numbers, the x and y positions from the respective filenames, and merge them with opencv that we already included in our Dockerfile

First, the pipeline spec,

We essentially pick the folder that contains all our frames and provide it as input to merger.py

Roughly the logic is that for a given frame number we

1. Merge all the X tiles together horizontally to form broader X tiles
2. Merge all the broad X tiles together vertically from the step above to form the final frame
3. Stash the frame in the merger repo under it’s respective frame number

To run the merger, from the folder that contains the required files above (split-blender-renderer/blender-framemerger in my repo), run:

pachctl create pipeline -f merger.json --build --username <YourDockerUsernameHere>

Wait a few seconds, and run:
pachctl list file merger@master:/gallardo

You should see something like:

NAME                 TYPE SIZE
/gallardo/000001.png file 982.8KiB
/gallardo/000002.png file 821.6KiB
/gallardo/000003.png file 923.3KiB
/gallardo/000004.png file 1.068MiB
/gallardo/000005.png file 1.135MiB
/gallardo/000006.png file 950.3KiB
/gallardo/000007.png file 848.6KiB
/gallardo/000008.png file 1.137MiB
/gallardo/000009.png file 1018KiB
/gallardo/000010.png file 1.119MiB

Here’s a digram of what happens behind the scenes

Split tile rendering

Pat yourself on the back! You have now split frames into tiles, got them rendered independently, and merged them back together, all in less than 250 lines of code!

Make friends first, make sales second, make containers third. In no particular order.

Final Thoughts

Depending on your proclivities, you could consider building on top of this in a variety of ways,

From an engineering for the fun of it standpoint:

You might realize after submitting a blend file that has compositing enabled that it doesn’t render properly, that you’re able to see werid artifacts in the rendered frames, (after uncommenting line 37 on blend_render.py only in the split-blender-framerenderer it works properly in the simple-blender-framerenderer) this is because compositing requires the whole frame to be in the context to work properly, because we render tiles independently and immedietly composite them, it tends to introduce these weird artifacts

To solve it, you could export to OpenEXR format, since the format allows you to store an arbitrary number of channels, blender bakes compositing metadata into this format when you render directly to openexr, you could then manually stitch the tiles together using the compositor’s combine nodes, or automatically stitch the tiles together by creating two more pipelines, the first one would stitch two openexr(s) horizontally, and the next pipeline would stitch two openexr(s) vertically, then you could tweak merger.py until you’re able to merge all the tiles until you get one frame

From a cost optimization standpoint:

If you were running Pachyderm on your own cloud account say for example GCP, you could create a node pool with the new Spot VM instances and assign renderer pods to only be scheduled on these nodes using Node Selector this way if you had a high enough number of parallel jobs, you could save 60–90% of the total cost of ownership in running your own render farm

From a time optimization standpoint:

If you wanted to insanely parallelize the tiles and optimize for least time spent, you could at the end of the splitter pipeline, use an AWS lambda function to render each tile and write the base64 encoded tile back into the merger pipeline using a Spout

From a commercialization standpoint:

In order to package and sell similar DAGs as-a-service, you would need to integrate with a Pachyderm Client such as python-pachyderm or node-pachyderm (I helped! ❤) and programatically put file(s) and create pipeline(s) when the DAG is completed, you can then use the same script to clean up any repos/pipelines that had to be created for that particular job

Pro tip: when you do run something like this in production you’ll quickly realize that the whole system quickly becomes unstable because there are too many pipelines attempting to get scheduled and sometimes that ends up evicting your pachd and etcd pods, so I found it a neccisity to setup a very high Priority Class for pachd and etcd so it doesn’t get pre-empted in favor of job specific pods

Special Thanks

To Packt publishing for giving me a copy of the “Reproducible Data Science with Pachyderm”, I loved reading the book and highly recommend it, especially from knowing how you might want to apply some of these concepts to more production oriented machine learning use cases, I was particularly intrigued by the chapter on Distributed Hyper Parameter Tuning which I found was solved while maintaining a really high bar for Developer Experience which I loved ❤

Thanks for reading!

— Loki