Behind-the-scenes of ChainZoku — Episode 5: Not Your Usual Render Farm

8 min readJun 20, 2024

GM! EKCO here again, co-Founder of Miinded & lead dev on ChainZoku, which I’m going to talk about again today. This article follows on from the previous one: Episode 4 — Expectations VS Reality. Make sure you read it first!

Today we’re going to talk about the real-time renderer, simply put…

If you’re new here, make sure to read the previous chapters and the very thorough Guide to ChainZoku, which will tell you a lot about the project.

To make this article a bit more interesting, I’ve decided to interview myself using the questions we asked ourselves throughout the development process, and the answers we found.

Please note that this article is more technical and will require more attention.

What’s the best solution?

As seen in previous chapters, a Zoku can be modified at any time by the holders themselves. When a Zoku is modified, we need to re-generate the Zoku’s 2D image in order to display it on the different marketplaces as well as the My Zoku page. The goal and challenge here is to do it in as little time as possible, so that there are as few differences as possible between the NFT and its matching image.

Creating an image from 3D Assets is much more complicated than with 2D Assets. As all the assets have been created in Blender, it simply made sense to use Blender to create the renderings.

While brainstorming on the project, we explored several avenues for generating Zoku images in real time:

Paying a human to do the rendering manually? An extremely simple solution that has no advantages, is very expensive, no fun, and very time-consuming. We needed to create a tool that would allow us to create a Zoku while minimizing human error. So obviously, this idea was instantly abandoned (never actually considered, in case you missed the joke).
Buy a beast of PC dedicated solely to rendering, and put it in the home of a ChainZoku team member with a good Internet connection? This was the cheapest solution, but involved other problems: requiring a physical presence in the event of a problem, difficulty in updating, risk in the event of a power or internet cut… We considered it if no other solution was possible.
Using a “render farm”, i.e. large servers in a datacenter dedicated solely to rendering an image. This solution was very expensive, around €1 per image (i.e. €20,000 for the initial generation of 10,000 Zoku with 2 images). What’s more, as the servers were used by a huge number of people, it was very likely that waiting times for renders would skyrocket.
Create our own “Render Farm”. LMAO! Yeah, sure, let’s to create our own Render Farm! It’s a good one…

Creating our own Render Farm

First, we looked for a datacenter that offered servers with large, high-performance graphics cards at a reasonable price. We opted for Scaleway, our French hosting provider based in Paris. At the time, they only offered one type of server with an NVidia RTX 3070 8GB, 8-core CPU, 16GB RAM, running Linux (Debian). The price wasn’t cheap: €1.20 / hour, or €865 / month.

But with this solution, we can manage our own rendering queue and optimize our scripts to perfection. And if we need to deploy a second server in case of heavy demand, we can do so in a matter of minutes.

We’ve done some benchmarking on the server to get an idea of average rendering times and evaluate the total monthly cost.

It takes between 30s and 3min to render the 2 images of a Zoku (portrait and profile). That’s an average of 40 Zoku per hour, or €0.03 per Zoku. Totally acceptable. To do better would have been far too expensive. To render the initial 10,000 Zoku, we had planned to use 2 rendering servers to generate the collection in less than 3 days. In the end, we only used one server.

24/7 rendering

Having our own rendering server is cool, but expensive. It could render 28,800 Zoku per month at full load and justify the cost of 0.03 € / Zoku.

BUT…

There aren’t 28,800 Zoku to return per month…. In our busiest month, we returned around 2,500 Zoku, and on average we render 350 Zoku per month. This gives an average price of 1.7 € / Zoku (just a small difference of 5600%).

There’s no point in keeping the server running 24/7 when it’s only used 9 hours a month on average.

So we developed a script that starts the server when there’s a rendering request, and stops it when the rendering is complete and the queue is empty. When the server is off, nothing is charged. As a result, we were able to divide the cost of the server by 5, to 0.35 € / Zoku. This is still 10 times more than the initial target, but more bearable in the long run.

Communication is the key to a good relationship

What better way to explain an intra-server communication system than with a good diagram?

Glossary :

ChainZoku Backend: the server where all raw data is stored.
RabbitMQ: the queue management system.
Miinded Render Farm: Miinded’s 3D render management server.
Render Server: The 3D rendering server(s).

To summarize:

The user makes a modification to his Zoku, ChainZoku Backend verifies, processes and stores the information.
ChainZoku Backend contacts Miinded Render Farm to request a render with all Zoku information formatted in JSON.
Miinded Render Farm processes the information and adds a render to the RabbitMQ queue with the Zoku information.
Render Server constantly asks RabbitMQ if there’s anything to render. If so, it retrieves the Zoku information stored in RabbitMQ and launches its rendering mill (I’ll explain this later).
Once rendering is complete, Render Farm Server contacts Miinded Render Farm to let it know that rendering is complete. It then uploads the images to ChainZoku’s CDN, which stores, compresses and makes them public.
Finally, Miinded Render Farm confirms that the rendering is complete, deletes the task from RabbitMQ and contacts ChainZoku Backend to tell them that the rendering is finished.
ChainZoku Backend then contacts OpenSea to tell it that the image for this NFT has changed, OpenSea downloads the new image and displays it in the list.
Finally, the user is notified on the site that the rendering is complete, and the site uploads the new image.

Steps 1 to 4 take a total of around 0.5 seconds. Step 5 takes between 30s and 3min depending on the Zoku’s complexity. Steps 6 and 7 take a total of around 5 seconds. Finally, step 8 can take between 0 and 15 seconds.

In the end, from the moment the user clicks on “Validate” to the moment the user is notified that the image has been rendered, it takes between 1 and 4 minutes. We think that’s fair. Definitely hard to beat this without having the costs go up way too high.

The rendering mill

To render a Zoku, we have developed a Linux service, which we installed on the Render Servers. This service consists of several different Python scripts that are used for various tasks:

File updates
The ChainZoku team is constantly adding new Assets to equip on your Zokus. The Render Servers must therefore always be up to date with the new Assets. To do this, we have a storage server dedicated exclusively to Blender scenes and textures. This script queries our storage server to find out whether Blender files have been updated, and downloads them locally to the server if necessary. It does this every 20 min or before each render.
Check queue
This script constantly asks RabbitMQ if there’s a new job to process. If so, it triggers the next scripts in line. Otherwise, it waits.
Launch Blender
The script launches Blender from the command line, without an interface.
Zoku asset management
To generate the Zoku, we created an asset-free Blender scene with only backgrounds and Lights. Using the Zoku information, the script automatically imports each asset present in the other Blender files into the blank scene.
Launching the 3D render
Once the assets have been loaded, activate the first camera (Profile) and launch an EEVEE render (this is the name of the render engine in Blender). Then switch to the second camera (Portrait) and launch a new render.
Image upload
Finally, if all goes well, we retrieve the 2 freshly created images and upload them to the ChainZoku CDN.

Hardware limitations

If you know anything about computer hardware and 3D, you’ll know that a good graphics card can speed up 3D rendering. But that’s not all. The amount of RAM is also very important.

Scaleway’s servers have “only” 16 GB of RAM. For classic use, this is sufficient, but for ChainZoku it’s not enough.

When we render a 3D scene, we import assets from other Blender scenes into a single Blender scene. This means we have to open and close a large number of scenes, which consumes a lot of RAM. Opening a ChainZoku scene can consume up to 250 GB of RAM! Our servers have only 16Go…. This leads to severe slowdowns at the best of times, and even outright server crashes.

To overcome this problem, we use swap partitions on our hard disks. In layman’s terms, we store data normally stored in RAM on the server’s SSD. This reduces performance, as hard disks are slower than RAM, but avoids server crashes.

It’s amazing what you can do with a few good lines of code, isn’t it?

In the next article, I’ll tell you all about gas costs and their benefits on an NFT project (or not).

Don’t hesitate if you have any questions, I’ll be happy to answer them.