Leveraging Google Compute Engine for Low-cost GPU Rendering (and Other Fun Exercises in Patience)
With my game project Elliot & Ko, I tend to require a lot of upfront GPU resources and time. Rather than deal with the traditional pains of making things look pretty on various low end devices in real-time, I’ve opted to pre-render extremely high quality (and accordingly resource demanding) renders which are then composited in real-time as 2D assets with fake 3D physics.
That’s all quite well and good save for the fact that my GPU resembles a potato. After borrowing the better hardware of a friend who, with a certain sense of irony, works as a developer Google Compute Engine, I decided to go straight to the source and setup a low-cost remote system on cloud hardware.
Use Cases at a Glance
- Save time and free up PC resources by remotely processing demanding jobs (while saving power consumption)
- 3D rendering or baking, particularly in Blender
- High GPU yields like After Effects renders, various Unity tasks (pre-calculated lighting etc.)
Hardware & Pricing
If you didn’t already know, Google offers $300 of credit when you first sign up for its cloud platform. Pair this with the fact that earlier this year, Google introduced “preemptible” GPUs at a reduced cost, and you can get several thousand dollar hardware running at cents/hour, technically free under trial for the first… who knows how long.
The costs and availability vary by region, but ultimately it came out cheaper than Amazon’s equivalent of “spot instances”, along with the tempting credit. Still, if you’re interested in AWS everything outlined here applies there, and a brilliant and similar draft here goes into details for an AWS setup.
Note that both services price per hour, rounded up. They’re both very versatile with their configuration options as well, so often times it’s cost and time efficient to go for a much more powerful (and costlier per/hour) setup that will complete in a fraction of the time. It’s quite easy to play around and adjust to your liking.
In this article, we’ll also be using Linux on our instances for two reasons: it’s cheaper and (generally) faster. This is particularly apparent with certain programs like Blender on Windows 10 which goes through something called the Windows Display Driver Model, with significant overhead. That said, someone wrote a fantastic article on bypassing this particular problem here.
Getting Started
The first step is to register at Google if you haven’t already. Voilà! $300. You have exactly 365 days to spend it. From there, head to the instances control panel and select Create Instance, where you’ll be greeted with the following options menu:
Name your instance something memorable or distinguishable, like GCE Rendering or so. After that, click Change to configure the operating system and size. I opt for Ubuntu 16.04.5 LTS as it is what I’m used to, but if you’re more familiar with another Linux distribution, have at it! You’ll also need to set a disk size — storage is cheap but you are still charged for it, so choose what you need with a little room to spare; I opted for around 15 GB or so, none of which I’ve come close to fulfilling.
Next we can set the machine types and the region; recall that not all hardware is available for all regions, and that pricing varies. Review over at Google, and configure accordingly; you can effortlessly swap out GPU, CPU and RAM on your VM later, but changing the zone requires you to create a new VM instance altogether. For my purposes, I went with a US Eastern region which gave me access to the Tesla K80 and beefier Tesla P100 which are both more than enough for my own purposes.
If you’re looking for an exact specification of machine type — I can’t help you! It’s entirely dependent on the job at hand and the resource constraints. Some jobs rely heavily on CPU and require lots of memory, other jobs could make great use of a 2x or 4x GPU setup. Indeed my instances are adjusted all the time based on profiling for any given job, and that’s really the best way to approach things; evaluate your jobs as they happen, your usage, limits and execution times, and adjust as needed. I’ve added a Profiling section below, but overshooting for a single hour is hardly a costly error when we’re talking pennies.
Next, click Management, security, disks, networking, sole tenancy to expand some additional options.
Startup script
Paste the following for the startup script (or the equivalent commands for your chosen Linux distribution, Google has similar instructions here). This will install CUDA among other things during startup which is an absolute must for GPU rendering on Nvidia cards. You’ll notice I’ve included — but commented out — the last line for explicitly setting the application clock speed, which varies by hardware. Nvidia has a guide for determining the available speeds using nvidia-smi -q -i 0 -d SUPPORTED_CLOCKS here. When you’ve found your maximum clock speed, you can adjust the startup script to match or just run the command manually, but this isn’t strictly necessary.
Turn this option On! I hope it’s obvious why at this point. Forgetting to do so will yield the normal, much higher rates.
Automatic Restart and more
This is up to you, but personally I have no need for automatically restarting during a failure. If a remote job stops halfway, I don’t have any benefit from restarting and having hardware sitting by idly without my notice.
…and we’re ready! Click create. Once it’s ready, you can then toggle it on and off using the Start and Stop buttons atop the console. Note that you’ll be charged hourly while it’s on, so don’t forget to manage this explicitly. As expected, it’ll need to be on for you to connect.
Connecting to your Instance via Command Line and SFTP
Once an instance is up and running, you’ll be given an external IP you can use to connect. Note that this changes between startups on preemptibles.
You’ll need command line SSH access to install and run your programs, and SFTP access to send your job files to the remote machine and download the resulting output. This article assumes some familiarity with both, but if you get stuck just send me a message on Twitter and I’ll try to expand it.
Google seems hellbent on advocating its own console for connecting here, but I prefer to use external tools which they’ve detailed instructions here. On Windows this will typically be PuTTy or Cmder for SSH and WinSCP for SFTP.
Blender
And finally, we get to a use case! This part’s fun. Go ahead and SSH to your machine and we can install Blender. You won’t have to mess with drivers and such since the startup script handled that for us.
I’d recommend not using apt to install Blender, because at the time of writing the packaged build of 2.79 had some major rendering artifacts in my experience. You can opt for the stable releases at https://download.blender.org/release/ or the cutting edge (new features, less stable) at https://builder.blender.org/download/. Or even grab a few and try for different results. I had the least issues with the former.
cd into a directory of your choice (I just tend to use render/) and then run the following, substituting the URL as appropriate:
wget ‘https://download.blender.org/release/Blender2.79/blender-2.79b-linux-glibc219-x86_64.tar.bz2'
tar -jxvf blender*.tar.bz2Upload your .blend file(s) somewhere easily accessible, and to reduce upload time overall, set Compress File on Save and re-save.
Blender lets you run headless commands with or without a blend file, and you can even configure additional settings like disabling unnecessary features. The full command-line documentation is here, but you’ll notice we can pass in a python script to execute some actions for us! I tend to make heavy use of this with two files:
- An
adjust.pyfor making quick adjustments such as number of samples, toggling a layer (or whatever the 2.8 equivalent ends up being) and more without rendering it. This means you can upload a file and tweak its values on the fly without having to re-upload for every change. - A
render.pyfor setting (or overwriting) global settings and performing the render job. This is also crucial for ensuring GPU rendering is enabled regardless of the preferences you had when you saved the file locally.
As an example, in 2.79 I make heavy use of layers to render out separate scene elements at a time. Occasionally I like to up the sample count, turn off/on some layers, or adjust other settings, like lighting and denoising. To get the python property for a specific setting, you can right click it in Blender and select Copy Data Path.
Then, to update the blend file with the new settings, I’d simply run:
installdir/blender -b blends/projectname.blend -P “scripts/adjust.py”
using -b to run in the background and -P to pass in the python file to execute.
Finally, the render.py:
…which we can execute with
installdir/blender -b blends/projectname.blend -P “scripts/render.py”
Note that it prints the devices and their enabled status at the start. Pay very close to attention to this! You can comment out line 23 with a # if you need to as a verification. If your desired GPU devices are not visible or aren’t enabled, either you or Blender has done something wrong. The last thing you want to happen is to go through all this trouble only to use CPU rendering on a GPU powerhouse!
You’ll get a constant feed of output in your shell, including an estimate of remaining time and more. If you want to keep the render running even if your local machine disconnects, consider using Tmux! It’s super easy to use; type tmux, run the command as usual, then ctrl-b, d to detach. tmux attach to re-attach if need be.
When the rendering’s done, it’ll let you know! And you’ll get all the files according to your node setup, which you can then transfer via SFTP back to your local machine. Be sure to power off the instance when finished!
Unity
I’ve not explored this option myself as I don’t have need for it, but it could certainly be handy for expensive processes, such as baking lighting using the Lightmapper. Note that only the Progressive GPU Lightmapper leverages GPU acceleration, with the standard running on CPU only (still, would be a nice use case for CPU Compute Instances rather than GPU). The Progressive GPU Lightmapper appears first in Unity 2018.3.
The workflow would be similar to Blender’s, using a combination of command line arguments and scripting (in C# rather than Python) to run some expensive jobs and save. You can check out the command line options here, and there are a few options like -executeMethod which allow you to run editor code. From there you could consult the documentation and figure out what actions you’d like to run by script. Be sure to save your scene file at the end of your script!
I’ve never run Unity on Linux so I can speak to its performance or even how well it works between sharing projects saved in different platforms, so it might be worthwhile to start a GCE instance using your local operating system (and change the CUDA driver startup script accordingly!)
Profiling
For CPU and memory monitoring on Ubuntu, I tend to like Glances!
For GPU, it’s dead easy with nvidia-smi, like watch -n 1 nvidia-smi or so.
At that point I just check the usage and make sure there is no significant peaking or underuse!
When it comes to Blender and its Cycles renderer (as I’ve not yet explored Eeevee) I tend to have a little idea of baseline GPU/CPU/RAM to throw at a render given the scene elements. In any case, I still profile anyway — directly on the instance rather than locally so as to not make false assumptions based on my local environment.
For that, I just quickly change my render.py or adjust.py to decrease the overall resolution and, for excessively large scenes, maybe set the render region to a smaller area (maybe one with a lot of density or so) and then run for a couple of minutes, just enough to get an idea of monitoring and to also pre-emptively discover any oddities in the particular blender build I am using. I hinted at discrepancies earlier, and one of the unstable Linux builds tended to mysteriously lighten certain tiles when doing a mixed GPU+CPU render.
So I reiterate: test and profile often.
Closing Thoughts
I hope this was insightful, and maybe saved you a few headaches and hours like it did for me! If you have any questions or if you’d like to check out other little things I document, drop me a message or follow me on Twitter at @cloutsocks
Happy rendering!
