Setting up a HW accelerated desktop on AWS G2 instances
GPU access is becoming a more and more common requirement in a growing number of use cases.
Whether we’re doing some cool 3D rendering, building Skynet with the latest machine learning tech, or you-name-it, we’ll have to deal with a fundamental question: “Would this run on my GPU?”
In my case, given that the laptop I work with is a MBP mounting an Intel Iris Pro card the usual answer is “nope”.
Thinking outside your box
Until a few years ago the only available option was buying a new graphic card (or a new laptop), waiting for it to be delivered, then setting it up.
That option now still stands, and it might still be the best option for some use cases, but since AWS made hardware accelerated instances globally available, GPUs became de-facto a commodity.
Apart from the obvious satisfaction gushing from finally running the software we wanted, renting a GPU box comes with all the perks of running on AWS (replicability, scalability, competitive pricing, monitoring, huge ecosystem, just to name a few), which make this solution extremely appealing.
Pricing
At the time writing a g2.2xlarge box comes at $0.702 per hour in Ireland, and $0.650 in US-East and US-West-2.
(source: https://calculator.s3.amazonaws.com/index.html)
Setting up a hardware accelerated box
At Plumbee GSN Games we rolled our own GPU enabled Ubuntu box featuring
- Ubuntu 16.04 and Mate desktop
- Nvidia drivers
- TurboVNC
- VirtualGL
- Docker and Nvidia Docker
Why do we need VirtualGL?
The problem
VNC sets up a desktop environment on display :1, but such display is not hardware accelerated.
Hardware acceleration is available to display :0 only, therefore:
- a program running on display :0 can access the GPU, but is not captured by VNC
- a program running on display :1 is grabbed by VNC but has no HW acceleration
The solution
VirtualGL comes to the rescue allowing a “split rendering” (GLX forking).
It intercepts GLX calls and redirects the content rendered by the GPU from display :0 to display :1, so that that content can be captured by VNC.
VirtualGL installs the “vglrun” command that allows the GLX interposer to be initiated (e.g.: run “vglrun firefox” to start firefox on split rendering).
Why do we need Nvidia Docker?
The nvidia-docker CLI wraps Docker’s APIs. It automatically detects and links the host machine’s nvidia devices (/dev/nvidia) and drivers into each container created via the nvidia-docker command, enabling them to access the underlying host nvidia GPU.
It also provides a REST interface other utilities (such as nvidia-docker-compose) can query to obtain info about the available nvidia drivers and nvidia devices.
Note that it’s still possible to link all the needed volumes manually using plain Docker. So in case we prefer not to install yet another CLI we don’t have to.
Sharing is caring
Before rolling our own setup we spent some time searching for a similar one, unfortunately without luck.
We hope to save you the time to do this ground work, so we made our box setup public, feel free to use it, fork it, adapt it to your needs.
You can find the repo here. Enjoy.
Piergiorgio Niero
Director of Engineering @GSN London
