Raymond Ferguson
Aug 27, 2018 · 4 min read

Userns LXD, nested Docker, and accelerated GPU graphics

Userns LXD, nested Docker, and accelerated GPU graphics

LXD profile with nested docker and accelerated GPU support for JetBrains toolbox.

Prettier, github, medium.

Why?

There are certainly more use cases for this type of container, but in my case, I’m using it to contain development environments without losing the advantages of direct access to runtime system from advanced tools like pycharm or Idea.

There are certainly other approaches to the issue of getting an advanced IDE in the same process space as your running app including remote debugging and various IDEs that have moved toward client server models or a browser model instead of a direct GUI application. I like those ideas, but I don’t like to waste time figuring out new IDEs that usually have more limited language support and very different UI designs.

For my money, jetbrains gives me a consistent layout and functionality across all of the languages I currently program in with deep support in each language. The trade off is often remote debugging or contaminating your workstation with development jobs that come and go.

This approach eliminates most of those issues and keeps things clean and safe.

Priors

I strongly recommend checking out my turtles article for background, as I don’t intend to recap nested docker, LXD, and cloud-init here.

See Turtles on github, on Devendor Tech for prettiest formatting, and on medium for a walkthrough of nested container support for LXD.

My Environment

The following details of my workstation environment effect the lxd profile I use for turning up a new development environment.

  • Local UID/GID: 1000
  • local user: rferguson
  • Ubuntu 18.04
  • Nvidia GPU
  • Jetbrains toolbox installed to ~/.local on host.

How To

Starting where Turtles left off.

Get and edit the new devUser.yml profile. .. code-block:

rferguson@myhost$ git clone https://github.com/devendor/turtles.git

Edit the profile to suit your environment as noted in TODO tags. .. code-block:

rferguson@myhost$ vim turtles/devUser.yml

Create a lxd profile and import devUser.yml to it. .. code-block:

rferguson@myhost$ lxc profile create devUser
rferguson@myhost$ lxc profile edit devUser <devUser.yml

Launch a new dev machine. .. code-block:

rferguson@myhost$ lxc launch b dev1 -p devUser -p default
rferguson@myhost$ lxc exec dev1 -- tail -f /var/log/cloud-init-output.log

Map a project dir if desired. .. code-block:

rferguson@myhost$ lxc config device add dev1 myproject \
disk source=/home/rferguson/code path=/home/me/code

Launch pycharm. .. code-block:

rferguson@myhost$ lxc exec dev1 -- runuser me -c "pycharm ~/code" &

Final thoughts

You wil have to do some initial setup unless you also map your IDE setting directory, but it can be nice to use one of the settings sync options of this particular ide and keep per instance settings separated and use the various settings sync options to syncronize or archive IDE settings on a per-project basis.

Mapping user rferguson to user me has pragmatic value since I can now image the entire dev environment and give it to the new guy or push it somewhere else and run it under a different local host user. Me@dev1 is also a shorter PS1 for cleaner looking docs.

There is a lot that isn’t covered here. The docker nesting are already in turtles and there is some good information on the details of GPU features of LXD containers from existing sources.

Happy coding!

devUser.yml Profile

Checkout the devUser.yml on github for the maintained version.

name: devUser
description: LXD profile with nested docker and accelerated GPU support for JetBrains toolbox.
config:
environment.LANG: en_US.UTF-8
environment.LANGUAGE: en_US:en
environment.DISPLAY: :0.0
environment.XAUTHORITY: /home/me/.Xauthority
nvidia.runtime: "true" # TODO only if you have an nvidia GPU.
raw.idmap: | # TODO Set your UID/GID
both 1000 1000
linux.kernel_modules: ip_tables,btrfs
security.nesting: "true"
security.privileged: "false"
user.user-data: |-
#cloud-config
output:
all: '| tee -a /var/log/cloud-init-output.log'
package_update: true
package_upgrade: true
runcmd:
- set -xe
- curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
- apt-get install -y apt-transport-https curl
- add-apt-repository
"deb [arch=amd64] https://download.docker.com/\
linux/ubuntu $(lsb_release -cs) stable"
- apt-get update
- apt-get install -y
docker-ce
docker-compose
vim
git
squashfuse
- usermod -aG docker me
- systemctl start docker
- docker image pull hello-world
- docker run --rm hello-world
- apt-get install -y
x11-apps
mesa-utils
alsa-utils
libxtst6
libgtk-3-common
libswt-gtk-3-java
libnvidia-gl-390 # TODO validate appropriate gl library.
- "export DISPLAY=:0.0 XAUTHORITY=/home/me/.Xauthority"
- nvidia-smi
- runuser me -c "glxinfo -B"
- runuser me -c "glxgears -info" &
- sleep 12
- killall glxgears
write_files:
- path: /etc/rsylog.conf
content: |
module(load="imuxsock")
*.* @log.virtdmz
owner: root:root
permissions: '0644'
- path: /etc/docker/daemon.json
content: |
{
"hosts": [
"fd://",
"tcp://0.0.0.0:2345"
],
"storage-driver": "btrfs"
}
permissions: '0644'
owner: root:root
- path: /etc/systemd/system/docker.service.d/override.conf
content: |
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd
permissions: '0644'
owner: root:root
- path: /bin/pycharm
permissions: '0755'
owner: root:root # TODO Check your install path.
content: "\
#!/bin/bash\n\
exec $( ls -1c ~/.local/share/JetBrains/Toolbox/apps/\
PyCharm-P/ch-0/*/bin/pycharm.sh | head -1) $@\n"
users:
- name: me
groups:
- adm
lock_passwd: true
shell: /bin/bash
uid: 1000 # TODO Swap to your numeric UID/GID
gid: 1000
ssh-authorized-keys: # TODO Add your own keys
- "ssh-rsa AB3NzaC1yc2EAAAADAQABAAABAQDPS4YhPW5BkRbYkazw\
oUu43rMsGLjL/gyD5XNJntdSuENYWH rferguson@booger"
sudo:
- ALL=(ALL) NOPASSWD:ALL
devices: # TODO Swap in your home dir path.
Xauthority:
path: /home/me/.Xauthority
source: /home/rferguson/.Xauthority
type: disk
nvgpu:
type: gpu
uid: "0"
gid: "0"
x11:
path: /tmp/.X11-unix/X0
source: /tmp/.X11-unix/X0
type: disk
melocal:
source: /home/rferguson/.local
path: /home/me/.local
type: disk

Devendor Tech

Technical publications from Devendor Tech

Raymond Ferguson

Written by

Devendor Tech

Technical publications from Devendor Tech

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade