Intel Developer Forum 2016 Snap Lab

Last week, I attended Intel Developer Forum (IDF) for the first time. It was a blast trying VR headsets and test driving the latest hardware kits. But for me, the most rewarding part was seeing Snap, the open source telemetry framework we started at Intel, highlighted at the keynote and teaching a lab jam packed with students eager to try Snap for the first time.

Snap: the open telemetry framework

In the lab, we wanted students to have hands-on experience with Snap and Grafana as well as Snap running on Kubernetes. One of the challenges of running labs at a conference is that internet connections are notoriously unreliable. To ensure the exercises were repeatable, we not only had to bring our own hardware, but also pre-cache packages and deploy a local Docker registry to avoid downloading files over a congested network. The benefit of all this work is we created a reliable and repeatable process building and updating the lab. If you missed our session, jump to the end of this article for a link on how to complete it at home.

One of the first tasks of having a working lab was installing Ubuntu on 50 laptops. This turned out to be trickier than expected because UEFI boot required jumping through a few extra hoops compared to the pxeboot installation process I’m used to. To save someone else the trouble of finding all the gotchas in this process, here’s the example preseed.conf and a list of issues:

class "pxeclients" {
match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
next-server 192.168.0.1;

if option arch = 00:06 {
filename "pxelinux/bootia32.efi";
} else if option arch = 00:07 {
filename "pxelinux/bootx64.efi";
} else {
filename "pxelinux/pxelinux.0";
}
}
  • Make sure the EFI partition is smaller than 256 MB preceded with 1 MB grub partition:
d-i partman-auto/expert_recipe string                         \
boot-root :: \
1 1 1 free \
$bios_boot{ } \
method{ biosgrub } \
. \
256 40 256 fat32 \
$primary{ } \
$lvmignore{ } \
method{ efi } \
format{ } \
.

The last step took lots of trial and error before discovering auto partition wasn’t aware of the EFI partition 256 MB limit, which resulted in a fatal error on next boot:

The attempt to mount a file system with type vfat in SCSI2 (0,0,0), partition #1 (sda) at /boot/efi failed
You may resume partitioning from the partitioning menu.

With those roadblocks out of the way, my home lab churned out laptops at a blistering pace especially once all the software was cached in a proxy server.

For the lab, we pre-cached the Ubuntu Snap package on the laptops by downloading the deb file along with all the plugins. This allowed the students to install Snap even if there was no network connectivity. If you want to test Snap, please use the software repos instead, which have the additional benefit of providing upgrades when we release new versions:

$ echo "deb https://packagecloud.io/intelsdi-x/snap/ubuntu/ xenial main" > /etc/apt/sources.list.d/snap_repo.list
$ sudo apt-get install snap-telemetry
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
snap-telemetry
1 upgraded, 0 newly installed, 0 to remove and 121 not upgraded.
Need to get 7,725 kB of archives.
After this operation, 1,024 B disk space will be freed.
Get:1 https://packagecloud.io/intelsdi-x/snap/ubuntu xenial/main amd64 snap-telemetry amd64 0.15.0-1xenial [7,725 kB]
Fetched 7,725 kB in 2s (2,732 kB/s)
(Reading database ... 61431 files and directories currently installed.)
Preparing to unpack .../snap-telemetry_0.15.0-1xenial_amd64.deb ...
Unpacking snap-telemetry (0.15.0-1xenial) over (0.15.0-1xenial) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up snap-telemetry (0.15.0-1xenial) ...

For now we provide packages for RHEL 6/7 and Ubuntu 14.04/16.04. Additional installation and mirror instructions are available at: https://packagecloud.io/intelsdi-x/snap/install. Reach out if you are interested in packages for other operating systems.

The next step for the lab was to pull and deploy Grafana containers so we can create and watch Snap tasks via the dashboard. We used Ansible to automate all the software installation, and the docker module ensured the containers were loaded and running:

- name: cache container images
docker_image:
name: "{{ item.value.name }}"
tag: "{{ item.value.tag }}"
with_dict:
grafana:
name: grafana/grafana
tag: 3.1.0
- name: run grafana
docker_container:
name: grafana-snap
image: grafana/grafana:3.1.0
network_mode: host
ports:
- '3000:3000'
env:
GF_INSTALL_PLUGINS: raintank-snap-app

The host network mode is necessary so Grafana can communicate with Snap service running on port 8181. If you want to checkout Grafana and already have Snap installed simply use the following docker commands:

$ docker pull grafana/grafana:3.1.0
$ docker run -d \
--name grafana-snap \
--network host \
-p 3000:3000 \
--env "GF_INSTALL_PLUGINS=raintank-snap-app" \
grafana/grafana:3.1.0

The last and most complicated part of the lab was creating a working Kubernetes (k8s) for each student for both session. We used Kargo to create three VMs per student, with a small patch to include socat and nsenter to allow kubectl port-forward command to work properly. The trick to speed up the whole process and reduce network dependency was to create a snapshot so the students could just restore the preconfigured Kubernetes VMs:

$ vagrant snapshot pop
==> k8s-01: Restoring the snapshot 'push_1471994794_8030'...
==> k8s-01: Deleting the snapshot 'push_1471994794_8030'...
==> k8s-01: Snapshot deleted!
==> k8s-01: Resuming suspended VM...
==> k8s-01: Booting VM...

Besides caching the VM, we also couldn’t depend on gcr.io to serve Snap, Heapster, Grafana, and InfluxDB containers. A registry-proxy was created for each Kubernetes node to connect to a single registry service:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube-registry-proxy
namespace: kube-system
labels:
app: kube-registry-proxy
spec:
template:
metadata:
name: kube-registry-proxy
labels:
daemon: kube-registry-proxy
spec:
containers:
- name: kube-registry-proxy
image: gcr.io/google_containers/kube-registry-proxy:0.3
resources:
limits:
cpu: 100m
memory: 50Mi
env:
- name: REGISTRY_HOST
value: kube-registry.kube-system.svc.cluster.local
- name: REGISTRY_PORT
value: "5000"
- name: FORWARD_PORT
value: "5000"
ports:
- name: registry
containerPort: 5000
hostPort: 5000

The registry proxy in conjunction with vagrant port forwarding, allows us to tag and push any docker containers into the registry service and make it available to all Kubernetes nodes:

$ docker tag nanliu/heapster-snap localhost:5000/heapster-snap
$ docker push localhost:5000/heapster-snap

As you can see below this takes several hops from the local system before the image is finally pushed the to the registry pod. The files are stored on a persistent volume so the registry pod can move between any VM and still keep a copy of all the docker images:

In the end there was no network drama at the lab, but I wished there had been a few extra seats for all the people who wanted to try out Snap!

If all this peaked your interest, and you want to try out the exercises, here’s a link to the repository with all the instructions:

We’ve updated the Grafana and Kubernetes sections so they can be completed at home at your own pace. Open a GitHub issue if you run into problems, and see you at next year’s IDF!

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.