Open-Sourcing Shoelaces: A Tool for Unmanned Server Bootstrapping

Published in

Cisco® ThousandEyes® Engineering

6 min readMay 14, 2018

Author: Raúl Benencia, Senior Site Reliability Engineer at ThousandEyes.

Shoelaces is a tool that provides the glue for the unmanned bootstrapping of a server using a lightweight mechanism. That means you can boot a remote server either manually or automatically without local assistance. Here at ThousandEyes, we have been using Shoelaces for almost two years now as a key component of our infrastructure management environment.

Over the last few months, Shoelaces has reached a state of maturity, so we thought why not make it available to a larger community. Today, we are happy to announce the open-sourcing of Shoelaces under the Apache License Version 2.0. This blog post will explain its core functionality, how it works, how to get access to it and how you can contribute to improving it.

Overview

Shoelaces is implemented as a web service. This means that it listens for HTTP requests coming from servers that are booting, responding with a boot script based on environment heuristics such as the source IP, the subnet it belongs to, the hostname or optional user input.

In order to have a working Shoelaces environment, DHCP and TFTP servers are required (usually living on the same machine) along with a server for hosting Shoelaces (which can also co-reside with the DHCP, TFTP servers).

Why a New Tool?

There are some tools in the open source community with similar objectives to Shoelaces. MAAS, for example, has a more complex setup and lacks some key functionality that Shoelaces supports. MAAS requires that you have to setup several moving parts, including a PostgreSQL database. Shoelaces, on the other hand, requires fewer steps to start working. Setup involves installing the package, then tweaking some values in the configuration file (or just send those values as parameters or environment variables), and Shoelaces is ready to start serving your infrastructure.

Another tool that is available is pixiecore. Its functionality is similar to Shoelaces, to boot a remote server, but it solves the problem in a different way. Its approach is to control ProxyDHCP and TFTP servers and act more as the glue between PXE and a user-implemented service to retrieve the actual script to boot. Shoelaces currently implements these features, providing an off-the-shelf solution for booting a network server using user-provided, customized scripts that can be turned into templates for greater automation.

How It Works?

Figure 1 shows a high-level call flow of how an external server contacts Shoelaces. We can see that as soon as the server performs a network boot using PXE, Shoelaces instructs the machine to switch to an iPXE ROM. This happens so that subsequent HTTP requests can be sent to Shoelaces since the PXE interface does not directly support HTTP.

Figure 1: A high-level overview of how a server contacts Shoelaces.

So, when a server boots, the DHCP server will instruct it to retrieve an iPXE ROM from a TFTP server. When the booting server receives the iPXE ROM, it will chainload into it and trigger a new DHCP request. Finally, the server will detect that the request comes from an iPXE ROM, allowing it to respond with an HTTP URL. This URL, as you may have guessed, will be pointing to Shoelaces.

For example, in the ISC DHCP server, you can achieve the described behavior by using the following configuration:

next-server <your-tftp-server>; 
if exists user-class and option user-class = “iPXE” { 
    filename “http://<shoelaces-server>/poll/1/${netX/mac:hexhyp}”; } else { 
    filename “undionly.kpxe”; 
}

The file undionly.kpxe is the iPXE ROM that is downloadable from the iPXE website. The ${netX/mac:hexhyp} string represents the MAC address of the booting server. iPXE will be in charge of replacing that string for the actual value. In the end, Shoelaces will receive a request to /poll/1/, and you will see the magic happening! Shoelaces will then decide what to do based on the MAC address, the IP, subnet or hostname (resolved using rDNS).

Booting Methods

There are several ways you can boot your servers using Shoelaces, manually, where a web front end will assist you, or automatically, where you can boot your servers in an automated fashion using pre-configured parameters.

Manual Boot

Shoelaces has a web UI that will allow you to manually choose a task and set its parameters.

Figure 2: Shoelaces is waiting for a server to boot.

When the server contacts Shoelaces, the web front end is updated to show the following:

Figure 3: Select the server you want to boot. In this example, server with MAC 1e:a7:de:ad:be:ef and the debian.ipxe task was selected.

Once the fields are completed, simply click Boot! and enjoy not having to go to the data center to boot your servers!

Automated Boot

Shoelaces also provides a set of mechanisms to entirely automate the bootstrapping of a server. This means that once it’s set up, no user intervention is required.

Hostname-based booting

The first automated booting mechanism is based on the server’s hostname. When the server boots and queries Shoelaces, the application will trigger a reverse DNS lookup for its IP. If successful, it will attempt to match the hostname to a set of user-configured regex. Let’s analyze an example:

hostnameMaps: 
  — hostname: (web|db)\.example\.com
    script: 
      name: debian.ipxe 
      params: 
        release: stretch

This YAML script tells Shoelaces to use the debian.ipxe task with the release parameter set to stretch whenever the reverse-resolved domain name matches either “web.example.com” or “db.example.com”.

Network-based booting

The second automated booting mechanism relies on the IP address of the booting server. Shoelaces extracts the IP address from the received HTTP request and attempts to match it to a set of user-configured subnets. For example:

networkMaps: 
  — network: 192.168.0.0/24
    script: 
      name: debian.ipxe 
      params: 
        release: jessie

This YAML script tells Shoelaces to boot using the debian.ipxe task with the release parameter set to jessie whenever the booting server is part of the 192.168.0.0/24 subnet.

Customizable Parameters and Templates System

Shoelaces is a tool written in Go, and as such, it uses Go’s template system to generate the dynamic configurations provided by its users. The Shoelaces source code repository offers template examples that can be used as a starting point for customization.

An example is worth a thousand words, so let’s jump right now to the template we used in the previous figure:

{{define “debian.ipxe” -}}
#!ipxe echo Debian {{.release}}
set mirror http://ftp.debian.org/debian/dists/{{.release}}/main/installer-amd64/current/images/netboot/debian-installer/amd64 kernel ${mirror}/linux auto=true priority=critical initrd=initrd.gz preseed/url=http://{{.baseURL}}/configs/debian.preseed ${linuxargs} initrd ${mirror}/initrd.gz 
boot 
{{end}}

In Figure 3 we saw that, after choosing the debian.ipxe task, two input fields were summoned for the parameters release and hostname. These parameters are dynamically generated by scrapping the template for the selected task. Then, when the server finally makes the request for retrieving the boot script, Shoelaces will grab the chosen template and interpolate the variables in order to serve the requested file.

In the same way, we created the above debian.ipxe script, you can provide any kind of plain text file. Here at ThousandEyes, we find it particularly useful with cloud-config files, preseed files and shell scripts.

Events

Shoelaces features an event system that gives visibility into what’s going on during the bootstrap process. The event system will show you when:

A server polls for a script for the first time.
A user chose a script for a server that just polled.
A server boots with a script served by Shoelaces. In this case, it will tell you whether the script booted manually or automatically using one of the mappings methods.

Figure 4: A sneak peek of the Shoelaces events view.

When a modern server infrastructure starts scaling, booting new servers manually becomes a tedious, repetitive, and resource consuming task. Automating this repetitive process is the natural thing to do, and Shoelaces eases the task by providing the means for booting servers automatically without hands-on server assistance.

If Shoelaces piqued your interest, feel free to download the latest version. If you find a bug or want to suggest improvements, don’t hesitate to create an issue in the project or even better yet, send a pull request. Happy booting!

Originally published at blog.thousandeyes.com on May 14, 2018.