Flexo — a Car Deployment Platform

By: Mathias Gug and Sebastian Brannstrom, Level 5 Software Engineers

After one and a half years of bootstrapping Level 5, no two cars were exactly alike from a software perspective, but all were quite similar. Engineers and operations were spending more and more time learning about the subtle differences and unique environments of each car. Taking a page out of the Cloud book, we built a car deployment platform named Flexo to ensure the software for our car fleet is built automatically and an identical environment is deployed to every autonomous vehicle (AV). Nowadays, every car uses the same disk images. Engineers and operations have stable environments to conduct their work on.

The challenge

At Level 5, we operate our own fleet of AVs built in-house by our hardware team. As it’s still early days in AVs, this fleet must serve two very different use cases. One is as a stable platform for our operations team to conduct missions like shuttle services and data collects. The other is as a development platform for our software engineers to keep improving our stack.

The requirements from these two groups of users are quite different: operations wants a turnkey appliance-like platform with minimal options and highly predictable behavior, while engineers want maximum flexibility so they can quickly iterate.

We started with a simple configuration management system built on Salt, that configured the vehicle for the type of user who needed the vehicle. An operations mission would start by cloning our software, re-provisioning the High Performance Computer (HPC) in the car, and building from source. Likewise, an engineer would pull their own branch, build, and re-provision the HPC for their needs.

This worked OK in the early days because the development was early and the cars were not always busy. So, we could use the car HPC to build our software stack. Once the AV fleet scaled up and our software stack complexity increased it meant lots of wasted time re-provisioning and building on dozens of cars whenever the mission type changed. Clearly, we needed something better!

Disk swapping versus network transfers

Early on at Level 5, we debated the merits of disk swapping versus network connections for offloading recorded sensor data. For the data sizes at play — easily in the terabytes per vehicle per day — we chose disk swapping. There were a number of reasons for this, including mechanical rigidity of high speed Ethernet cables and connectors, and that we couldn’t guarantee high-speed network ports in our garages.

Most importantly, turnaround time for the vehicles is critical for the productivity of the fleet. Swapping out a drive can be done in 30 seconds anywhere while recycling a 4TB drive takes on the order of 10 minutes with a 50 Gbps Ethernet connection. Moreover, garages can be built and moved to any location quickly. Having access to high speed networking is not a hard requirement given that disks can be mailed at first.

That’s why we adopted a process to swap data drives in and out of cars. We decided on a similar method to deploy software on boot drives.

The solution

Simply put, Flexo builds boot drives. It’s a hardware and software solution that lets us “burn” dozens of identical hard disks with the complete set of software, all the way from the Linux boot loader to the specific version of the self-driving car software that we build everyday, configured for the mission type.

At a high level, Flexo is a standard Ubuntu 18.04 server system built using:

• Docker for building images
• systemd units for burning images
• Python scripts for building and managing images
• BASH to glue it all together
• Telegraf and Wavefront for monitoring

A schematic view of Flexo

The Flexo deployment platform transforms the source code hosted in git repositories into disk images that can be booted on computers located in our AVs. It can be broken down into the following functional components:

• Image builder: Responsible for building bootable file systems from source code stored in git repositories
• Image burner: Takes these bootable file systems and burns them to multiple hard drives
• Image selector: Chooses which image to boot on a car
• Overlay Root: Provides an ephemeral root filesystem
• Hardware platform: What this system runs on

Image builder

The main tasks of Flexo are building and managing images. An image is simply a tar archive of a complete bootable file system that the image burner will later use to “burn” boot drives. These images are commonly on the order of 100s of GBs as they include high-definition (HD) Maps.

Lyft has used containers for many years, so the natural tool of choice for building images is Docker. Docker defines a mature and flexible language and toolchain for building container images. In our use case, we’re only using Docker as a tool to build the image, not to run it. So, we need to manually install a bootloader (grub), a kernel, and its initial ramdisk inside the container.

We use several different Dockerfiles depending on the configuration we want for the image. This is the rough content of our Dockerfiles:

1. Start with a standard Ubuntu 16.04 docker image
2. Install a linux kernel and grub so that the image can boot on bare-metal (docker images don’t have a kernel as containers shares the kernel with the host at runtime).
3. Clone the relevant software repositories
4. Configure the filesystem using SaltStack for the configuration desired, specifically
5. Set up users and permissions
6. Install all Ubuntu packages we want
7. Install and configure systemd units and udev rules to find vehicle specific data (see below)
8. Install system build dependencies
9. Build our own software for the configuration we want
10. Export the image as a tar file

Today Release Managers manually start building an image whenever they have approved a new release for production use. We’re looking at ways to streamline this.

The image building component is decoupled from the host it runs on thanks to the use of container technologies.

Vehicle-specific data

The images are built to be entirely vehicle-agnostic, since all our vehicles of a particular generation are built to be identical from a hardware perspective, and even generational differences are currently handled at runtime.

However, there is some vehicle-specific data we need to keep, including the vehicle’s identity (so that we can trace recorded data and logs), its cryptographic keys and certificates, and its calibration parameters.

Since any boot drive created by Flexo can be installed into any vehicle, we added local storage that is never removed from the car in the form of a USB stick.

Image burner

The image burner component is responsible for building bootable file systems onto multiple hard drives. The component needs to handle 20+ hard drives that are plugged in and out by human operators when the system is on.

Each hard drive is managed by a unique sync_all_image_to_disk@sd* systemd unit.

Given that hard drives can be added and removed at any time we leverage udev — the generic device manager used by the linux kernel. We favored udev over a cron job so that we could start image burning processes as soon as disks are plugged in. We use a udev rule to start a systemd job for each hard drive:

ENV{DEVTYPE}==”disk”, ENV{ID_PART_TABLE_UUID}==”00000000-*”, TAG+=”systemd”, ENV{SYSTEMD_WANTS}+=”sync_all_images_to_disk@$kernel.service”

The rule uses the $kernel udev substitution to set the systemd unit instance to the hard disk device path. The systemd unit passes on the device path to actual sync_all_images_to_disk script via the %i identifier:

ExecStart=/usr/local/bin/sync_all_images_to_disk /dev/%i

This means that the burning process is completely decoupled from the building process. It starts when a drive is plugged in. We created an elegant way for operators to know when they’re ready, which is outlined in the Hardware section below.

One day as we were developing the system, we ended up wiping the O/S drive of the Flexo system itself. We decided to mark the hard drives that should be used by Flexo. Each Flexo drive has a disk GUID starting with 00000000-. The udev rule uses an additional filter based on the ID_PART_TABLE_UUID environment variable to start a sync_all_images_to_disk@ job only for disks that are marked as Flexo drives:

ENV{ID_PART_TABLE_UUID}==”00000000-*”

Marking drives also helps with operator mistakes. If a non-Flexo drive is inserted by error it will not be overwritten.

Image selector and anatomy of a boot disk

Each Flexo hard drive supports multiple versions of our full software stack. We use GRUB as the primary UI for operators to select an image to boot at the beginning of their mission:

A master GRUB configuration maintains the list of images available in each partition of the hard drive. Each image provides a secondary GRUB bootloader with the kernel and initial ramdisk configuration. Images are chainloaded from the master bootloader to decouple each image as much as possible. A faulty configuration in one image won’t impact the other images of the hard drive.

As can be seen in the diagram above, each partition type uses UUID prefixes to indicate what type of partition it is. We also UUID prefix the file system.

It should be noted that getting this part of Flexo stable was incredibly time-consuming. The image burner process today is less than 1000 lines of BASH, but each line took a lot of work to get right as these low-level tools are somewhat poorly documented and there are not many posts in forums and blogs about it.

For example, while Linux does support dynamic mounting of file systems, keeping the kernel up-to-date as we create up to a dozen partitions per drive on up to 24 drives in the system causes a lot of contention at the kernel and system level. We had to sprinkle a heavy dose of read/write locks using flock, and do explicit calls to partprobe to update the kernel’s view of new partitions, before this finally became stable.

Preventing drift: overlayroot

One pain point we experienced before the design of Flexo was that the state of each boot disk would drift over time given that boot disks would stay in the car and reused by successive missions. For our missions, it is crucial that we start with a clean slate every time, so that a previous mission doesn’t affect the next.

To do this we use the overlayroot package to provide a writable layer on top of the existing image. The overlay partition on the hard drive is used as the temporary place to store changes to the image while it runs. We’re using the crypt backend with a random password to ensure that any changes made during the live system are erased upon reboot. The overlayroot setting is turned on in the GRUB configuration for all operations targeted images:

overlayroot=”crypt:dev=/dev/disk/by-partuuid/55555555-<DISK_ID>-555555555555,mkfs=1,fstype=ext4,recurse=0"

This feature, combined with instructions to the operations team to always restart the computer between missions, has greatly dropped the amount of time engineers spend supporting the operations team.

Testing images in kvm

As we were developing the Flexo system, we quickly realized that physically moving a hard drive from one system to another to test it leads to long iteration cycles. We started to leverage kvm and OVMF to speed up our development. OVMF provides a UEFI bios similar to the computers in the cars. This virtualized test environment also includes the local usb drive used for car identity.

Starting a test environment for a given hard drive is done via a kvm command similar to this:

kvm -m 4096 -bios OVMF.fd -drive format=raw,file=/dev/sdk -drive format=raw,file=”car_data.img” — vnc :59

We can test the full boot sequence from the BIOS to the actual start of the AV software stack (including the graphical part) on the Flexo system itself. We can even include a virtual version of the vehicle specific USB stick (the car_data.img above). Not everything boots cleanly in a VM, but enough to verify most system settings.

Hardware

Since we need at least one boot disk per car (and we generally keep multiple boot disks around so that we don’t have to wait for new ones to get built while they are in use), each Flexo system is a standard rack mounted server with 24 drive bays that allows us to swap drives in bulk. We keep several systems per site where we store cars.

I/O throughput is the most important metric. Keeping the entire working set in RAM is crucial for high speed burning of disks, so we stock our Flexo machines with as much RAM as possible.

As the disk burning part of the Flexo systems is self-running, we use ledmon to control the enclosure LED to indicate disk status. When a disk is inserted, the LED goes dark, and then starts blinking rapidly as the burning process is underway. Finally, a solid red light means the drive is ready to be taken out and moved to a vehicle.

Streamlined workflow to shorten feedback loops

All of our cars and AV operators are now using the Flexo deployment platform. We’ve seen a substantial decrease in pre-mission startup times thanks to fully working bootable images.

Developers also stopped wondering about the state of the O/S. Stateless environments remove a variable in their troubleshooting process.

Next steps

As we scale up our fleet, the Flexo deployment platform will be deployed to multiple systems. We’re considering moving the image builder component to the cloud to ensure the images are the same across all Flexo systems. We’re also looking at extending Flexo with the capability to build an image for a specific developer branch as well as continuously testing images in the cloud. The Flexo deployment pipeline will keep playing a key role in tightening the development loop and giving feedback to the developer as quickly as possible.

If this post piqued your interest, note that we’re hiring! Check out our open roles here and follow our page for more technical content.