Create a bootable Linux installer with customizations on a USB Flash Drive with Fully Automatic Installer (FAI)

A step-by-step guide

Introduction

Let’s start with the problem I had in hand a while back. It was a rather simple one, install Debian OS and some extra packages on physical machines, with just a USB stick and next to no supervision.

This problem should be fairly simple to solve. It’s nothing new or groundbreaking; the need to duplicate an OS setup onto many devices has been around for decades.

However, the resources to achieve as such seem few and far between. So let’s see if we can simplify this whole thing.

Objective

To create our own one-button installer to install a Linux based OS into any computer, something like this: https://fai-project.org/slideshow/page101.html

Why would you need this?

There are times where we repeat the same OS setup on many machines, like:

• Setting up a fleet of physical servers with the same OS and packages
• Manufacturing products which use a Linux based OS with modifications
• Maintain a standard company computer setup or development environment

What is Fully Automatic Installer (FAI)

FAI - Homepage

FAI is a tool for unattended mass deployment of Linux. It’s a system to install and configure Linux systems and software packages on computers as well as virtual machines, from small labs to large-scale infrastructures like clusters and virtual environments. You can take one or more virgin PC’s, turn on the power, and after a few minutes, the systems are installed, and completely configured to your exact needs, without any interaction necessary.

FAI was released in December 2009, which at the time of writing, is more than a decade old. Yet, it is still one of the most elegant solutions to date because,

• It supports almost any Linux based distribution: Debian, Ubuntu, CentOS, SuSE, etc. In this article, we will set up a Debian installer.
• Additional packages can be added to the installer itself to be installed during the setup process.
• Quite literally one-button installation, or maybe two depending on your configuration.
• The creation of the installation image can be automated via a Continuous Integration (CI) pipeline, which ironically only became popular quite a while later.
• It supports multiple boot methods. Primarily it is designed for booting a new machine via the network card (ideally with PXE support). However, we are simple folks, only interested in creating an image to be put onto a USB Flash Drive 😂.
• Still actively maintained at https://github.com/faiproject/fai

To very briefly summarize how FAI works, it runs an automated installation process on a new machine by mounting a root file system (NFSRoot) and installing packages and scripts that are either hosted on a remote machine or removable media.

However, the documentation is mostly focused on setting up a FAI server for network installation. On the other hand, creating installation via removable media is scarcely documented, there is no step-by-step guide, just a whole bunch of fai- commands and lots of man pages. Unless that’s your kind of thing, no offense to man pages zealots out there, but that’s just not mine.

What do you need?

If you do not need extensive customizations to your installation, do check out https://fai-project.org/FAIme/ if it fits your use case.

Otherwise, we will be needing:

1. A fresh copy of a Debian Based OS installation with sudo access; or
2. A Continuous Integration (CI) machine with a Debian Based OS installed

You can use local VMs, AWS EC2 instances, GCP Compute Instances, build boxes from common CI providers, it doesn’t really matter. It is very much recommended to avoid using your personal computer’s Linux installation, as cleaning it up afterward might be a bit of a chore.

In this example, we will use a machine with Ubuntu 20.04 installed. We will create a simple one-button installer for Debian 10 with our own post-installation scripts.

Note: It is likely possible to run this guide without sudo access, with chroot, fakeroot, or similar. However, that will not be covered.

Step 1: Creating the configuration directory

Clone (or fork) this project:

faiproject/fai-config

$ git clone git@github.com:faiproject/fai-config.git

Next up, visit

Index of /download/basefiles

Over here, pick a distribution that you are basing off. We will download BUSTER64.tar.xz for this example.

# From the root of the cloned repository
$ cd basefiles
$ wget https://fai-project.org/download/basefiles/BUSTER64.tar.xz

Step 2: Customizing the Installation

Before we start customizing, it might be worth familiarizing with the class concept in FAI

https://fai-project.org/fai-guide/#_a_id_classc_a_the_class_concept

For our purpose, we will be modifying the file at /class/50-host-classes
For simplicity sake, we change the contents to the following:

#! /bin/bash
echo DEBIAN BUSTER64 DHCPC DEMO FAIBASE BUSTER XORG GNOME STANDARD NONFREE FAIME

The classes XORG, GNOMEand NONFREEare optional, depending on your needs. Refer to the corresponding files /package_config to view the list of additional packages each of them will install.

We also introduced our own class, FAIME , which we use to specify our own customizations in the coming sections.

If you choose to create your own file instead of overwriting 50-host-classes, remember to grant executable permissions to the file

2.1 Modifying the default settings

In the classes directory, there are a bunch of variable definition files with the extension .var. Some of these variables can be overridden. Check out FAIBASE.var, as we will be overriding some of these variables.

We will create a new file in the classes directory, FAIME.var , with the following contents

# Override the Timezone to something of our choice
TIMEZONE=Asia/Singapore
HOSTNAME=myhostname
# Set the password to "password123", you can generate a password with the command echo "password123" | mkpasswd -m md5 -s
ROOTPW=$1$/x8wix9y$qdLUD.FfTE2P9vEmtOr9Q.
USERPW=$1$/x8wix9y$qdLUD.FfTE2P9vEmtOr9Q.

The sequence in which classes are defined in the 50-host-classes file above will determine which variables take precedence. Because FAIME is defined right at the end, it will be executed last and thus overwrite the variables defined in FAIBASE.

2.2 Modifying the installed packages

Packages will be installed in the sequence their corresponding classes are defined in /class/50-host-classes.

To install additional packages, we create a new file FAIME in /package_config with the following contents:

# For example, add curl, vim and nano packages to installation
PACKAGES install
curl vim nano

The installer will be created with all the packages defined in /package_config, not just the classes we want to run. Therefore, we may want to do some cleaning up of unused packages for our installation.

Based on our example, we will delete these unused files (YMMV):

$ rm package_config/CENTOS
$ rm package_config/GERMAN
$ rm package_config/XFCE
$ rm package_config/UBUNTU # We don't need Ubuntu as we are installing Debian

Next, take a look at package_config/DEBIAN. We will mark the candidates for deletion:

PACKAGES install-norec
apt-transport-https # is only needed for stretch
debconf-utils
file
less
linuxlogo
rsync
openssh-client openssh-server
time
procinfo
nullmailer
eject
locales
console-setup kbd
pciutils usbutils
unattended-upgrades
PACKAGES install NONFREE
# you may want these non-free kernel drivers
firmware-bnx2 firmware-bnx2x firmware-realtek
firmware-linux-nonfree
# Remove this unless we need I386 architecture
# PACKAGES install I386
# linux-image-686-pae
# memtest86+
# Remove these
# PACKAGES install CHROOT
# linux-image-686-pae-
# linux-image-amd64-
PACKAGES install AMD64
linux-image-amd64
memtest86+
# In this example we support AMD64 architecture, not both. YMMV.
# PACKAGES install ARM64
# grub-efi-arm64
# linux-image-arm64
PACKAGES install GRUB_PC
grub-pc
PACKAGES install GRUB_EFI
grub-efi
# Remove these
# PACKAGES install LVM
# lvm2
# PACKAGES install CLOUD
# unattended-upgrades

Add post-install scripts

Should we wish to perform additional configurations post-installation, we can do so under the /scripts directory. Create a new folder FAIME . Within the folder, create a file 01-custom-scripts . Be sure to add executable permissions to this script

chmod +x ./scripts/FAIME/01-custom-scripts

Let’s assume we want to hide the grub bootloader using a post install script, the contents of 01-custom-scripts will be something like:

#!/bin/bash
echo "Running My Custom Installation Scripts"
# Prevent grub bootloader from showing on boot
sed -i '/GRUB_TIMEOUT/c\GRUB_TIMEOUT=0' $target/etc/default/grub
$ROOTCMD update-grub2
chroot $target /bin/bash << "EOT"
# An alternative to prepending $ROOTCMD for every command you need to run on the newly installed OS
EOT

A few useful variables which are automatically set:

$target — The full path to the root of the newly installed OS. Note it is not mounted as '/' at this point
$ROOTCMD - Run a command using chroot on the newly installed OS
$FAI - The full path to the fai-config directory. Your entire fai-config working directory in your repo will be copied in the final image and thus available in these scripts, which means you can include custom assets on top of scripts, e.g copying new wallpapers as part of your post installation.

Similar to package installations, scripts will be executed in the sequence their corresponding classes are defined in /class/50-host-classes, but only after all package installation finishes. Furthermore, we number our scripts 01 , 02 etc, so they are executed in sequence.

Modify partitioning format

If we look at the /disk_config folder, there are a bunch of files that specify how partitioning should be set up. At this point, the FAIBASE (and FAIBASE_EFI ) are chosen based on our classes. Should you see a need to modify, you can either modify the file directly or create your own class for partitioning.

Step 3: Set up NFSRoot

Next up, we have to install fai-server on our VM.

$ wget http://deb.debian.org/debian/pool/main/f/fai/fai-server_5.9.4_all.deb
$ sudo apt install ./fai-server_5.9.4_all.deb

Also, install some additional packages which fai depends on to create ISO files.

$ sudo apt install -y reprepro xorriso

Now it’s time to create NFSRoot. It might take a while.

$ sudo fai-make-nfsroot -v -f

Once done, we would have our NFSRoot set up in /srv/fai, along with some configuration files in /etc/fai

We will need to modify the variable FAI_CONFIGDIR in /etc/fai/nfsroot.conf . In the root of your FAI project

$ FAI_CONFIGDIR="$(pwd)"
$ sudo sed -i '/FAI_CONFIGDIR=/c\FAI_CONFIGDIR='"$FAI_CONFIGDIR" /etc/fai/nfsroot.conf

Step 4: Creating the Installer ISO

Before we can create the ISO image, we have to set up a local mirror for all the packages we have included in our package_config . The command fai-mirror does exactly that.

$ mkdir /tmp/fai-mirror
$ fai-mirror -b -v /tmp/fai-mirror

We are using /tmp directory for the mirror because the packages can use up significant disk space. The intention is to have them cleared soon enough automatically. However, if you’d like to permanently store these packages to speed up subsequent mirror creation runs, feel free to use some other directory.

When you’re done, you should see something like this printed on your screen

Calling reprepro
Exporting indices...
/usr/bin/fai-mirror finished.
Mirror size and location: 601M  /tmp/fai-mirror

Otherwise, check if any package failed to download.

Finally, we create our ISO image

$ sudo fai-cd -m /tmp/fai-mirror -eJ my-os-installer.iso

fai-cd will copy your NFSRoot together with the files from the mirror into a bootable ISO image.
The option -e specifies to ignore the /tmp folder in the NFSRoot.
The option -J favors xz compression instead of gz.

Hooray, we can finally burn the ISO into a removable storage media. Do take note of important information in the next section.

Step 5: Burning the ISO to a USB Flash Drive (or whatever storage media)

First things first, the bootable ISO Image uses ext4 file system. This is important because the kernel included in our bootable NFSRoot does not include modules to load vfat.

Many image-to-disk programs assume it is safe to format as FAT16 or FAT32 before copying the contents in the ISO. Using them to put the ISO into a USB Flash Drive or similar will only result in boot errors.

There could be smarter ways to solve this, but for this example we use dd to put the image into a USB Flash Drive, preserving the file system of the ISO.

# Assuming the USB drive is /dev/sdb
# Make sure it is not mounted anywhere
$ wipefs -a /dev/sdb
$ sudo dd if=my-os-installer.iso of=/dev/sdb bs=1M

Conclusion and additional notes

Now, we can bring the removal media to a brand new machine and run the installer. It should support both MBR and EFI boot, although I have not tested the MBR myself.

On the grub bootloader screen, select the first option and enjoy the fruits of your labor!

NOTE: The grub bootloader can be customized by editing /etc/fai/grub.cfg as it will be copied into the ISO.

As we can see FAI is a powerful tool that can automate the entire installer image creation process. Personally, I run the entire workflow on Github Actions and upload the images somewhere on build completion.

Afterthought: What about disk cloning?

Yes, disk cloning is great for the manufacturing process. It is (probably) the most commonly used method to copy a particular setup to a fleet of similarly configured machines.

However,

• Disk cloning assumes we somehow find ourselves the master copy with everything we need properly set up (drivers, packages, etc). It does not govern the process to create the master copy itself.
• Depending on the drivers available in the master copy, it can be very sensitive to hardware variations, more often than not, you will have to create a new master copy. It is often not in our best interests to be overzealous with drivers either.
• Automatically building clonable base images via a CI pipeline or similar is an art by itself.

References & Credits

Fai Homepage
FaiME
https://github.com/faiproject/fai-config