Create a bootable Linux installer with customizations on a USB Flash Drive with Fully Automatic Installer (FAI)
Let’s start with the problem I had in hand a while back. It was a rather simple one, install Debian OS and some extra packages on physical machines, with just a USB stick and next to no supervision.
This problem should be fairly simple to solve. It’s nothing new or groundbreaking; the need to duplicate an OS setup onto many devices has been around for decades.
However, the resources to achieve as such seem few and far between. So let’s see if we can simplify this whole thing.
To create our own one-button installer to install a Linux based OS into any computer, something like this: https://fai-project.org/slideshow/page101.html
Why would you need this?
There are times where we repeat the same OS setup on many machines, like:
- Setting up a fleet of physical servers with the same OS and packages
- Manufacturing products which use a Linux based OS with modifications
- Maintain a standard company computer setup or development environment
What is Fully Automatic Installer (FAI)
FAI - Homepage
FAI is a tool for unattended mass deployment of Linux. It's a system to install and configure Linux systems and…
FAI is a tool for unattended mass deployment of Linux. It’s a system to install and configure Linux systems and software packages on computers as well as virtual machines, from small labs to large-scale infrastructures like clusters and virtual environments. You can take one or more virgin PC’s, turn on the power, and after a few minutes, the systems are installed, and completely configured to your exact needs, without any interaction necessary.
FAI was released in December 2009, which at the time of writing, is more than a decade old. Yet, it is still one of the most elegant solutions to date because,
- It supports almost any Linux based distribution: Debian, Ubuntu, CentOS, SuSE, etc. In this article, we will set up a Debian installer.
- Additional packages can be added to the installer itself to be installed during the setup process.
- Quite literally one-button installation, or maybe two depending on your configuration.
- The creation of the installation image can be automated via a Continuous Integration (CI) pipeline, which ironically only became popular quite a while later.
- It supports multiple boot methods. Primarily it is designed for booting a new machine via the network card (ideally with PXE support). However, we are simple folks, only interested in creating an image to be put onto a USB Flash Drive 😂.
- Still actively maintained at https://github.com/faiproject/fai
To very briefly summarize how FAI works, it runs an automated installation process on a new machine by mounting a root file system (NFSRoot) and installing packages and scripts that are either hosted on a remote machine or removable media.
However, the documentation is mostly focused on setting up a FAI server for network installation. On the other hand, creating installation via removable media is scarcely documented, there is no step-by-step guide, just a whole bunch of
fai- commands and lots of
man pages. Unless that’s your kind of thing, no offense to man pages zealots out there, but that’s just not mine.
What do you need?
If you do not need extensive customizations to your installation, do check out https://fai-project.org/FAIme/ if it fits your use case.
Otherwise, we will be needing:
- A fresh copy of a Debian Based OS installation with
- A Continuous Integration (CI) machine with a Debian Based OS installed
You can use local VMs, AWS EC2 instances, GCP Compute Instances, build boxes from common CI providers, it doesn’t really matter. It is very much recommended to avoid using your personal computer’s Linux installation, as cleaning it up afterward might be a bit of a chore.
In this example, we will use a machine with Ubuntu 20.04 installed. We will create a simple one-button installer for Debian 10 with our own post-installation scripts.
Note: It is likely possible to run this guide without
sudo access, with chroot, fakeroot, or similar. However, that will not be covered.
Step 1: Creating the configuration directory
Clone (or fork) this project:
You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or…
$ git clone firstname.lastname@example.org:faiproject/fai-config.git
Next up, visit
Over here, pick a distribution that you are basing off. We will download
BUSTER64.tar.xz for this example.
# From the root of the cloned repository
$ cd basefiles
$ wget https://fai-project.org/download/basefiles/BUSTER64.tar.xz
Step 2: Customizing the Installation
Before we start customizing, it might be worth familiarizing with the
class concept in FAI
For our purpose, we will be modifying the file at
For simplicity sake, we change the contents to the following:
#! /bin/bashecho DEBIAN BUSTER64 DHCPC DEMO FAIBASE BUSTER XORG GNOME STANDARD NONFREE FAIME
NONFREEare optional, depending on your needs. Refer to the corresponding files
/package_config to view the list of additional packages each of them will install.
We also introduced our own class,
FAIME , which we use to specify our own customizations in the coming sections.
If you choose to create your own file instead of overwriting
50-host-classes, remember to grant executable permissions to the file
2.1 Modifying the default settings
classes directory, there are a bunch of variable definition files with the extension
.var. Some of these variables can be overridden. Check out
FAIBASE.var, as we will be overriding some of these variables.
We will create a new file in the
FAIME.var , with the following contents
# Override the Timezone to something of our choice
HOSTNAME=myhostname# Set the password to "password123", you can generate a password with the command echo "password123" | mkpasswd -m md5 -s
The sequence in which classes are defined in the
50-host-classes file above will determine which variables take precedence. Because
FAIME is defined right at the end, it will be executed last and thus overwrite the variables defined in
2.2 Modifying the installed packages
Packages will be installed in the sequence their corresponding classes are defined in
To install additional packages, we create a new file
/package_config with the following contents:
# For example, add curl, vim and nano packages to installation
curl vim nano
The installer will be created with all the packages defined in
/package_config, not just the classes we want to run. Therefore, we may want to do some cleaning up of unused packages for our installation.
Based on our example, we will delete these unused files (YMMV):
$ rm package_config/CENTOS
$ rm package_config/GERMAN
$ rm package_config/XFCE
$ rm package_config/UBUNTU # We don't need Ubuntu as we are installing Debian
Next, take a look at
package_config/DEBIAN. We will mark the candidates for deletion:
apt-transport-https # is only needed for stretch
unattended-upgradesPACKAGES install NONFREE
# you may want these non-free kernel drivers
firmware-bnx2 firmware-bnx2x firmware-realtek
firmware-linux-nonfree# Remove this unless we need I386 architecture
# PACKAGES install I386
# memtest86+# Remove these
# PACKAGES install CHROOT
# linux-image-amd64-PACKAGES install AMD64
memtest86+# In this example we support AMD64 architecture, not both. YMMV.
# PACKAGES install ARM64
# linux-image-arm64PACKAGES install GRUB_PC
PACKAGES install GRUB_EFI
grub-efi# Remove these
# PACKAGES install LVM
# PACKAGES install CLOUD
Add post-install scripts
Should we wish to perform additional configurations post-installation, we can do so under the
/scripts directory. Create a new folder
FAIME . Within the folder, create a file
01-custom-scripts . Be sure to add executable permissions to this script
chmod +x ./scripts/FAIME/01-custom-scripts
Let’s assume we want to hide the grub bootloader using a post install script, the contents of
01-custom-scripts will be something like:
#!/bin/bashecho "Running My Custom Installation Scripts"# Prevent grub bootloader from showing on boot
sed -i '/GRUB_TIMEOUT/c\GRUB_TIMEOUT=0' $target/etc/default/grub
$ROOTCMD update-grub2chroot $target /bin/bash << "EOT"
# An alternative to prepending $ROOTCMD for every command you need to run on the newly installed OS
A few useful variables which are automatically set:
$target — The full path to the root of the newly installed OS. Note it is not mounted as '/' at this point$ROOTCMD - Run a command using chroot on the newly installed OS$FAI - The full path to the fai-config directory. Your entire fai-config working directory in your repo will be copied in the final image and thus available in these scripts, which means you can include custom assets on top of scripts, e.g copying new wallpapers as part of your post installation.
Similar to package installations, scripts will be executed in the sequence their corresponding classes are defined in
/class/50-host-classes, but only after all package installation finishes. Furthermore, we number our scripts
02 etc, so they are executed in sequence.
Modify partitioning format
If we look at the
/disk_config folder, there are a bunch of files that specify how partitioning should be set up. At this point, the
FAIBASE_EFI ) are chosen based on our classes. Should you see a need to modify, you can either modify the file directly or create your own class for partitioning.
Step 3: Set up NFSRoot
Next up, we have to install
fai-server on our VM.
$ wget http://deb.debian.org/debian/pool/main/f/fai/fai-server_5.9.4_all.deb
$ sudo apt install ./fai-server_5.9.4_all.deb
Also, install some additional packages which fai depends on to create ISO files.
$ sudo apt install -y reprepro xorriso
Now it’s time to create NFSRoot. It might take a while.
$ sudo fai-make-nfsroot -v -f
Once done, we would have our NFSRoot set up in
/srv/fai, along with some configuration files in
We will need to modify the variable
/etc/fai/nfsroot.conf . In the root of your FAI project
$ sudo sed -i '/FAI_CONFIGDIR=/c\FAI_CONFIGDIR='"$FAI_CONFIGDIR" /etc/fai/nfsroot.conf
Step 4: Creating the Installer ISO
Before we can create the ISO image, we have to set up a local mirror for all the packages we have included in our
package_config . The command
fai-mirror does exactly that.
$ mkdir /tmp/fai-mirror
$ fai-mirror -b -v /tmp/fai-mirror
We are using
/tmp directory for the mirror because the packages can use up significant disk space. The intention is to have them cleared soon enough automatically. However, if you’d like to permanently store these packages to speed up subsequent mirror creation runs, feel free to use some other directory.
When you’re done, you should see something like this printed on your screen
Mirror size and location: 601M /tmp/fai-mirror
Otherwise, check if any package failed to download.
Finally, we create our ISO image
$ sudo fai-cd -m /tmp/fai-mirror -eJ my-os-installer.iso
fai-cd will copy your NFSRoot together with the files from the mirror into a bootable ISO image.
-e specifies to ignore the
/tmp folder in the NFSRoot.
-J favors xz compression instead of gz.
Hooray, we can finally burn the ISO into a removable storage media. Do take note of important information in the next section.
Step 5: Burning the ISO to a USB Flash Drive (or whatever storage media)
First things first, the bootable ISO Image uses ext4 file system. This is important because the kernel included in our bootable NFSRoot does not include modules to load vfat.
Many image-to-disk programs assume it is safe to format as FAT16 or FAT32 before copying the contents in the ISO. Using them to put the ISO into a USB Flash Drive or similar will only result in boot errors.
There could be smarter ways to solve this, but for this example we use
dd to put the image into a USB Flash Drive, preserving the file system of the ISO.
# Assuming the USB drive is /dev/sdb
# Make sure it is not mounted anywhere
$ wipefs -a /dev/sdb
$ sudo dd if=my-os-installer.iso of=/dev/sdb bs=1M
Conclusion and additional notes
Now, we can bring the removal media to a brand new machine and run the installer. It should support both MBR and EFI boot, although I have not tested the MBR myself.
On the grub bootloader screen, select the first option and enjoy the fruits of your labor!
NOTE: The grub bootloader can be customized by editing
/etc/fai/grub.cfg as it will be copied into the ISO.
As we can see FAI is a powerful tool that can automate the entire installer image creation process. Personally, I run the entire workflow on Github Actions and upload the images somewhere on build completion.
Afterthought: What about disk cloning?
Yes, disk cloning is great for the manufacturing process. It is (probably) the most commonly used method to copy a particular setup to a fleet of similarly configured machines.
- Disk cloning assumes we somehow find ourselves the master copy with everything we need properly set up (drivers, packages, etc). It does not govern the process to create the master copy itself.
- Depending on the drivers available in the master copy, it can be very sensitive to hardware variations, more often than not, you will have to create a new master copy. It is often not in our best interests to be overzealous with drivers either.
- Automatically building clonable base images via a CI pipeline or similar is an art by itself.