Automating Red Hat Enterprise Linux installation

12 min readJun 2, 2023

Foreword

Automation is not about replacing humans; it’s about amplifying human potential.
— Unknown

The ability to automatically install Linux OS offers several benefits. Firstly, it saves significant time and effort by eliminating the need for manual installation on multiple systems, making it ideal for large-scale deployments. Secondly, it ensures consistency and accuracy in the installation process, reducing the chances of human errors. It also allows for easy replication of installations, enabling you to quickly set up new machines with predefined configurations, software packages, and security settings.

Kickstart

Kickstart is a tool used to automate the installation of an operating system. It involves creating a configuration also referred to as an answer file that contains all the necessary settings and options for the installation, such as disk partitioning, package selection, and user accounts. The Kickstart configuration file is then used by the Anaconda installer to perform a fully automated installation without any user intervention. This saves time and reduces the potential for errors during the installation process. Kickstart is commonly used in enterprise environments where large numbers of systems need to be installed and configured quickly and efficiently.

To generate a kickstart file, we can use the Kickstart Generator or we can write your own as we will do in this article.

📝Note: If you performed a manual installation described in Installing Red Hat Enterprise Linux, then a kickstart file was automatically generated by Anaconda and stored in root’s home directory as anaconda-ks.cfg.

Kickstart Customization

Objectives

A coworker from the database team is frustrated by manual installation for Red Hat Enterprise Linux, which he performs every week. The current process is time-consuming and often produces inconsistent results. He asked for your wisdom in setting up a more advanced installation process that he could use for virtual and bare metal machines.

He is looking for a reusable configuration template, which he could leverage for automated installation for all future builds.

He shared that the provisioning network where these machines are connected offers dynamic network configuration through DHCP, but each new machine needs to have a unique hostname. He would like to have the ability to supply this value at the beginning of the installation.

For local accounts setup, besides the root account, he asked for one additional privileged user to create called ansible with a password of ansible. After the installation, his existing post-deployment script will update these local users’ credentials once the machine is reachable via SSH. For software selection, he is happy with the minimal install installation as he will customize the software stack based on machine usage. Finally, he would like to use your private repository for the time being.

Prerequisites

The following prerequisites are required before you can continue with the solution:

- An existing Red Hat Enterprise Linux system setup for hosting private BaseOS and AppSteam repositories

📝Note: Not sure if you meet these prerequisites? Consult the Managing Software in Red Hat Enterprise Linux guide.

Solution

We start by gaining root privileges on the existing rhel01 machine. We will use this virtual machine to generate a custom kickstart file for generic installations.

Before that, let us explore the existing kickstart file that was generated from the answers we provided in the manual installation process described in Installing Red Hat Enterprise Linux guide.

Nothing is more painful than spending hours perfecting the custom kickstart file to find out that contains errors that prevent the installation process to continue. Therefore it is vital to perform validation during the development phase using ksvalidator which is part of pykickstart package.

# Intalls tools
dnf install -y pykickstart

Now, we are ready to validate and inspect the reference file located in /root/anaconda-ks.cfg. If ksvalidator returns no output that means no errors were found.

# Validate default ks file.
ksvalidator /root/anaconda-ks.cfg

# Inspect the reference file
cat /root/anaconda-ks.cfg | head
# Generated by Anaconda 34.25.2.10
# Generated by pykickstart v3.32
#version=RHEL9
# Use graphical install
graphical
repo --name="AppStream" --baseurl=file:///run/install/sources/mount-0000-cdrom/AppStream

%addon com_redhat_kdump --enable --reserve-mb='auto'

%end

💡Tip: If you are refactoring existing kickstart files used for older OS releases, then the ksverdiff might be handy. For example, to show the differences from RHEL8 to RHEL9 use the -f and -t arguments. For example ksverdiff -f RHEL8 -t RHEL9.

The reference kickstart file anaconda-ks.cfg contains roughly 45 lines. We are only going to focus on a few that concern your objectives. For a full list consult the pykickstart’s official documentation.

For convenience, we start from the end. We download the final kickstart file and make it available under ks folder located in the web server root directory.

mkdir -pv /var/www/html/ks
curl -o /var/www/html/ks/generic-ks.cfg \
        https://raw.githubusercontent.com/maroskukan/rhcsa/main/support/ks/generic-ks.cfg

⚠️ Warning: Depending on the origin of this file, you might need to verify permissions and the SELinux label to be served by the web server.

Next, let us inspect this kickstart file. Concerning our objectives, there was a need to be able to take user input for the hostname and use it to adjust the configuration. We address this requirement in the Network information section displayed below.

grep "Network information" -A15 generic-ks.cfg
# Network information
%pre
#!/bin/bash

# Default hostname
echo "network --device eth0 --bootproto dhcp --hostname localhost.localdomain" > /tmp/network-include
# Search for hostname parameter, if found update the network ks
for args in `cat /proc/cmdline`; do
        case $args in hostname*)
            eval $args
            sed -i "s/localhost.localdomain/$hostname/g" /tmp/network-include
            ;;
        esac;
    done
%end
%include /tmp/network-include

We first use the pre-installation script ( %pre). where we write a default network configuration into a temporary file /tmp/network-include. Next, we iterate over a list of kernel arguments that are space-separated and stored in /proc/cmdline. When we find a match for the hostname* string in the list we source this key-value pair in the environment, making it available through the $hostname variable. When this happens, we use sed to update the default value for the hostname. Finally, we include this custom configuration file.

⚠️ Warning: Be aware of the End of Line Sequence. If you edit these files in Windows using the CRLF line endings may contain special characters invisible to naked eye. This causes issues when you redirect output to a file e.g. > /tmp/network-include. Therefore make sure you use LF for line endings.

The local user account credentials requirement is implemented in the following section.

grep Root -A4 generic-ks.cfg
# Root password - "redhat"
rootpw --iscrypted $6$R8s8JcJVnjPVpjrR$b3iEiZdRzC50M/5qUT9Pf1O07QfXVlKneXWyVmkFV/g2TzBdTC05ffz7dzHMiJ1CxmueCTYoE5mNpuBTk2sJ0.
# User password - "ansible"
user --groups=wheel --name=ansible --password=$6$BErsrjivm5Y.UExP$glz8qL9vfGScQT1CAr3T1WuW7pK1RBtNgQyzvomFGrRO19M/b8PXEIWAcbmjm97//KXwWx1Z.SecOrgGKAqtR0 --iscrypted --gecos="ansible"

We used the mkpasswd --method=SHA-512 --stdin to generate password hashes for both root and ansible users. We also ensure that the ansible user has administrative privileges by defining membership in the wheel supplementary group.

The package selection requirement is implemented by the following section.

grep %packages -A1 generic-ks.cfg
%packages
@^minimal-environment

We defined only installing the minimal-environment package group.

The repository definition part is implemented in the following sections.

grep https -B1 generic-ks.cfg
# Repositories
repo --name="AppStream" --noverifyssl --baseurl=https://rhel01.mshome.net/rhel9.2/x86_64/dvd/AppStream
--
# Use network installation
url --noverifyssl --url="https://rhel01.mshome.net/rhel9.2/x86_64/dvd/BaseOS/"

We defined custom URLs pointing to the private repositories hosted on rhel01. We need to disable SSL verification since we are using self-signed certificates.

That is all when it comes to the template itself. Now let us verify that it works by creating a new virtual machine.

From Hyper-V host provision a new virtual machine rhel03.

# From Hyper-V Host, download the vm creation script
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/maroskukan/hypervisor-cookbook/main/hyperv/scripts/vm_create.ps1" `
                  -OutFile "vm_create.ps1"

# From Hyper-V Host, execute the script, update isoPath according to your environment
.\vm_create.ps1 -Name rhel03 `
                -Size S `
                -IsoPath D:\iso\rhel-9.2-x86_64-boot.iso

Once the GRUB2 menu loads, use arrows to navigate to the first menu entry labeled Install Red Hat Enterprise Linux 9.2 and hit e to edit commands before booting.

Using arrows navigate the line starting with linuxefi and append the following three additional boot options.

inst.ks=https://rhel01.mshome.net/ks/generic-ks.cfg inst.noverifyssl hostname=rhel03

💡Tip: Looking for more information on boot options? The following documentation is great resource.

Afterwards, hit CTRL + xor F10 to boot.

💡Tip: An alternative approach is to invoke the GRUB2 command line interface using CTRL + c and specifying the initial RAM disk (initrd) and kernel with custom parameters and then type boot to continue.

linuxefi /images/pxeboot/vmlinuz inst.stage2=hd:LABEL=RHEL-9-2-0-BaseOS-x86_64 quite inst.ks=https://rhel01.mshome.net/ks/generic-ks.cfg inst.noverifyssl hostname=rhel03
initrdefi /images/pxeboot/initrd.img
boot

When successful, you should see that the kickstart file was downloaded and validated. The installation will then begin.

💡Tip: Having trouble with Anaconda installation? Switch to shell and check logs at /tmp/anaconda.log By the way the the ks file is downloaded to /run/install/ks.cfg.

Once the installation completes and the virtual machine reboots. Verify from the Hyper-V host that the SSH service is running.

# From Hyper-V Host, verify that the VM listens on TCP/22 (SSH)
Test-NetConnection -ComputerName "rhel03.mshome.net" `
                   -Port 22 `
                   | Select-Object -Property `
                   @{Name="ComputerName"; Expression={$_.ComputerName}}, `
                   @{Name="RemoteAddress"; Expression={$_.RemoteAddress.IPAddressToString}}, `
                   @{Name="TcpTestSucceeded"; Expression={$_.TcpTestSucceeded}} `
                   | ConvertTo-Json

An example of happy output would look like the following.

{
  "ComputerName": "rhel03.mshome.net",
  "RemoteAddress": "172.19.102.38",
  "TcpTestSucceeded": true
}

In summary, we addressed all requirements received from our colleague. When done exploring the environment, we can clean up by removing the rhel03 virtual machine.

# From Hyper-V Host, download the vm creation script
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/maroskukan/hypervisor-cookbook/main/hyperv/scripts/vm_delete.ps1" `
                  -OutFile "vm_delete.ps1"

# From Hyper-V Host, delete the virtual machine
.\vm_delete.ps1 -name rhel03

Anaconda

Anaconda is the default installer for Red Hat Enterprise Linux (RHEL) and simplifies the installation process by providing a user-friendly interface. It allows users to customize their RHEL installation, including partitioning, software packages, networking, and security settings. With automatic hardware detection and support for various storage configurations, Anaconda ensures a smooth and reliable deployment of RHEL.

Anaconda Customization

Objectives

A colleague from the platform team is currently overloaded by manually deploying hundreds of new bare metal hypervisor hosts that will be used for Red Hat Virtualization. He heard about your recent contribution to the database team and how you used Kickstart to automate part of the process. He would like to go further and eliminate the need to define hostname as kernel argument manual during installation.

As he describes further, when new servers are racked, an entry in CMDB is created with their serial number and hostname. He would like to leverage information in this database to create one kickstart file that would be automatically fetched through the network by the installer during deployment. After this groundwork is completed he can focus on automating the attachment of ISO images via server Base Management Controller’s Redfish API.

Prerequisites

The following prerequisites are required before you can continue with the solution:

- An existing Red Hat Enterprise Linux system setup for hosting private BaseOS and AppSteam repositories

- Access to RHEL BOOT ISO file

Tip: Not sure if you meet these prerequisites? Consult the Managing Software in Red Hat Enterprise Linux guide.

Solution diagram for custom Anaconda installer

In the first portion of this solution, we focus on custom installation ISO generation. We start by mounting the boot DVD into the existing rhel01 virtual machine from the Windows Hyper-V host.

# From Hyper-V Host, attach the RHEL Installation BOOT ISO to rhel01 VM
Set-VMDvdDrive -VMName rhel01 `
               -ControllerNumber 1 `
               -ControllerLocation 0 `
               -Path D:\iso\rhel-9.2-x86_64-boot.iso

Next, on rhel01 we gain super user privileges by using sudo in interactive mode sudo -i. We verify that /dev/cdrom device is aware of the ISO and we mount it to /media mountpoint.

# From rhel01 VM, verify ISO Label
blkid /dev/cdrom
/dev/cdrom: UUID="2023-04-13-16-58-02-00" LABEL="RHEL-9-2-0-BaseOS-x86_64" TYPE="iso9660" PTUUID="54f155da" PTTYPE="dos"

# Mount DVD
mount /dev/cdrom /media

Next, we create directories inside the web root directory and recursively copy content from mounted DVD media.

# Create a temporary directory
tmpdir=$(mktemp -d)

# Copy all files from boot DVD to temporary directory
cp -pRf /media/* $tmpdir

Optionally, check out the ISO structure using tree.

# List the ISO structure
tree $tmpdir
/tmp/tmp.vHg3l32pCc
├── EFI
│   └── BOOT
│       ├── BOOTX64.EFI
│       ├── fonts
│       │   └── unicode.pf2
│       ├── grub.cfg
│       ├── grubx64.efi
│       └── mmx64.efi
├── images
│   ├── efiboot.img
│   ├── install.img
│   └── pxeboot
│       ├── initrd.img
│       └── vmlinuz
└── isolinux
    ├── boot.cat
    ├── boot.msg
    ├── grub.conf
    ├── initrd.img
    ├── isolinux.bin
    ├── isolinux.cfg
    ├── ldlinux.c32
    ├── libcom32.c32
    ├── libutil.c32
    ├── memtest
    ├── splash.png
    ├── vesamenu.c32
    └── vmlinuz

Next, we update the GRUB2 configuration file for EUFI-based systems.

# Descrease timeout from 60s to 5s
sed -i 's/set timeout=60/set timeout=5/' $tmpdir/EFI/BOOT/grub.cfg
# Update default menu entry to first one
sed -i 's/set default="1"/set default="0"/' $tmpdir/EFI/BOOT/grub.cfg
# Append inst.* kernel parameters
sed -i '/BaseOS-x86_64 quiet/ s/$/ inst.ks=https:\/\/rhel01.mshome.net\/ks\/cluster-ks.cfg inst.noverifyssl/' $tmpdir/EFI/BOOT/grub.cfg

Next, we update the ISOLINUX configuration file for BIOS-based systems.

# Descrease timeout from 60s to 5s
sed -i 's/timeout 600/timeout 50/' $tmpdir/isolinux/isolinux.cfg
# Update default menu entry to first one
sed -i '/^  menu default$/d' $tmpdir/isolinux/isolinux.cfg
sed -i '/menu label \^Install Red Hat Enterprise Linux 9.2/ a\  menu default' $tmpdir/isolinux/isolinux.cfg
# Append inst.* kernel parameters
sed -i '/BaseOS-x86_64 quiet/ s/$/ inst.ks=https:\/\/rhel01.mshome.net\/ks\/cluster-ks.cfg inst.noverifyssl/' $tmpdir/isolinux/isolinux.cfg

💡Tip: Looking for more information about these configuration files? Have a look at anaconda_customization_guide.

Next, we generate a new ISO image.

# Install tools required to generate ISO image
dnf install -y xorriso isomd5sum

# Create ISO image
genisoimage -U -r -v -T -J -joliet-long \
            -V "RHEL-9-2-0-BaseOS-x86_64" \
            -volset "RHEL-9-2-0-BaseOS-x86_64" \
            -A "RHEL-9-2-0-BaseOS-x86_64" \
            -b isolinux/isolinux.bin \
            -c isolinux/boot.cat \
            -no-emul-boot \
            -boot-load-size 4 \
            -boot-info-table \
            -eltorito-alt-boot \
            -e images/efiboot.img \
            -no-emul-boot -o \
            /tmp/rhel-9.2-x86_64-cluster.iso \
            $tmpdir

# Optionally implant checksum that can be used when rd.live.check option is invoked
implantisomd5 /tmp/rhel-9.2-x86_64-cluster.iso

Next, unmount the ISO from the virtual machine and clean up the temporary build folder.

# From rhel01 virtual machine, unmount DVD
umount /media
rm -rf $tmpdir

Finally, disconnect the DVD from the Hyper-V host machine and download the newly generated ISO image.

# From Hyper-V host, diconnect the ISO from DVD Drive
Set-VMDvdDrive -VMName rhel01 `
               -ControllerNumber 1 `
               -ControllerLocation 0 `
               -Path $null

# Download the custom ISO from rhel01 guest
scp ansible@rhel01.mshome.net:/tmp/rhel-9.2-x86_64-cluster.iso D:\iso\

Next, to simulate a CMDB where real serial numbers would be stored, we create three new virtual machines and note their serial numbers.

$vmNames = "kvm01", "kvm02", "kvm03"

foreach ($vmName in $vmNames) {
    .\vm_create.ps1 -name $vmName `
                    -size S `
                    -isoPath "D:\iso\rhel-9.2-x86_64-cluster.iso" `
                    -state create
}

Below is an example output from the script above.

{
  "ElementName": "kvm01",
  "BIOSSerialNumber": "6087-4370-6019-5585-3621-5785-34"
}
{
  "ElementName": "kvm02",
  "BIOSSerialNumber": "5948-8072-6590-5481-7600-8443-67"
}
{
  "ElementName": "kvm03",
  "BIOSSerialNumber": "6859-8687-6280-6177-6603-0753-51"
}

Next, from rhel01 virtual machine, download and update the /var/www/html/ks/cluster-ks.cfg file. Using sed, we update the placeholders with values from the above output.

# Download the boilerplate kickstart file
curl -o /var/www/html/ks/cluster-ks.cfg \
        https://raw.githubusercontent.com/maroskukan/rhcsa/main/support/ks/cluster-ks.cfg

# Update the serial numbers for the tree VMs
sed -i 's/VM1_SN_PLACEHOLDER/6087-4370-6019-5585-3621-5785-34/' \
       /var/www/html/ks/cluster-ks.cfg
sed -i 's/VM2_SN_PLACEHOLDER/5948-8072-6590-5481-7600-8443-67/' \
       /var/www/html/ks/cluster-ks.cfg
sed -i 's/VM3_SN_PLACEHOLDER/6859-8687-6280-6177-6603-0753-51/' \
       /var/www/html/ks/cluster-ks.cfg

From Hyper-V host shell, power on the three virtual machines using Start-VM wrapped in foreach loop.

foreach ($vmName in $vmNames) {
    Start-VM $vmName
}

After a few minutes, perform a simple connection test targeting SSH.

# From Hyper-V Host, verify if VMs listen on TCP/22 (SSH)
foreach ($vmName in $vmNames) {
  Test-NetConnection -ComputerName "$vmName.mshome.net" `
                     -Port 22 `
                     | Select-Object -Property `
                     @{Name="ComputerName"; Expression={$_.ComputerName}}, `
                     @{Name="RemoteAddress"; Expression={$_.RemoteAddress.IPAddressToString}}, `
                     @{Name="TcpTestSucceeded"; Expression={$_.TcpTestSucceeded}} `
                     | ConvertTo-Json
}

An example of happy output would look like the following:

{
  "ComputerName": "kvm01.mshome.net",
  "RemoteAddress": "172.19.108.188",
  "TcpTestSucceeded": true
}
{
  "ComputerName": "kvm02.mshome.net",
  "RemoteAddress": "172.19.104.7",
  "TcpTestSucceeded": true
}
{
  "ComputerName": "kvm03.mshome.net",
  "RemoteAddress": "172.19.97.83",
  "TcpTestSucceeded": true
}

In summary, we addressed all requirements received from our colleague. When done exploring the environment, we can clean up by removing the three new virtual machine.

foreach ($vmName in $vmNames) {
  .\vm_delete.ps1 -name $vmName
}

Closing thoughts

In conclusion, automating Linux installation saves time, ensures consistency, and boosts productivity for system administrators. Join the conversation in the comments below to share your experiences and thoughts on this powerful approach to installation. Let’s dive deeper into the world of Linux automation together!