Linux Kernel Compilation and Adding a Custom System Call

Hi, in this tutorial, we’ll learn how we can compile Linux Kernel and then how we can make changes to our kernel.

Prerequisites

Please make sure you have a good amount of free space. For Virtual Machines, people were having issues when they allocated only 20GB of storage space. So at least allocate 50GBs of storage and a minimum of 3GBs of RAM to avoid issues.

Before moving ahead:

sudo apt apdate && sudo apt upgrade

Install the following packages before moving to the next step:

sudo apt install gcc make bc build-essential libelf-dev libssl-dev bison flex initramfs-tools git-core libncurses5-dev dwarves

Now you can follow these steps. First, we’ll compile the kernel and test it’s running. Then we’ll add our custom system call.

Compiling Linux Kernel

Step 1

Download the kernel file from www.kernel.org. I recommend downloading Linux Kernel 5.3.7 Or above version because the method of adding system calls described here is for kernel 5.3.7 or above.

For simplicity, you can download it from this direct link (Kernel 5.10.117):

https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.10.117.tar.xz

If you are looking to download any other version, for simplicity, you will need to download targun file, as shown in the picture below.

Step 2

When you have downloaded the archive file linux-5.x.x.tar.xz, you need to extract the file in some directory.

tar xvf linux-5.x.x.tar.xz  #use correct file name
cd linux-5.x.x.tar.xz

Now you are in the directory of kernel files. You can verify by list file using ls the command.

Step 3

Now you need to configure the flavour of the kernel. Configuring the kernel involves selecting which features you want to build into the kernel image, which features you want to build as loadable modules, and which features you want to omit entirely. The configure process will finally create a .config script for building the kernel. The .config file contains the configuration information (which has information about the features to be installed) of the kernel to be compiled.

There are different methods to configure the Linux kernel, but we’ll go with the old kernel configuration. Here we’ll copy the old configurations form /boot/config-x.x.x-x-generic , and we’ll create a new .config in our kernel’s directory.

like

cp /boot/config-5.13.0-40-generic ./.config

Now in this .config file, we need to make some changes.

1. Open the .config file in vim.
2. Search for CONFIG_SYSTEM_TRUSTED_KEYS and assign an empty string to it (i.e., double quotes"")
3. Search for CONFIG_SYSTEM_REVOCATION_KEYS and assign an empty string to it (i.e., double quotes"")

Now save the file and quit. :wq!

Now run the following command to configure:

yes '' | make oldconfig # be patience. It will take some time

Till now, you have made the configuration. But still, you need to distinguish your kernel from other versions. To do so, Open the Makefile in the kernel’s root directory and change the variable. EXTRAVERSION With some names like this:

Now to make a copy of the kernel release, run the following command:

make kernelrelease # after success a kernel release will be printed like 5.10.117-arm

Step 4

Now it’s time to build/compile the kernel release.

To do so, you can you the following commands:

make -j $(nproc) bzImage # nproc prints the number of your system's cores. and by this command i am using all of my system's cores to make this process faster
# after success of the above command, run the following command.
make -j $(nproc) modules

Step 5

Till now, your kernel release has been compiled. Now to install it, use the following command.

sudo make INSTALL_MOD_STRIP=1 modules_install
# INSTALL_MOD_STRIP=1 will reduce the size of the modules before installing them.

This step will take just a second and create a new directory /lib/modules/5.10.117-arm/ and copy all the .ko files (modules) over there.

Now run the following command.

sudo make install

The install section of Makefile will move the files to their destination locations mentioned in the DIR variables (in the Makefile like BIN_DIR, MAN_DIR, BIN_DIR_D, …). Instead of using mv or cp command the install target of Makefile use Linux install command that not only moves files but also changes permissions to those files.

Step 6

Till now, your kernel has been compiled and installed. But still, we need to update the grub boot loader to boot our system from the custom kernel. We need to increase the booting selection process to select our kernel. To do so, open the file /etc/default/grub in vim and change the GRUB_TIMEOUT value from 5 to 30.

Moreover, let the GRUB_DEFAULT value to 0, which means, by default booting from the first entry of the kernel in the /boot/grub/grub.cfg file, which will be the newly installed kernel.

After making these changes, you need to update the boot loader using the following command:

sudo update-grub2

This will update GRUB with the new kernel. And if it is the latest kernel version, it will become the default to be loaded when rebooting. Otherwise, we need to explicitly select the kernel to run during the booting process. Restart the system to open the bootloader; hold the shift key during system startup, and you will see a screen with all installed kernels.

Congratulations, your kernel is ready to boot.

Restart your system to boot from the new kernel.

sudo systemctl reboot

Adding system call

Till now, we have compiled and installed our configured Linux kernel release. Now it’s time to make actual changes in the kernel source code to have the motivation that we are making changes to the kernel.

We’ll add only our custom hello world system call this time.

💡 Make sure that you should not have booted from your custom kernel to follow these steps.

Step 1

1. Create a c file in which you’ll define your system call. As for now, we are adding a just hello world system call, so use these c files.

mkdir my_syscall # create this directory inside linux source root directory
cd my_syscall
touch hello_world.c

1. Add the following code snippet, as this is your custom system call code.

#include <linux/kernel.h>
#include <linux/syscalls.h>
SYSCALL_DEFINE0(hello_world) 
{
	printk("Hello Ali. This is me, your custom system call.\\n");
  // printk will print to the kernel log. do see man page for more details
	return 0;
}

1. Create a Makefile inside your systems call directory and add the following line in it.

vim Makefile
obj-y := hello_world.o

Step 2

Till now, we have designed our system call. Now we need to add and link this system call with our kernel.

We need to link our system call with the kernel’s syscalls header. To do so, add the following line at the end of include/linux/syscalls.h

vim include/linux/syscalls.h
# add this like at the end of 
asmlinkage long sys_hello_world (void);

Step 3

We need to register our system call in the syscall table to get a unique system call number. For this open arch/x86/entry/syscalls/syscall_64.tbl

vim arch/x86/entry/syscalls/syscall_64.tbl

Add the following entry at the end of the file where the last system call is registered.

💡 Remember, use a unique system call number in the 1st column of this table and do not use the reserved number. Reserved numbers are already mentioned in this word table file.

For simplicity, I have used the 696 number for my system call.

696 64 hello_world sys_hello_world

Step 4

We need to add our system call’s directory in the kernel source. Makefile.

Open the Makefile and search for the core-y entry.

At the end of this line, add the system call’s directory after a space and a / at the end of the line. For simplicity, see the following picture.

core-y          += kernel/ certs/ mm/ fs/ ipc/ security/ crypto/ block/ my_syscall/

Step 5

We need to recompile and install our kernel to test this system call.

make -j $(nproc)
sudo make install

Use the following command to reboot and hold the shift key to select your custom kernel.

sudo systemctl reboot

💡 Remember, this time, you’ll find a .old kernel file in the grub boot loader but don’t boot from it. Boot from your custom kernel.

Step 6

Now it’s time to test the changes you made with your kernel release, i.e., testing system call.

💡 But remember, you must be booted from your new kernel release. You can verify your kernel that is running by using uname -r the command.

Write a driver program anywhere in the system and test your system call.

#include <stdio.h>
#include <unistd.h>
#include <sys/syscall.h>
int
main(){
        long rv = syscall(696); // 696 is my system call's number
        if(rv == -1){ // it means no system call found with provided syacll number
                printf("error\\n");
                return -1;
        }
        return 0;
}

Since I have use printk Write output on the kernel log file so it won’t be printed on shell/terminal. To check the kernel log, use dmesg and bind it to see the last entries after running the driver program. You’ll find your system call’s output. Which means your system call is working.

Yes, you did it.

I hope it was an informative tutorial.

Happy Learning and, of course:

Learning linux is fun with Dr. Arif Butt ;)

~Ali Raza Mumtaz