Creating a minimal kernel development setup using QEMU and Archlinux

I wanted to setup a linux kernel development environment. Because I use both GNU/Linux, macOS and Windows in my daily life, I also wanted a develoment environment as light as possible.

The best choice I found was using KVM/QEMU: QEMU is a generic and open source machine emulator and virtualizer. QEMU is very flexible and portable. If both host and guest are the same architecture, QEMU can make use of KVM. For instance, when running qemu-system-x86 on a x86 compatible processor, you can take advantage of the KVM acceleration giving you a slightly more powerfull environment.

As the title suggest, I’m using Archlinux. In order to install qemu (without GUI) and all the tools needed to build your image, use:

root:~ # pacman -Suy base-devel linux-headers fakeroot libvirt bc qemu

Installing Archlinux in QEMU

Now, we are ready to start building your kernel development environment.

Check if the host system support kvm virtualization:

boinc:~/lkd $ lscpu | grep --color 'Virtualization:'
Virtualization: VT-x

Creating a hard disk image

To run QEMU you will need a hard disk image. The hard disk image is a file which stores the contents of the emulated hard disk:

boinc:~/lkd $ mkdir qemu-stuff
boinc:~/lkd $ cd qemu-stuff
boinc:~/lkd/qemu-stuff $ qemu-image -f raw kernel-dev-archlinux.img 4G
Formatting 'kernel-deb-archlinux.img', fmt=raw size=4294967296
boinc:~/lkd/qemu-stuff $ ls -l --block-size=G kernel-deb-archlinux.img
-rw-r--r-- 1 boinc sudo 4G Apr 7 15:12 kernel-deb-archlinux.img

Prepare the virtual machine

First, get the latest Archlinux image from archlinux.org:

boinc:~/lkd/qemu-stuff $ wget http://archlinux.mirrors.ovh.net/archlinux/iso/2018.04.01/archlinux-2018.04.01-x86_64.iso
--2018-04-07 15:33:26-- http://archlinux.mirrors.ovh.net/archlinux/iso/2018.04.01/archlinux-2018.04.01-x86_64.iso
Resolving archlinux.mirrors.ovh.net (archlinux.mirrors.ovh.net)... 213.32.5.7, 2001:41d0:202:100:213:32:5:7
Connecting to archlinux.mirrors.ovh.net (archlinux.mirrors.ovh.net)|213.32.5.7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘archlinux-2018.04.01-x86_64.iso’

archlinux-2018.04.01-x86_6 [ <=> ] 556.00M 1.41MB/s in 6m 58s

2018-04-07 15:40:24 (1.33 MB/s) - ‘archlinux-2018.04.01-x86_64.iso’ saved [583008256]

NOTES: be careful to use the latest image.

To install the operating system on the disk image, you must attach both the disk image and the installation media to the virtual machine and have it boot from the installation media. For example, on _x86_64 guests, to install from a bootable ISO file as CD-ROM and a raw disk image:

boinc:~/lkd/qemu-stuff $ qemu-system-x86_64 -cdrom archlinux-2018.04.01-x86_64.iso -boot order=d -drive file=kernel-dev-archlinux.img,format=raw -m 2G

Installing the operating system

Here we will install Archlinux. The Archlinux install is not the subject here, simply follow the installation guide.


Kernel configuration

Go to kernel.org and click on the big yellow button. Once you have the kernel source on your filesystem, you will need to extract the tarball in your workspace:

boinc:~/lkd $ xz -cd linux-*.tar.xz | tar xvf -

When compiling the kernel, all output files will be stored together with kernel source code. Using the option make O=dir/ allows you to specify an alternate place for the output files.

boinc:~/lkd $ mkdir kernel.build

To configure the kernel, use:

boinc:~/lkd/ $ cd linux-4.16
boinc:~/lkd/linux-4.16 $ make O=../kernel.build defconfig
make[1]: Entering directory '/home/boinc/lkd/kernel.build'
HOSTCC scripts/basic/fixdep
GEN ./Makefile
HOSTCC scripts/kconfig/conf.o
YACC scripts/kconfig/zconf.tab.c
LEX scripts/kconfig/zconf.lex.c
HOSTCC scripts/kconfig/zconf.tab.o
HOSTLD scripts/kconfig/conf
make[1]: Leaving directory '/home/boinc/lkd/kernel.build'

To build the kernel, use:

boinc:~/lkd/linux-4.16 $ make O=../kernel.build -j$(grep -c ^processor /proc/cpuinfo) bzImage
make[1]: Entering directory '/home/boinc/lkd/kernel.build'
CHK include/config/kernel.release
GEN ./Makefile
CHK include/generated/uapi/linux/version.h
CHK include/generated/utsrelease.h
DESCEND objtool
Using /home/boinc/lkd/linux-4.16 as source for kernel
CHK scripts/mod/devicetable-offsets.h
CHK include/generated/bounds.h
CHK include/generated/timeconst.h
CHK include/generated/asm-offsets.h
CALL /home/boinc/lkd/linux-4.16/scripts/checksyscalls.sh
CHK include/generated/compile.h
DATAREL arch/x86/boot/compressed/vmlinux
[...]
Kernel: arch/x86/boot/bzImage is ready (#1)
make[1]: Leaving directory '/home/boinc/lkd/kernel.build'

Boot the virtual machine

To make the connection to our virtual machine more simple, we will write a bash script to set up QEMU with all the options.

boinc:~/lkd $ cat qemu-stuff/qemu-term.sh
#!/bin/bash

KERNEL="/home/boinc/lkd/kernel.build/arch/x86_64/boot/bzImage"
RAM=2G
DISK="/home/boinc/lkd/qemu-stuff/kernel-dev-archlinux.img"

qemu-system-x86_64 \
-enable-kvm \
-hda $DISK \
-m $RAM \
-net nic -net user,hostfwd=tcp::2222-:22 \
-serial stdio \
-kernel $KERNEL \
-append "root=/dev/sda2 console=ttyS0 rw" \
-display none

Hello world

To compile the module, I will use a Makefile found here.

boinc:~/lkd/hello $ cat Makefile
# If KERNELRELEASE is defined, we've been invoked from the
# kernel build system and can use its language.
ifneq ($(KERNELRELEASE),)
obj-m := hello.o
# Otherwise we were called directly from the command
# line; invoke the kernel build system.
else
KERNELDIR ?= ../linux-4.16
BUILDDIR ?= ../kernel.build
PWD := $(shell pwd)
default:
$(MAKE) -C $(KERNELDIR) O=$(BUILDDIR) M=$(PWD) modules
endif

Of course, you need to adjust the KERNELDIR and BUILDDIR variables to point to the root of your kernel sources, and to your build directory, respectively.

Now the first module example, which simply prints hello world in the log file.

#include <linux/init.h>
#include <linux/module.h>

static int hello_init(void)
{
printk(KERN_ALERT "Hello, world\n");
return 0;
}

static void hello_exit(void)
{
printk(KERN_ALERT "Goodbye, cruel world\n");
}

module_init(hello_init);
module_exit(hello_exit);
MODULE_LICENSE("GPL");

Then, compile the hello modules:

boinc:~/lkd/hello $ make
make -C ../linux-4.16 O=../kernel.build M=/home/boinc/lkd/hello modules
make[1]: Entering directory '/home/boinc/lkd/linux-4.16'
make[2]: Entering directory '/home/boinc/lkd/kernel.build'
Building modules, stage 2.
MODPOST 1 modules
LD [M] /home/boinc/lkd/hello/hello.ko
make[2]: Leaving directory '/home/boinc/lkd/kernel.build'
make[1]: Leaving directory '/home/boinc/lkd/linux-4.16'

Copy the module to the virutal machine with scp command, type:

boinc:~/lkd/hello $ scp -P 10022 hello.ko 127.0.0.1:hello.ko
boinc@127.0.0.1's password: ********
hello.ko 100% 3584 5.8MB/s 00:00

To load the module, type:

[root@archlinux boinc]# insmod hello.ko
[ 720.135159] hello: loading out-of-tree module taints kernel.
[ 720.137814] Hello, world

To remove the module, type:

[root@archlinux boinc]# rmmod hello
[ 757.934665] Goodbye, cruel world

Thank you for following this tutorial :-)

[1] http://www.lifl.fr/~lipari/courses/ase_lkp/ase_lkp.html

[2] https://saurorja.org/2011/07/04/creating-a-minimal-kernel-development-setup-using-kvmqemu/

[3] https://www.kernel.org/

[4] https://mgalgs.github.io/2015/05/16/how-to-build-a-custom-linux-kernel-for-qemu-2015-edition.html

[5] https://serverfault.com/questions/208693/difference-between-kvm-and-qemu

[6] https://wiki.qemu.org/Features/KVM

[7] https://wiki.archlinux.org/index.php/QEMU

[8] https://wiki.archlinux.org/index.php/Installation_guide

[9] https://unix.stackexchange.com/questions/124681/how-to-ssh-from-host-to-guest-using-qemu/196074

[10] https://askubuntu.com/questions/205179/ssh-problem-read-from-socket-failed-connection-reset-by-peer