Prepare the environment for developing Linux kernel with qemu.

10 min readOct 4, 2019

I would share the environment for developing linux kernel with Qemu. I think the Qemu is powerful hardware emulator to run Linux kernel for developer. This ables you to run Linux kernel without real hardware and also provides features to analyze Linux kernel itself.

OS what I used for preparing the environment to develop Linux kernel is Ubuntu 18.04 and I will install qemu for x86_64 guest. It has advantage of performance with KVM if the architecture is same with host and qemu guest.

Start following order:

Install packages releated to building Linux kernel and running it on Qemu.
Clone Linux kernel source code, configure and build for running on qemu-x86_64 guest.
GDB to set break point
buildroot is for making root file system.

Install packages

Install git packages to clone the Linux Kernel source code from the linux-next branch in mainline of Linux Kernel Repository

$ sudo apt install git

Install packages to build linux kernel from the source code.

$ sudo apt install build-essential kernel-package fakeroot libncurses5-dev libssl-dev ccache flex bison libelf-dev

Clone Linux Kernel Source from the development branch.

Linux kernel has ‘linux-next’ branch to take patches from developers. You can take a look at the tree with this link. If you want to fix issues or adds features to the Linux kernel, you should get this(linux-next) branch and remain latest. There are lots of subsystem maintainer’s branches but linux-next branch is good to start Linux Kernel development.

$ git clone https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next.git

If you are tracking linux-next branch with git, you should not use git pull. Instead, you should use git fetch origin and git reset --hard remotes/origin/master.

$ cd linux-next
$ git checkout master
$ git fetch origin
$ git reset --hard remotes/origin/master

These set latest released source code in master (local) branch.

Configure the Linux Kernel

Configure the Linux kernel to build kernel image that is able to run on qemu(x86_64 architecture). You can find config file what is used as base for x86_64 architecture in arch/x86/configs/x86_64_defconfig. Let’s find out how to .config file as default configuration file for x86_64 architecture.

$ make ARCH=x86_64 x86_64_defconfig 
  YACC    scripts/kconfig/parser.tab.[ch]
  HOSTCC  scripts/kconfig/lexer.lex.o
  HOSTCC  scripts/kconfig/parser.tab.o
  HOSTCC  scripts/kconfig/preprocess.o
  HOSTCC  scripts/kconfig/symbol.o
  HOSTLD  scripts/kconfig/conf
#
# configuration written to .config
#

You can check the .config file is generated and it will be used in kernel compile. But there are some options what are added to .config for GDB debugger.

$ make ARCH=x86_64 menuconfig

Above command will open TUI(Text-based UI) to help you configure the linux kernel easly and also load current .config file automatically.

[pic-1] Text-Based UI Linux Kernel Configuration tool — Text-Based UI Kernel configuration tool.

Some options should be enabled to debug kernel with qemu. Move cursor to “Kernel hacking” and enter into it.

Select “Compile the kernel with debug info” option first, and then show more options related to kernel debug. Need to select “Provide GDB scripts for kernel debugging (New)” option. But you should leave “Reduce debugging information” option off.

Exit kernel configuration with selecting “< Exit >” menu in bottom of menu. And save it with your selection.

Now we can compile the Linux kernel with “make” command.

$ make -j8

If the build is done, you can see the message like below:

...
OBJCOPY arch/x86/boot/setup.bin
  BUILD   arch/x86/boot/bzImage
Setup is 16124 bytes (padded to 16384 bytes).
System is 8673 kB
CRC f5ca994b
Kernel: arch/x86/boot/bzImage is ready  (#5)

Install the qemu and run it with kernel image

You are ready to run Linux kernel on qemu, but before, you should install packages to run qemu.

sudo apt install qemu qemu-system

Then, you can use the “qemu-system-x86_64" tool for running our kernel image what compiled in previous step.

$ qemu-system-x86_64 -no-kvm -kernel arch/x86/boot/bzImage -hda /dev/zero -append "root=/dev/zero console=ttyS0" -serial stdio -display none

I just use pseudo console to show the kernel logs instead of using virtual display. If it works properly, you can see the kernel messages on your terminal.

$ qemu-system-x86_64 -no-kvm -kernel arch/x86/boot/bzImage -hda /dev/zero -append "root=/dev/zero console=ttyS0" -serial stdio -display none
[    0.000000] Linux version 5.3.0-next-20190920 (daeseok@AD01277334) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #5 SMP Thu Sep 26 19:29:00 KST 2019
[    0.000000] Command line: root=/dev/zero console=ttyS0
[    0.000000] x86/fpu: x87 FPU will use FXSAVE
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000007fdffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000007fe0000-0x0000000007ffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.8 present.
[    0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 3599.980 MHz processor
[    0.004163] last_pfn = 0x7fe0 max_arch_pfn = 0x400000000
[    0.004784] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
[    0.013624] found SMP MP-table at [mem 0x000f6aa0-0x000f6aaf]
[    0.014744] check: Scanning 1 areas for low memory corruption
[    0.017488] ACPI: Early table checksum verification disabled
[    0.017710] ACPI: RSDP 0x00000000000F68C0 000014 (v00 BOCHS )
[    0.017855] ACPI: RSDT 0x0000000007FE15FC 000030 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
[    0.018231] ACPI: FACP 0x0000000007FE1458 000074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
[    0.018638] ACPI: DSDT 0x0000000007FE0040 001418 (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
[    0.018699] ACPI: FACS 0x0000000007FE0000 000040
.....

Above execution has no file system image which could be mounted in qemu virtual environment, you can see the kernel panic message related to mounting file system. no wrorry about that. And “Ctrl + c” key will exit from the qemu.

Access running qemu system and debug the kernel with GDB

You need to “gdb” tool for accessing running linux kernel on Qemu. Install it following command:

$ sudo apt install gdb

There are two options what you should add it to qemu before running. One is “-s”(lowercase ) that option means use the port 1234 as tcp(tcp::1234) to connect it through gdb. Another is “-S” (uppercase) that option means stop cpu until continue from gdb what is connected to tcp 1234 port.

$ qemu-system-x86_64 -s -S -no-kvm -kernel arch/x86/boot/bzImage -hda /dev/zero -append "root=/dev/zero console=ttyS0 nokaslr" -serial stdio -display none

There is one more point to take care before debugging with gdb. You should turn off the KASLR option by adding “nokaslr” to the kernel command line. Take a look after “-append”, you can see the “nokaslr”. If you want to know more details about KASLR, visit this article.

If you run above command, qemu will wait to be continued by gdb. Qemu will give some warning message but you don’t need to care about it.

Try to open another terminal to run gdb.

$ cd /path/to/top/of/linux/kernel/source
$ gdb ./vmlinuz
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
....
(gdb)

Type “target remote localhost:1234” in gdb shell.

(gdb) target remote localhost:1234
Remote debugging using localhost:1234
0x000000000000fff0 in entry_stack_storage ()
(gdb) break start_kernel
break start_kernel
Breakpoint 1 at 0xffffffff829abb5b: file init/main.c, line 576.

You can connect qemu through localhost:1234 and then you can break start point of kernel(start_kernel function is first C language code) with ‘break’ command of gdb.

(gdb) continue
Continuing.Breakpoint 1, start_kernel () at init/main.c:576
576 {
(gdb)

Start kernel by putting ‘continue’ command of gdb, and break the kernel right after. The terminal that qemu was excuted didn’t any changes because the ‘start_kernel’ function is early stage of linux kernel so there is nothing to show kernel message. You can check the source code in gdb terminal.

(gdb) list
571 {
572  rest_init();
573 }
574 
575 asmlinkage __visible void __init start_kernel(void)
576 {
577  char *command_line;
578  char *after_dashes;
579 
580  set_task_stack_end_magic(&init_task);
(gdb)

The kernel is stopped in main.c:575. You can use ’n’ (next), ‘c’ (continue) and so on. Put another ‘continue’ command, then you can see the messages in qemu executed terminal. If you didn’t set other break point to debug, the gdb could not show ‘(gdb)’ and wait qemu to be exited.

Create Root file system from buildroot project

Buildroot helps you making root file system easly and created root file system image will be mounted in qemu system. The buildroot is normally used for embedded system but I think it is good enough to debug the linux kernel with qemu.

First, clone the buildroot project.

$ git clone git://git.buildroot.net/buildroot
$ cd buildroot

Second, configure buildroot option with ‘make menuconfig’.

$ make menuconfig

Select “Target Options” to run it on x86_64 virtualized system with qemu.

Target Options -> Target Architecture →

Filesystem images → ext2/3/4 root file system

Choose ext4 file system to be mounted as default file system for the system.

Build ‘buildroot’ project with ‘make’

$ make -j8
....
ln -sf rootfs.ext2 /home/daeseok/work/buildroot/output/images/rootfs.ext4
>>>   Generating filesystem image rootfs.tar
mkdir -p /home/daeseok/work/buildroot/output/images
rm -rf /home/daeseok/work/buildroot/output/build/buildroot-fs/tar
mkdir -p /home/daeseok/work/buildroot/output/build/buildroot-fs/tar
rsync -auH --exclude=/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM /home/daeseok/work/buildroot/output/target/ /home/daeseok/work/buildroot/output/build/buildroot-fs/tar/target
echo '#!/bin/sh' > /home/daeseok/work/buildroot/output/build/buildroot-fs/tar/fakeroot
echo "set -e" >> /home/daeseok/work/buildroot/output/build/buildroot-fs/tar/fakeroot
echo "chown -h -R 0:0 /home/daeseok/work/buildroot/output/build/buildroot-fs/tar/target" >> /home/daeseok/work/buildroot/output/build/buildroot-fs/tar/fakeroot
PATH="/home/daeseok/work/buildroot/output/host/bin:/home/daeseok/work/buildroot/output/host/sbin:/home/daeseok/.nvm/versions/node/v10.16.0/bin:/home/daeseok/.local/bin:/home/daeseok/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/daeseok/bin" /home/daeseok/work/buildroot/support/scripts/mkusers /home/daeseok/work/buildroot/output/build/buildroot-fs/full_users_table.txt /home/daeseok/work/buildroot/output/build/buildroot-fs/tar/target >> /home/daeseok/work/buildroot/output/build/buildroot-fs/tar/fakeroot
....

The rootfs.ext4 has been created, so you can use the generated image could be used as root file system in qemu. Check the output file in buildroot/output/images/ directory. This image file will be used in qemu execution.

Run qemu with own Linux Kernel and Root file system

$ cd /path/to/linux-next
$ qemu-system-x86_64 -kernel arch/x86/boot/bzImage \
-boot c -m 2049M -hda ../buildroot/output/images/rootfs.ext4 \
-append "root=/dev/sda rw console=ttyS0,115200 acpi=off nokaslr" \
-serial stdio -display none

Test rootfs.ext4 image file that it works properly with qemu. Run qemu (above) with rootfs.ext4.(Please check the path of rootfs.ext4 file) Then you will see the login shell in last kernel message.

[    0.000000] Linux version 5.3.0-next-20190920 (daeseok@AD01277334) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #5 SMP Thu Sep 26 19:29:00 KST 2019
[    0.000000] Command line: root=/dev/sda rw console=ttyS0,115200 acpi=off nokaslr
[    0.000000] x86/fpu: x87 FPU will use FXSAVE
[    0.000000] BIOS-provided physical RAM map:
....skip
[    2.913761] EXT4-fs (sda): mounted filesystem with ordered data mode. Opts: (null)
[    2.915210] Mounted ext4 file system at /root supports timestamps until 2038 (0x7fffffff)
[    2.916011] VFS: Mounted root (ext4 filesystem) on device 8:0.
[    2.921939] devtmpfs: mounted
[    2.962863] Freeing unused kernel image memory: 1332K
[    2.963173] Write protecting the kernel read-only data: 20480k
[    2.965482] Freeing unused kernel image memory: 2004K
[    2.966324] Freeing unused kernel image memory: 956K
[    2.966546] Run /sbin/init as init process
[    3.062589] EXT4-fs (sda): re-mounted. Opts: (null)
[    3.062871] Mounted ext4 file system at / supports timestamps until 2038 (0x7fffffff)
Starting syslogd: OK
Starting klogd: OK
Running sysctl: [    3.357887] find (116) used greatest stack depth: 13976 bytes left
[    3.363719] S02sysctl (115) used greatest stack depth: 13880 bytes left
OK
Initializing random number generator... [    3.461757] random: dd: uninitialized urandom read (512 bytes read)
done.
Starting network: [    3.553592] ip (132) used greatest stack depth: 13800 bytes left
OKWelcome to Buildroot
buildroot login:

Buildroot has ‘root’ as default login user without password. Just put root after buildroot login: then you will get the shell.

Welcome to Buildroot
buildroot login: root
# mount
/dev/root on / type ext4 (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=1015368k,nr_inodes=253842,mode=755)
proc on /proc type proc (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
tmpfs on /dev/shm type tmpfs (rw,relatime,mode=777)
tmpfs on /tmp type tmpfs (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
sysfs on /sys type sysfs (rw,relatime)
# pwd
/root
#

Now, you can connect qemu with gdb. It means you can trace the source code any kernel features like file system, scheduler and memory management. It helps you study the inside of Linux kernel. Try set break point of source code what you want to analyze flow of some function. Note that you should use ‘-s’ and ‘-S’ options in qemu to debugging Linux kernel.

For example)

1. Run Qemu
<please have a look at options for qemu.>

$ qemu-system-x86_64 -s -kernel arch/x86/boot/bzImage -boot c -m 2049M -hda ../buildroot/output/images/rootfs.ext2 -append "root=/dev/sda rw console=ttyS0,115200 acpi=off nokaslr" -serial stdio -display none
...
Welcome to Buildroot
buildroot login:

2. Run gdb and connect qemu in gdb shell

$ gdb ./vmlinux
...
(gdb) target remote :1234
Remote debugging using :1234
default_idle () at arch/x86/kernel/process.c:581
581  trace_cpu_idle_rcuidle(PWR_EVENT_EXIT, smp_processor_id());

When you put ‘target remote :1234’, the qemu stops in somewhere of linux kernel and print breaked point. And set another break point. I tried to set break point in mm/page_alloc.c:4767 (in __get_free_pages() function) and continue kernel with c command.

(gdb) b mm/page_alloc.c:4767
Breakpoint 1 at 0xffffffff811a5f60: file mm/page_alloc.c, line 4767.
(gdb) c
Continuing.

After put ‘c’ command in gdb, you can input ‘root’ as user to login buildroot shell.

Welcome to Buildroot
buildroot login: root <enter>

And nothing happended in qemu, but you can see the message in gdb shell. You can input command gdb shell. Just put list command. You will get the source code what you set break point before.

Breakpoint 1, __get_free_pages (gfp_mask=4197824, order=1) at mm/page_alloc.c:4767
4767  page = alloc_pages(gfp_mask & ~__GFP_HIGHMEM, order);
(gdb)list
4762  */
4763 unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order)
4764 {
4765  struct page *page;
4766 
4767  page = alloc_pages(gfp_mask & ~__GFP_HIGHMEM, order);
4768  if (!page)
4769   return 0;
4770  return (unsigned long) page_address(page);
4771 }

Conclusion

I think the Linux kernel is really hard to start studying. Because I could not figure out where to start the code looking and also I want to see the values and status when some function is called like __get_free_pages(). So if you use qemu and gdb, these make you easy to read the source code and easy to check the values while the function is handled.