Xen virtualization on BeagleBoard-x15: what was done during GSoC 2019
Last three months I was working on adding virtualization support for Beagleboard-x15.
Link to the code: link
Link to changes from master Xen: contribution
Part of the code was already sent as a patch for Xen and in the process of discussion, other part is in the process of preparing the patch for Xen.
The main problem in virtualization on OMAP AM5728 was its interrupt controller. By default, Xen supports GICv2 and GICv3 interrupts controllers on arm32 and arm64 target. And the Beagleboard-x15’s SoC also has GICv2 but it also provides users with the crossbar device.
A small note: "omap5" is the platform but the actual devices that are available are the AM57xx or DRA74xx. As background omap5 was the first TI product with the dual A15 architecture but it was targeted at mobile handsets and is no longer available. The AM57xx and DRA74xx devices were later derivatives reusing the main processing core but with a range of peripherals targetting the industrial and automotive markets respectively.
The crossbar device is a special kind of a device that allows to map one of the 420 (SPI) interrupts to 191 lines of GICv2. More detailed information can be found here: spruhz6k.pdf. The default mapping is listed in chapter 17 of the mentioned TRM.
The crossbar device is listed in the device-tree as an interrupt-controller. And the main problem was that Xen is designed to work with only one interrupt controller (which is the case for most of the platforms). Another noticeable exception from this rule is nVidia Tegra platform. Some time ago a patch series was proposed: tegra patch. The main difference between OMAP’s crossbar and tegra’s lic (legacy interrupt controller) is that crossbar allows to remap interrupts to different GIC input lines. So, in my work the following tasks were solved:
- to teach Xen deal with several interrupt controllers, some of them being platform-specific;
- to implement the logic of dealing with crossbar managing by dom0 virtual machine.
After several attempts, the following solution was created. Firstly, the crossbar is initialized by Xen and set up in the way that allows Xen’s console to receive interrupts. To achieve this, we statically map uart3 (serial debug) interrupts to the predefined number (to line 4). Also, we hide this interrupt line from being accessed by dom0 machine and prevent accesses to the register controlling the line.
Secondly, Xen was taught to traverse the device tree while looking for an appropriate interrupt-controller during dom0 domain creation (if the interrupt parent is not GICv2, Xen will try to achieve it transitively by looking at interrupt-parent property of a node listed as an interrupt-parent of the current node and etc). After that, all the virtual interrupts are mapped to dom0 in a straight way, so VGIC interrupt numbers are equal to GIC interrupt numbers. At the same time, interrupts from devices are mapped iteratively. So, each next device interrupt will be assigned to the next available interrupt crossbar slot. If there are no available crossbar lines, the interrupts are mapped to the last 159 crossbar’s line. It is important for dom0 to initialize the crossbar (which is done in Linux by default) in a preferred manner.
Small note: in device trees a device has interrupt property which specifies its interrupt number and interrupt-parent that shows to what interrupt controller the interrupt is routed. In our case, we have the following problem. A device has some interrupt number and crossbar as an interrupt-parent. Thus, this number shows to what input line of crossbar the device’s interrupt is attached. On the other side, Xen can work only with one interrupt controller (GIC: GICv2 or GICv3). So, when Xen sees that a device interrupt is attached to some interrupt controller other than GIC, it can’t appropriately handle it. Another difficulty is that Xen exposes not the GIC itself but a virtual interrupt controller — vGIC. And generally GIC interrupt number is not equal to vGIC interrupt number. Thus, in my solution I need to show Xen how to treat interrupts with additional interrupt controllers between GIC and device and map interrupts through crossbar to the real GIC considering the fact that only vGIC can be seen by dom0. The last fact means that any changes made to the crossbar (rerouting to another GIC line) should be correctly reflected in vGIC.
And finally, special IO handlers are installed to filter writes and reads to/from the crossbar. Those IO handlers are special functions which are called when dom0/domU tries to access the handler’s region.
How to use it
You need the following components:
- u-boot bootloader
- Xen (version with my patch you can download from here: temporarily link)
- Linux kernel
- busybox.
First step is to build those components. You can build them following these instructions: how to build images. After that step you should have the following images:
- Linux zImage
- initrd (rootfs.img.gz) containing the root filesystem
- u-boot (u-boot.bin and MLO) first stage and second stage bootloaders
- bbx15.dtb binary device tree for BeagleBoard-x15
- Xen image.
Having those components, we can start booting. Firstly, we should connect our BB-x15 to out computer. We should connect it via Serial Debug port (using e.g. ftdi adapter), the scheme of Serial Debug port is (you can find its description in BeagleBoard X15 System Reference Manual or in BeagleBoard-x15 Quick Start Guide):
1 — GND4 — RX5 — TX
Now, we can open the console and run a remote terminal session:
sudo minicom -D /dev/ttyUSB0where -D specifies your connection file.
And we also should connect via Ethernet cable, I found out that the top RJ-45 port is working for loading.
Preparing the bootloader
On this stage we can use two different options — you can either use the default u-boot bootloader in on-board emmc memory of your board or to write your own image to SD card.
In the first case you should just start your board connected to the host via Serial Debug and Ethernet cables and push the button during the u-boot stage of loading. After that you will be able to write u-boot commands. In my case I push space when I see this line:
Press SPACE to abort autoboot in 2 secondsThen I am prompted to write something.
The second option assumes that you have MLO and u-boot.bin files. You can write them on your SD card. To do so, you should firstly clear your card (it’ll delete your data), and write both the first- and second stage bootloaders using instruction from the following link and link.
Booting via FTP
After that, on the host computer we should start tftpd daemon. I installed tftpd-hpa
apt install tftpd-hpaand used the following config:
# /etc/default/tftpd-hpa
TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/home/embden/Projects/bb-x15/build/"
TFTP_ADDRESS=":69"
TFTP_OPTIONS="--secure"RUN_DAEMON="yes"
where TFTP_DIRECTORY contains built images of Xen, busybox and Linux.
Now, you can restart tftpd service:
systemctl restart tftpd-hpaBooting
When we boot into u-boot, we should paste the following u-boot commands:
setenv serverip 192.168.255.1
setenv ipaddr 192.168.255.2
setenv xen_bootargs 'dom0_mem=512M console=dtuart dtuart=serial2 sync_console'tftpboot 0x82000000 zImage; tftpboot 0x83000000 bbx15.dtb; tftpboot 0x84000000 rootfs.img.gz; tftpboot 0x85000000 xen;fdt addr 0x83000000
fdt resize
fdt set /timer clock-frequency <6144000>
fdt set /chosen \#address-cells <1>
fdt set /chosen \#size-cells <1>
fdt set /chosen xen,xen-bootargs \"$xen_bootargs\"
fdt mknod /chosen module@0
fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
fdt set /chosen/module@0 reg <0x82000000 LINUX_IMAGE_SIZE_HERE>
fdt set /chosen/module@0 bootargs "rw root=/dev/ram rdinit=/sbin/init console=hvc0 earlycon=xenboot mem=512M clk_ignore_unused"fdt mknod /chosen module@1
fdt set /chosen/module@1 compatible "xen,linux-initrd" "xen,multiboot-module"
fdt set /chosen/module@1 reg <0x84000000 INITRD_IMAGE_SIZE_HERE>
fdt rm /ocp/interconnect@48000000/segment@0/target-module@b4000
fdt rm /ocp/interconnect@48000000/segment@0/target-module@9c000
fdt set /ocp/crossbar ti,irqs-skip <0x00000004 0x0000000a 0x00000085 0x0000008b 0x0000008c>bootz 0x85000000 - 0x83000000
In first three lines we specify network settings, then we boot images at the specified memory addresses, then we put into the device tree some options needed for Xen, like Xen-bootargs, missing clock frequency, Linux and initrd locations and sizes, we also remove some elements that make booting fail and we remove an interrupt used by Xen’s console (serial debug port, uart3) by adding it to ti,irqs-skip node.
To find out a file size, I use the following command:
printf “0x%x\n” $(stat -c %s ../build/zImage)where ../build/zImage is the path to a file (Linux kernel image in my case).
Since, we use interrupt line 4 (the first available in crossbar) for the Xen console, we want to mark it in ti,irqs-skip. That makes Linux treat the fourth interrupt line as not usable. If everything is right you will see the busybox command line prompt:
/ #In order to test our system we can ping the host. Firstly, we should set up the connection:
ip address add 192.168.255.3/24 dev eth0
ip link set eth0 upAnd then ping the host:
ping 192.168.255.1If you can’t stop pinging (it doesn’t react on CTRL-C) then you should type the following command:
kill $(pgrep ping)Just type it and hit Enter.
Project status
The project timeline can be found here.
Some milestones weren’t achieved and some of them turned out to be redundant. For example, though documentation on debugging (with a jtag-debugger) was planned, it didn’t contribute directly to the main goal, so the decision was taken to omit the step. At the same time, domU-related milestones weren’t achieved at all. Since, the support of domU more or less trivial and should work almost “out of box”, I decided to work more on the code for better dom0 support.
Future work
The work done during GSoC 2019 provides users with ability to run Xen on Beagleboard-x15. Though, it is now possible to run Xen and dom0, there are still a lot of things to do.
Firstly, the solution wasn’t properly tested and might be not completely secure. For example, we have only visually concluded that interrupts work (they appear in /proc/interrupts and Ethernet start working). Thus, some testing work should be done, e.g. writing a user-space application utilizing timer interrupts.
Secondly, only a Linux kernel with bare necessities was shown to be working. It is still needed to test the solution with something bigger. At the same time, though domU should work without problems, it also should be tested.
Finally, the current solution has several technical flaws. For example, the crossbar support was implemented not as a driver but as a part of Xen platform-support code. At the same time, it should be noticed that the “right way” crossbar support (or any other additional interrupt controller support) would require architectural changes in Xen.
Acknowledgements
I want to thank all my mentors who have done a great job helping me: Julien Grall, Iain Hunter, Hunyue Yau and Stefano Stabellini. Without their help this work wouldn’t be possible. Also, I want to thank all the people who were helping me in mailing lists and on IRC channels. And of course I want to express my gratitude to Google for making this collaboration possible.