Operating systems development for Dummies
--
If you’ve ever used a computer, you may have found yourself wondering how operating systems function on a low level, or even how you you would go about developing one yourself. To say that kernel development is difficult is a severe understatement, it really is “the great pinnacle of programming”. In this guide, we will introduce the basic tools needed and implement a simple operating system in C and x86 Assembly.
The system we will be developing is known as ‘Basilica OS’, named in tribute to the now late developer of TempleOS, Terry Davis. (If you don’t know about the story of Terry Davis, read about him here).
The system we’ll develop will be very simple and will act as an introduction to Operating System development, and so we won’t cover any complicated OS theory related topics (Executable formats, Serial communication etc…). We won’t yet implement keyboard support, rather we will set up a base working system, and begin to implement a standard Library. The assumed operating environment will be Ubuntu 18.04, although using WSL on Windows 10 may work.
Anyway, lets get started.
Setting up a Cross Compiler
The first thing that we will do is set up a Cross-Compiler, which will compile for the target of i686-elf. You don’t need to know much about ELF, but if you wish to learn how ELF files are structured, read this. If you’re developing on a Linux environment, you may already have a compiler capable of compiling for 32-bit ELF, however this will not work, as it will produce executable files targeted for Linux, which will be incompatible.
To set up the cross compiler, begin by installing the dependencies
sudo apt-get install build-essential bison flex libgmp3-dev libmpc-dev libmpfr-dev texinfo libcloog-isl-dev libisl-dev qemu grub-common xorriso nasm grub-pc-binNow, to make installation a bit easier, we’ll define a few values, create a directory in ~/srcto install our cross compiler, and add the newly assembled binaries to the system path so that the compiler can detect binutils once it has been built.
export PREFIX="$HOME/opt/cross"
export TARGET=i686-elf
export PATH="$PREFIX/bin:$PATH"
mkdir ~/srcNow, we download the latest release of binutils into a new ~/src/build-binutils directory. This can be done via the GNU ftp mirror, or via wget , which is what we will be doing. Whichever version you end up getting, ensure that you use the right commands by entering the correct version numbers.
cd ~/src
wget https://ftp.gnu.org/gnu/binutils/binutils-2.31.1.tar.xz
tar -xf binutils-2.31.1.tar.xz
rm binutils-2.31.1.tar.xz
mkdir build-binutils
cd build-binutils/
../binutils-2.31.1/configure --target=$TARGET --prefix="$PREFIX" \
--with-sysroot --disable-nls --disable-werror
make
make installOnce this is done, download GCC from the GNU ftp mirror, or by using wget to download it directly. The GCC build process may take a while, so get comfortable.
cd ~/src
wget https://ftp.gnu.org/gnu/gcc/gcc-8.2.0/gcc-8.2.0.tar.xz
tar -xf gcc-8.2.0.tar.xz
rm gcc-8.2.0.tar.xz
mkdir build-gcc
cd build-gcc
../gcc-8.2.0/configure --target=$TARGET --prefix="$PREFIX" \
--disable-nls --enable-languages=c,c++ --without-headers
make all-gcc
make all-target-libgcc
make install-gcc
make install-target-libgccAfter this has been installed, add the following line to ~/.bashrc
export PATH=$HOME/opt/cross/bin:$PATHNow, we can begin to build our operating system.
Booting and Linking
The first thing we will do is write some Assembly code that will handle how the system will run after booting. Creating A program that loads an operating system after booting up (or a Bootloader) is quite difficult, so we will be using the GRUB Bootloader, which will load the OS and then pass control of the computer to our program.
Start by creating a file boot.asm with the following instructions:
The first 5 lines define a few global variables that contain ‘Magic values’, that are searched for by the Bootloader, so that our kernel is recognized as being multiboot compatible.
Lines 7–11 declares the multiboot header containing the ‘Magic value’ and some flags, as well as a checksum to confirm that it is in fact a multiboot header. section .multiboot forces these values to be in the first 8KiB of the Kernel file. .align 4 aligns the file at 32-bit boundaries.
Lines 13–16 constructs a small stack. align 16 sets the stack as being 16-byte aligned which is the standard for stacks in x86. stack_bottom: creates a symbol for the bottom of the stack, resb 16384 reserves 16KiB of space for the stack and stack_top: creates a symbol for the top of the stack.
In lines 19–24, we define how the program will function once being called. In the linking script we will soon create, we will define _start as the entry point of the system, and so the bootloader will jump to this position after the kernel is loaded. mov esp, stack_top sets up our previously defined stack by moving stack_top into the stack pointer register. After this, we call the external function kernel_main which we will define shortly.
Now we will start to implement the kernel. Create a file kernel.c and write the following:
This doesn’t do anything yet, it will just be used temporarily while we set up linking. Note that in this function, void kernel_main() is the function being declared and called in boot.asm , and so is the entry point of our kernel.
Next, create a file linker.ld and add the following code to it.
ENTRY(_start) designates _start as the entry symbol. Pretty self explanatory. . = 1M; Begins by putting sections at 1MiB. We then put in the .multiboot header early, so that the bootloader recognizes the file format. This is followed by the .text section. We then place in the Read-only data, the read-write data, an area for the stack, and an area for other sections which may be created by the compiler.
Now that all of the files we need have been created, we can compile and build a bootable CDROM Image. Compile and build your system with the following commands.
nasm -felf32 boot.asm -o boot.o
i686-elf-gcc -c kernel.c -o kernel.o -std=gnu99 -ffreestanding \
-O2 -Wall -Wextra
i686-elf-gcc -T linker.ld -o basilica.bin -ffreestanding -O2 \
-nostdlib boot.o kernel.o -lgccAt this point, we can check to see if we have configured everything correctly by running:
grub-file --is-x86-multiboot basilica.bin
echo $?If this returns a 0, everything’s good. If it returns a 1, make sure you followed the steps correctly. We can now build a CDROM ISO image from our binary file by creating a config file called grub.cfg
menuentry "basilica" {
multiboot /boot/basilica.bin
}We then create a folder structure, copy over the needed files and build an ISO image.
mkdir -p isodir/boot/grub
cp basilica.bin isodir/boot/basilica.bin
cp grub.cfg isodir/boot/grub/grub.cfg
grub-mkrescue -o basilica.iso isodirYou’ll probably find that this is a lot of commands to type in to fully link, compile and build our image. Lets now make our lives a little bit easier by creating a makefile. Create a file just called makefile with the following
Now, you can compile and build your whole project to an ISO image with make and remove all .iso .bin and .o files by typing make clean . This is what we will be using from here on to be a lot quicker.
We will now run our program with qemu.
qemu-system-i386 -cdrom basilica.isoWe should be displayed with a Grub multiboot menu, followed by a blank black screen after proceeding.
Making the Kernel do something
Now that we’ve got everything working, lets start to make it do something. The first thing we’ll do is find a way to print to the screen, we can do this by writing information to the video memory for colour displays which is located at 0xb8000 . We will be using 80x25 mode. For every character on screen, text mode memory takes two bytes, one of which is the ASCII code byte and the other byte is the ‘Attribute Byte’ which contains foreground colour and background colour. In the attribute byte, the background colour is located in the first four bits, and the foreground colour in the last four bits.
0000 0000 00000000
BG FG ASCIIKnowing this, we can start to define some constants and make some functions to place characters on the screen.
Now we’re going to create a default colour for our terminal, choosing VGA_COLOUR_WHITE for the foreground and VGA_COLOUR_BLUE for the background. We’ll also add a way to keep track of what line and column we’re on so that we can place multiple characters in a row at a time. After we’ve done this, we’re going to add a terminal_initialize() function to set some default values, and change the background colour to something less depressing. Lets also print a few characters in the centre just to see if it works
Now if we build and run our new system, it should look something like this.
This is great, however, it is very inconvenient to have to write every character one at a time, as well as set the foreground and background colour every time. Lets write a few more functions to automatically keep track of placement and colour, to print strings to the terminal.
However, be careful, as this program will not compile and will give the error undefined reference to 'strlen' . This is because we do not have access to the standard library, or really any non-compiler library for that matter. At this point, you should create a file stdlib.h and include it in your project. After you’ve done that, put the following function into it, then build and run your system.
We should get the following:
This may look like the Windows Blue Screen of Death is screaming at us, but this is good, as it means that string wrapping works. You may notice that there is an odd looking character at the end of our string, this is because our kernel currently has no way of dealing with Newline characters and so they are being printed as just another character.
Newlines can be implemented rather easily by rewriting terminal_putchar()to treat reading '\n' the same as reaching VGA_WIDTH , and then refraining from writing to the display buffer. We can refactor terminal_putchar() as such:
Although, one thing you might notice is that when you reach the end of the screen, it will just wrap back to the front. To fix this, we need to implement terminal scrolling, so we can clear the bottom row, and move every other row up by one. At first you may be tempted just to use memmove , but remember that there is no standard library. A function to scroll the screen up can be implemented by looping through each character and setting it to the character VGA_WIDTH characters in front of it. terminal_putchar() and terminal_scroll_up() can be written as such:
Now, text should scroll upwards when it reaches the end of the screen. However, our screen looks fairly bleak, mostly because our screen is completely static. Lets fix that by adding a delay() function to our standard library. Since we don’t yet have access to any kind of CPU timer, the simplest (but laziest) way to implement this is by going into a for loop for a set amount of iterations. This is relatively easy to write.
The volatile keyword, as well as the use of __asm__("NOP") over ; is done to prevent some compiler optimisation that may prevent the loop from running. We can test both the delay() and terminal_scroll_up() functions by running
for(;;){
delay(100);
terminal_writestring("test ");
}
While we’re at it, lets also make a few more printing functions to make our lives easier. We’ll make terminal_writestring_colour() to print whole strings in specific colours. We’ll also add terminal_writeint() to print integers. These are both fairly easy to implement.
Extending the standard library and interacting with the CPU
We’re making some pretty good progress on our operating system. However, one thing you might notice is that at this point is that the screen will look and behave exactly the same every time we run it, as we currently have no way to change the way the screen behaves based on an external or random factor. Because of this, a useful function to add to our standard library would be rand() to generate random variables. In most C standards, rand() typically uses a simple linear congruential generator which is fairly easy to write. Here is a sample rand() implementation.
This function works, but you may be quick to realise that this only shifts the problem slightly, as we do not have any way to provide semi-random seeds to srand() and so rand() will produce the same numbers in the same order every time the system boots up. To fix this, we need a source of entropy. When you make your system more advanced later on, you may be able to use mouse movements or user interactions as source for your entropy, but for now we will take the easiest route and use the CPU’s Time-Stamp counter to seed our random values.
There is no builtin way to read the Time-Stamp counter in C and so we will need to use Assembly. You could do this using Inline assembly, but an easier way would be to modify the already existing boot.asm to add the functionality.
To do so, we will use the x86 Mnemonic RDTSC , which reads a 64-bit value to EDX:EAX , meaning that the high-order bits are loaded into EDX , and the low-order bits are loaded into EAX . Because the accumulator register EAX is only capable of storing 32 bits, we will define two functions to return the values stored at bothEDX and EAX .
First we declare _timestamp_edx and _timestamp_eax as global so that we can call it from kernel.c . The first function sets EDX:EAX to the value of the time stamp counter, moves the value stored at EDX into EAX and returns execution to kernel.c . The second function returns straight after calling RDTSC , as the value of EAX is already what we want.
Now, we can call each function in our kernel.c program by including the following function declaration:
extern uint32_t _timestamp_edx();
extern uint32_t _timestamp_eax();Now, as a final test, lets try making a simple screen and generating random values seeded with _timestamp_eax() .
Congradulations! You have successfully developed and implemented a basic kernel, bootloader and standard library. That’s all for the time being. With these tools you’ve developed, you may be able to to go on and develop a full fledged system. View the full source code on GitHub
Where to from here?
The next major tool you would want to implement would likely be keyboard support. Unless you feel like getting very acquainted with the 650-page specification, I would avoid USB and instead opt for a PS/2 Keyboard which would be much easier to implement. Read more about it Here.
If you wish to go further and implement a more advanced system, I would advise learning more about the theory. Some useful resources would be the OSDev Wiki, Operating Systems from 0 to 1, and Modern Operating systems.
Stay tuned in for the next part (which may take a while) where I will continue this series and show how to implement support for a PS/2 keyboard, among other things.
(Note: An operating system is loosely defined as a Kernel + Tools + applications. Depending on your definition, the system implemented in this tutorial may be more akin to a Kernel rather than an Operating System. I’m going to consider strlen() as a tool and rand() as an application to get around this.)
