Kernels and Logging in Linux

The booting process in a Linux system is quite a complete journey, it has a bunch of stages that accomplish certain tasks just to tune up your workstation or server in an efficient way.

The whole idea is to write a very compact overview on how kernels are triggered and how logging is performed in Linux systems, specifically on CentOS versions. I’m going to be pretty direct on some terminology because there’s a lot of information available for this, however, I’ll try to cover the much as possible in a concise and clear way.

The booting system

Starting from the firmware code execution, to boot loaders, kernels, ramdisk initialization up to whatever process management system, your server or workstation needs to perform all cycles/stages just to be ready for work in your system.

Just to warm up a little bit, we can set that the firmware executes code in two ways. In old systems the computer executes code in the BIOS, actually, this approach is mostly found in legacy systems. On the other side, newer computers perform code execution in the UEFI firmware.

The common approach here is that both are executed during the power-on-self-test known as POST process

To better understand this, you can refer to this article definition article from Geeksforgeeks.

Once the POST process is performed successfully, the Bootloader it’s started. At this point, the firmware executes the famous boot loader grub2 code on the drive, the Bootloader reads its configuration and boot the Linux kernel.

In cases when the BIOS is present, the configuration file for grub will be located at /boot/grub2/grub.cfg , in other cases of UEFI based systems, you should take a look at the /boot/efi/EFI/red/grub.efi

After it, the next big thing is the Kernel initialization, in this step the ramdisk is loaded into RAM memory, the ramdisk serves as a temporary filesystem that includes significant kernel modules, drivers, and some Kickstarter files, after that, the kernel itself unmount the ramdisk and mounts the actual root filesystem.

You can find more information about ramdisk here.

Once the filesystem is mounted, the initialization stage goes on. Here, you will be ruled by a certain service management system, whether using SysV init, launchd, upStart or Systemd

A simple grub overview

As you may know, CentOS 7 uses grub2 as their boot loader.

You can check the complete configuration of the grub loader in the /boot/grub2/grub.cfg file. Here are a few code lines that are worth mentioning

1. ## BEGIN /etc/grub.d/00_header internally create the GRUB menu header and sets up the screen for it
2. ## BEGIN /etc/grub.d/10_linux contains the actual kernel choices that we see at the boot-up. All entries start with menu entry and include options for the bootloader and the Kernel
3. The linux16 /vmlinuz-3-10.0 … is the actual Linux default system, this includes a route to the root filesystem.
4. The initrd16 /initramsfs-3.10.0 … specifies the ramdisk which matches the kernel. This is the previous step before mounting the root filesystem
5. ### BEGIN /etc/grub.d/30_os-prober this one can be useful if you have different partitions on the drive, the os-probe would find it and create a boot entry for it.
6. ### BEGIN /etc/grub.d/40_custom and this one is used for custom menu entries

All the names can be changed for modifying the sort order of execution. If you perform any changes, you will need to some sort of save the grub configuration just by running the grub2-mkconfig command.

Additionally, we have the default grub settings file located at /etc/default/grub which contains the full dump content used to set up grub properly. A cool example is the GRUB_TIMEOUT property that sets a timeout to the displayed menu before automatically booting the default kernel entry

The default kernel configuration is located at /boot/grub2/grubenv and for this, it is worth mentioning the saved_entry list that is nothing more than the menu items for the grub.cfg file.

In order to change the default grub2 loader, you can use the grub2-set-default command by adding the kernel list number to it.

Something like: grub2-set-default 1

After that, you’ll need to reconfigure the grub config file, using:

1. BIOS approach: grub2-mkconfig -o /boot/grub2/grub.cfg
2. UEFI approach: grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

Kernel overview

Understanding the semantic versioning will be helpful to know what type of kernel updates can be applicable to a certain system. In CentOS systems, you can check a list of installed kernels by running: sudo yum -q install list kernel-*

Here’s a kernel example version output: kernel.x86_64 3.10.0–514.el7

Where:

1. 3: is the major version
2. 10: is the major revision
3. 0: is the kernel patch
4. 514: refers to the Red Hat version
5. el7: refers to Enterprise Linux 7

The kernel files reside on the /boot directory, so, when you list it you’ll find vmlinuz files that represent the Linux kernel itself, the initramfs as the ramdisk directories, and additionally, there are some grub-related directories.

Another important directory is the /proc this one is created in memory as the system startup and gets deleted when the machine is shut down. The /proc directory as a direct interface to the kernel itself. Internally you will see a bunch of numeric folders, which represents each running PID with execution details, and the no-numeric folders contain information about the memory system that are represented in text files.

Additional modules

In addition to the kernel, Linux has modules that have to match the kernel version, these modules add functionality to the kernel, which can include filesystems and drivers. You can find it on the /lib and the /lib64 directories

Here’s how you can filter it: ls /lib/modules/$(uname -r)/kernel Internally there are libraries for architecture, crypto, drivers, filesystems, kernels, network, and more, however, you can check some of them with these util commands.

1. lsmod for showing modules loaded in memory
2. modinfo [module] to show information to a particular module
3. modprobe -v [module] outputs information and its dependencies

And so on.

Linux handles hardware drivers automatically, but sometimes it’s required need to force issues, and this is because devices addressed across the network has no way of knowing what’s in there, this is a common scenario for a storage area, network, and printers, but if the hardware is local, there’s another util tool called depmod So, for loading automatically at boot, you must:

1. mkdir /etc/modules-load.d (should include the module name)
2. Then: sudo vi /etc/modules-load.d/dm_mirror.conf (each file for each module), inside type the name of the module: dm_mirrorand that’s it

For Backlisting (the opposite):

1. sudo vi /etc/modprobe.d/ctxfi.conf and inside you should type the name of the module: blacklist [module] and that’s it
2. When booting, the module will not be loaded.

Services

Services, in a nutshell, are processes started by the OS that is managed by the not-so-recent but famous systemctl The systemctl command lets you manage (start, stop, restart, enable, disable, etc) a service.

In CentOS systems the SystemD manages services and other objects like devices, timers, and targets, where every object is called Units and for each unit, there’s a file for configuration.

Here are some examples:

1. systemctl list-unit-files -at service this basically show all enabled unit files, services configured to start up automatically
2. systemctl list-units -at service shows the enabled running services

The common output information consists of a service name, unit file status, a generated state, a current state, and a description. You can filter it as: systemctl list-units -t service --state running

However, you can output detailed information like this: systemctl cat [service] that will show detailed information about the service, including dependencies, command execution, fails, and more.

For management purposes, we have the following examples:

1. sudo systemctl stop atd where atd is the name of the process
2. sudo systemctl status [process] for knowing the status
3. sudo systemctl start [process] to starting a process

The same pattern applies to anything else, you can also run:

1. sudo systemctl is-active [service] to know if a service is started or not
2. sudo systemctl disable [service] to disabling
3. sudo systemctl enable [service] to enable a service, basically, this makes it permanent

Logging

A log is basically a file that contains a message about the system, there are different logs for different tasks, you can find logs for critical errors, failed errors, cronjobs statutes, and way more.

CentOS systems have 2 logging systems rsyslog and the journald

1. The rsyslog it’s compatible with sysklogd and it’s used for persistent logs. These types of logs are long text files that can be captured remotely via TCP or UDP
2. The journald is non-persistent by default, they don’t survive to reboot. These are in a binary format and are stored on RAM and also have a fixed size. But they can be persisted by doing some strategies

You can interact with the rsyslog service with the help of the systemctl command. All the configurations settings will be at /etc/rsyslog.conf

The rsyslog has some sort of rules where you can set which subsystem can produce the system message and its criteria (priority). Additionally, all the messages will be written in the /var/log/messages

On the other side, the journalctl log service is handled differently. Here are a bunch of command execution examples

1. sudo journalctl (all entries)
2. sudo journalctl -k
3. sudo journalctl /sbin/crond
4. sudo journalctl -u crond

In order to make it permanent, you’ll need to create a particular directory for it

[directory]
sudo mkdir -p /var/log/journal
[service]
sudo systemctl restart systemd-journald
[listing logs]
ls -l /var/log/journal

This is all for now. I’m pretty sure that more concepts are missing at this time, so please stay tuned, this probably will get updated periodically.

All these concepts are a must-know for Linux administrators, so you can see this as a refresher or as a learning material as well.

Happy coding :)