Openshift 4.x kernel troubleshooting using Kdump — Tech Preview

7 min readJun 13, 2021

Background Story

How did I even get to this topic?

During my work as a Senior Red Hat consultant, I’ve run into a very problematic bug that included kernel bugs with RHCOS.

We’ve experienced freeze in our nodes, without a lot of information in the logs. We noticed multiple soft CPU lockup error message in the dmesg of the relevant nodes, but we could not associate the issue with a specific process that may cause this bug to occur.

As part of our troubleshooting process, we’ve got introduced to a Tech-Preview feature which is called “Kdump”, and in this medium article, I would explain to you how to configure it on your Openshift (OCP) 4.x servers for a more efficient troubleshooting process.

OCP 4.x & RHCOS troubleshooting difficulties

A quick reminder: the Openshift 4.x platform is running on a containerized-dedicated OS which is called RHCOS (Red Hat CoreOS) that Red Hat acquired in 2018. Link to the acquisition announcement.

This OS is immutable by design; Putting it simply - it’s static and it should only changeable via the MachineConfigOperator that Openshift 4.x includes as part of its built-in infrastructure operators.

Changes to the OS may include — installing new rpm packages, editing files, make changes in the file system, etc.

In order to install the relevant tools for troubleshooting (Kdump + Crash Utility tool) and the patched kernel version for testing (rpms) we used the rpm-ostree project tool that comes with RHCOS which is designed to make updates to this unique OS.

Rpm-ostree approach to package installation is unique in the way that it refers to the OS as a layers-oriented environment, and each new package/update is a new layer that is built on top of the OS, without affecting the “core” of the system and can be removed /updated in any given time — in such a way it allows us to “stack” updates and add new capabilities to our immutable OS without changing its integrity. I’ll show exactly how to do it later on in that article.

Important note — This entire document can also be useful for standard RHEL systems. Use yum/dnf instead of rpm-ostree to install the relevant packages.

Introducing Kdump — Intro

To troubleshoot this kind of system Red Hat developed (and still expanding and adding features to it) the Kdump tool. As of today (Jun 2021), it’s still on tech preview for OCP 4.7.x, but during our troubleshooting I was required to install it manually on OCP 4.6 and I think you can find it helpful as well.

Troubleshooting operating system issues using Kdump is not a simple task. This tool writes a vmcore file (binary) that contains a complete dump of the kernel & memory during a crash. The investigation of this file is being done by the Crash Utility, and you need a very deep understanding of the kernel in order to make valuable conclusions out of it.

I’ve got the help of the engineering team that developed the tool in order to forward my investigation but in such deep bugs I doubt that there is any other way; It’s a very powerful tool to have in your arsenal, and you should definitely use it in case of issues that require it.

Installing & Configuring Kdump

Run as root for the entire process

Kdump comes in the exec-tools package which is included in RHCOS from OCP 4.7+ but it can also be imported and installed on earlier versions as I’m about to show you.

Make sure your exec-tools version is corresponding with the kernel version you have otherwise it won’t work

// Extract the kernel version
# uname -r // Add the relevant packages into your repository and generate the XXX.repo file on the RHCOS system// Update the new repositories in the rpm-ostree sources list
# rpm-ostree cleanup -m
# rpm-ostree refresh-md# rpm-ostree install kexec-toolsNote! You should not see the kdump service yet, it'll appear after a reboot, we will do it in a sec// Configure kernel args for the kdump to work with after the
# rpm-ostree kargs --append='crashkernel=256M'// Verify
# cat /proc/cmdline | grep crash# systemctl reboot

This is where it gets tricky. The Kdump requires a writable location in the system to write its configuration files. As explained — the RHCOS is immutable by nature and there aren’t too many writable locations on the system- so we should configure such a path to the Kdump so the service will come up successfully.

# BOOT_LOC=/boot$(cat /proc/cmdline | egrep -o "/ostree/.*/vmlinuz" | sed -e "s|/vmlinuz||g")# echo $BOOT_LOC# sed -i "s|^#KDUMP_BOOTDIR=\"/boot\"|KDUMP_BOOTDIR=\"${BOOT_LOC}\"|" /etc/sysconfig/kdump// Verfiy
# cat /etc/sysconfig/kdump

Another thing we should do is declaring who the Kdump should behave once it detects a crash.

// In /etc/kdump.conf - comment all the lines and leave uncomment only 3 lines:core_collector makedumpfile -l --message-level 1 -d 31path /ostree/deploy/rhcos/var/crashfailure_action shell// Verify
# cat /etc/kdump.conf | grep -v "^#"

Explainer:

Core_collector — how and what kind of message level should be written by the tool.
Path — location to write the collected data. If the dir does not exist, the kdump service will generate it by itself.

It’s useful to check upon enabling the service if the dir has being created — which will indicate the kdump works properly.

Failure_action — By default, when kdump fails to create a vmcore file at the configured target location, the system reboots and the dump is lost in the process. We’re changing this behaivor so it’ll save the file properly.

Enable “softlockup_panic” so the kdump will write the vmcore file before the system restarts in case of a crash, otherwise it’ll write it only after the system booted and it may not include the relevant logs you want;

# echo "1" >> /proc/sys/kernel/softlockup_panic

Another configuration we are required to make is declaring that even tho this system is immutable, the kdump should ignore the “luks” (disk encryption label that RHCOS includes by default even if the disk is not actually encrypted) configuration and write the vmcore file any

// In /etc/sysconfig/kdump add the following two parameters:KDUMP_COMMANDLINE_APPEND="........ hest_disable rhcos.root=crypt_rootfs rd.luks=0"

Finally, we should rebuild the kdump configuration and enable the service.

# kdumpctl rebuild# echo $? => Should be 0 (successful build)# kdumpctl reload# systemctl restart kdump# systemctl status kdump

Investigating the vmcore file

Once the system experience a crash or softlockup, we should see the vmcore file generate alongside a dmesg file — both in the directory we configured in the “/etc/kdump.conf” file (under path parameter).

To open the file you need the crash utility tool, you need the next three repositories to install it:

baseOS (corresponding with the rhel version of your system)
appstream (corresponding with the rhel version of your system)
rhel-8-for-x86_64-baseos-debug-rpms

// Update the new repositories in the rpm-ostree sources list
# rpm-ostree cleanup -m
# rpm-ostree refresh-md # rpm-ostree install crash
# rpm-ostree kernel-debuginfo

To start the crash utility, two necessary parameters need to be passed to the command:

The debug-info (a decompressed vmlinuz image), for example /usr/lib/debug/lib/modules/<version>/vmlinux provided through a specific kernel-debuginfo package.
The actual vmcore file, for example /path/to/vmcore/fileexample (e.g. /ostree/deploy/rhcos/var/crash/<date>/vmcore)
The resulting crash command then looks like this:

# crash /usr/lib/debug/lib/modules/<version>/vmlinux /path/to/vmcore/file

Upon the first time the file is being openedt it will take couple of minutes for crashto open it, but afterward it will be opened much quicker.

Some useful information to gather for support

From my experience there are some key commands you can run via the crash utility that can shed light on the problem and bring very important insights about the nature of the problem to the kernel engineering team;

First, identify the problmatic process (PID) from the dmesg file (located in the same path as the vmcore file) by searching for key words in the file such as “CPU” / “bug” / “error” / “soft” (softlockup) / “Out of Memory” / “OOM”, etc.

Examples:

[251074.559238] CPU: 22 PID: 4505 Comm: node_exporter Kdump: loaded Tainted: G W — — — — — — — 4.18.0–193.51.1.el8_2.x86_64 #1

Once you have a PID — using the crash utility run the following commands and gather the information for the support team:

crash> ps <PID of problematic process (e.g. 4505)>crash> bt -s <PID of problematic process (e.g. 4505)>crash> hexcrash> ps -A crash> bt -s -a

That’s it, I hope you’ll find this guide useful, and that you won’t need to use these techniques and that your troubleshooting will go much smoother than mine. XD