Debugging in the dark!
You have a program behaving poorly, nothing in the log files, nothing in stdout, no ports responding — it’s a tough place to be. What if I told you that with a Linux environment provided by Docker you can see exactly what it is doing? But before we get to that answer let’s detour and talk about system calls.
What even is a system call?
Most operating systems are split into two main areas of execution, the kernel and user-space programs.
The kernel is a privileged program with unfettered access to the computer’s hardware and is responsible for providing controlled hardware access to user-space programs. User-space programs are things like your web browser, terminal or that comatose program you wish would wake up, these programs must request the kernel to do anything from memory allocation to file access to network access.
A system call is the API the kernel exposes to user-space programs to make these requests. So whenever any user-space program is doing anything of interest, it will be doing it via system calls. Conveniently, Linux offers many tools to instrument these calls and we can use them to see what our programs are doing.
Spoiler Alert — you can use strace!
Strace is a super awesome program provided by Linux for logging all of the system calls another program makes to the Linux kernel i.e. everything meaningful the program does ever. The most simple usage tracing the echo program is below:
/ # strace echo 1execve(“/bin/echo”, [“echo”, “1”], 0x7ffdcc0ee6d0 /* 6 vars */) = 0arch_prctl(ARCH_SET_FS, 0x7f1cd755ab88) = 0set_tid_address(0x7f1cd755abc0) = 215mprotect(0x7f1cd7557000, 4096, PROT_READ) = 0mprotect(0x558e52df7000, 16384, PROT_READ) = 0getuid() = 0write(1, “1\n”, 21) = 2exit_group(0) = ?+++ exited with 0 +++
Woah! Now this is scary at first glance and probably second glance (and every time I use strace). Let’s break it down.
Each line in the output above represents a system call to the kernel, with a name, arguments and return value (<system call>(<parameters>) = <return value>).
To get more value from this output we need to actually know what the system calls do. Thankfully Linux predates widespread adoption of the internet and all that information is typically bundled with your operating system. You can use man 2 <system call> to read the documentation about any of the system calls.
Get Andrew Kiellor’s stories in your inbox
Join Medium for free to get updates from this writer.
Putting it all together:
- The system starts the
/bin/echocommand and initializes the process - The program calls
writewith file descriptor 1 (stdout) with content “1\n” - The program calls
exit_groupto quit the process
But how do I get strace?
So I know it’s not the Year of Linux on the Desktop, but with Docker (get it here) you do already have a Linux environment available and you only need to run the following in a terminal to start playing with strace:
docker run -it --cap-add sys_ptrace alpine sh -c 'apk add -U strace man man-pages; sh'This will run a new alpine container with the sys_ptrace capability enabled (a requirement of strace), install strace, man pages and spawn a shell.
From here things like strace echo 1 will work and give you details on how simple CLI programs work. Try it out with some other programs to see how they work.
Getting Fancy
Running strace in its own isolated container is great for learning, but it would be great if we could trace the program thats behaving poorly. With Docker’s PID namespace sharing feature we can!
Say our badly behaving program is running in a container named deplorable like so:
docker run -d --name deplorable alpine sh -c 'while true; do sleep 1; done'We can run our strace container sharing the bad container’s PID namespace and trace it.
docker run -it --pid=container:deplorable --cap-add sys_ptrace alpine sh -c 'apk add -U strace; strace -f -p 1'Magic!
Conclusion
Strace is a super useful tool to give you that extra information when a process is behaving poorly with a kicker of working with any programming language or framework.
The output can be daunting, but keeping the man pages handy will give you the information you need and in the end you’ll have some knowledge that will be useful for years to come.
