Audit logging with container id tagging

TL;DR

The solution is a fork of go-audit called go-audit-container where we get the pid from the audit event before it is logged, iterate up the process tree and get the ppid until a docker-container-shim process is found or a pid of 1 is found. If docker-container-shim is found, then it is running in a container. Use docker-container-shim’s pid as the container id and log the container id along with rest of the details of this audit event. If pid of 1 is found then the process is not running in a container.

Introduction to audit logging

In Linux, the audit system helps in logging some or all system calls made by applications. These audit logs are then used for audit trails, to analyze behavior of processes running on the machine, detect anomalies, to confirm with security compliance requirements, etc.

The audit system consists of two parts,

1. A kernel process called kauditd that logs details of system calls made at the system call interface, into a netlink socket.
2. A userland daemon called auditd that listens on this netlink socket and logs the events to file or to syslog.

Audit logging have been around for a long time and have become a norm when it comes to collecting logs. However, the stock audit logging mechanisms that come with Linux distros fall short of several current day needs. For example, audit logs are typically in syslog format. If it needs to be sent across to an object database like ElasticSearch, it need to be converted into JSON. That is where several tools like go-audit come in handy. These are userland daemons which, just like auditd, listen on the netlink socket where the kernel sends audit logs and log them as JSON.

Introduction to containers

Containers are user land applications that tie together the different namespacing features of the kernel (uid, pid, filesystem, etc) and provide an isolated environment for applications to run within this namespaced environment. So, all container implementations like Docker, rkt, etc are dependent on the namespacing functionality provided to them by the underlying kernel. If a particular feature can not be namespaced in the kernel, then all containers running across a given host will share them. There are several examples of such shared resource: kernel key ring, syslog, etc. Audit system is one of them.

Audit logging in a containerized world

Audit system is not namespaced in the kernel. As a result, audit logs can not have container details. Audit logs contain details of all system calls made on a host but, there can not be any mention of which container is performing which system call. This is a big drawback for audit logging in a containerized environment.

Also, namespacing of audit component is not in the immediate roadmap of the linux kernel. Please see this video by Richard Guy Briggs, a long time Linux kernel contributor: https://www.youtube.com/watch?v=iYbZGoTNQ4A (Please watch around 31:35)

More resources on people asking for container id tagging in audit logs:

Container Wide user namespaces

Docker container process tree

Here is some background to the solution. The process tree looks as follows for docker containers:

Docker daemon spawns a shim daemon (docker-containerd-daemon) which acts as an adaptor to the runc container and also is a daemon monitoring the actual container. The shim process is the parent process to the actual container processes in the process tree.

So, every container will have one container-shim process running.

Solution

My solution is a fork of go-audit called go-audit-container where I get the pid from the audit event before logging the event, iterate up the process tree getting the ppids until a docker-container-shim process is found or a pid of 1 is found. If docker-container-shim is found, use its pid as the container id and log the container id along with rest of the details. If pid of 1 is found then the process is not running in a container.

However, recursing up the process tree is going to be CPU intensive if we have to do this for every event being logged. Hence, I have also implemented a cache, which is a map of PIDs and container ids. Only when a new process gets created, a new entry of PID vs container id is create by iterating up the process tree. There is also a cleanup thread that will clean up details of already exited threads from the cache, every scheduled interval.

Here is the link to the code on github: https://github.com/ubercoolsec/go-audit-container