How much is too much? The Linux OOMKiller and “used” memory

Bob Cotton
Jan 9, 2019 · 3 min read
Image for post
Image for post

Not long after recently taking a job at Splunk, a co-worker (Hi Tameem!) hit me up over Slack and asked about one of my Kubernetes metrics blog posts.

His question was about which container “memory usage” metric the OOMKiller uses to determine if a container should be killed. The assertion I made in that post was:

You might think that memory utilization is easily tracked withcontainer_memory_usage_bytes, however, this metric also includes cached (think filesystem cache) items that can be evicted under memory pressure.The better metric is container_memory_working_set_bytes as this is what the OOM killer is watching for.

This is one of the more highlighted section of this post, so I decided I needed to see it action. Let’s see which of the metrics the OOMKiller is watching for.

I threw together a toy program that would continuously allocate memory until the OOMKiller got involved and killed the container in the pod.

Running this in minikube with memory requests and limits both set to 128MB we see that both container_memory_usage_bytes and container_memory_working_set_bytes track almost 1:1 with each other. When they both reach the limit set on the container, the OOMKiller kills the container and the process starts over.

Image for post
Image for post
Usage and working set tracking

As container_memory_usage_bytes also tracks the amount of filesystem cache that the process uses, I extended the toy program to also write bytes to a file on the filesystem.

After introducing the filesystem cache to the picture, we start to see container_memory_usage_bytes and container_memory_working_set_bytes start to diverge.

Image for post
Image for post
Usage and working set diverging

Now what’s interesting is that the container is still not allowed to use more than the amount of memory at the container limit, but the OOMKiller does not kill the container until container_memory_working_set_bytes gets to the memory limit.

Another interesting aspect of this behavior is that container_memory_usage_bytes tops out at the memory limit of the container, even though bytes continue to be written to disk.

If we take a look at container_memory_cache, we see that the amount of cache used continues to increase until container_memory_usage_bytes hits the limit, then it starts to decrease. Very interesting.

Image for post
Image for post

We can see from this experiment that container_memory_usage_bytes does account for some filesystem pages that are being cached. We can also see that OOMKiller is tracking container_memory_working_set_bytes. This makes sense as shared filesystem cache pages can be evicted from memory at any time. There’s no point in killing the process just for using disk I/O.

Hope this helps.


Thanks to Stephen Coles for the album image.

Image for post
Image for post

Join our community Slack and read our weekly Faun topics ⬇

Image for post
Image for post

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇

FAUN

The Must-Read Publication for Creative Developers & DevOps Enthusiasts

Bob Cotton

Written by

Technologist, father, fly fisherman, and brewer of tasty malty, often hoppy beverages.

FAUN

FAUN

The Must-Read Publication for Creative Developers & DevOps Enthusiasts. Medium’s largest DevOps publication.

Bob Cotton

Written by

Technologist, father, fly fisherman, and brewer of tasty malty, often hoppy beverages.

FAUN

FAUN

The Must-Read Publication for Creative Developers & DevOps Enthusiasts. Medium’s largest DevOps publication.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store