Kubelet: Pod The node was low on resource: [DiskPressure]

ismail yenigül
Apr 23, 2019 · 2 min read

Today, some of my Pods was in Evicted status in the dev environment

nginx-7cf887bf7d-kmgl9 0/1 Evicted 0 5h29m
nginx-7cf887bf7d-kszwn 0/1 Evicted 0 5h29m
nginx-7cf887bf7d-m9cvg 0/1 Evicted 0 5h29m
nginx-7cf887bf7d-sps6q 0/1 Evicted 0 5h29m

When I describe the pod, I saw the Pod The node was low on resource: [DiskPressure] message.

$ kubectl describe pod nginx-7cf887bf7d-ch7mn
Name: nginx-7cf887bf7d-ch7mn
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: ip-100–64–5-…
Start Time: Tue, 23 Apr 2019 13:14:16 +0300
Labels: app=nginx
pod-template-hash=7cf887bf7d
Annotations: <none>
Status: Failed
Reason: Evicted
Message: Pod The node was low on resource: [DiskPressure].

It means available disk space on one of the worker node was less than 10%

Here is the default hard limits in k8s codes.

// DefaultEvictionHard includes default options for hard eviction.                       var DefaultEvictionHard = map[string]string{                        "memory.available":  "100Mi",                        "nodefs.available":  "10%",                        "nodefs.inodesFree": "5%",                        "imagefs.available": "15%",

To fix this issue, I created another worker node with larger disk capacity. You can set --eviction-hard kubelet parameter to change values as described https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/

But I think it is better to keep using the default values and monitor the nodes status.

Ismail YENIGUL

Devops Engineer at Feedstock Inc.

Follow us on Twitter 🐦 and Facebook 👥 and join our Facebook Group 💬.

To join our community Slack 🗣️ and read our weekly Faun topics 🗞️, click here⬇

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇

Faun

The Must-Read Publication for Aspiring Developers & DevOps Enthusiasts. Medium’s largest DevOps publication.

ismail yenigül

Written by

Linux and Cloud

Faun

Faun

The Must-Read Publication for Aspiring Developers & DevOps Enthusiasts. Medium’s largest DevOps publication.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade