How to configuring kubelet garbage collection in EKS Cluster

You are correct that the flags ‘image-gc-high-threshold’ and ‘image-gc-low-threshold’ can be used to trigger image garbage collection based on disk usage.

If you want to know about Kubernetes(kubelet) Garbage Collection, please refer to the link below :

Configuring kubelet Garbage Collection

The kubelet flags are defined as :

• image-gc-high-threshold: The percent of disk usage which triggers image garbage collection. Default is 85%.
• image-gc-low-threshold: The percent of disk usage to which image garbage collection attempts to free. Default is 80%.

If you would like to configure these flags for existing nodes of an EKS worker node, you can follow the following steps:

1.Connect to an existing worker node using SSH.

2. Open the kubelet-config.json file in the worker nodes using the following command:$ sudo vi /etc/kubernetes/kubelet/kubelet-config.json

3. Add the flags to the kubelet-config.json file, and then save the file. For example:

----- kubelet-config.json ------  
{   
  "kind": "KubeletConfiguration", 
  "apiVersion": "kubelet.config.k8s.io/v1beta1", 
  ... 
  ... 
  ... 
  "imageGCHighThresholdPercent": 70,         ==> Add the argument under the same alignment as the "kind" 
  "imageGCLowThresholdPercent": 50, 
  "maxPods": ... 
}

4. After saving the file, restart the kubelet service in the worker node by running the following command: $ sudo service kubelet restart

Repeat the above steps on all worker nodes that have been set up. Now, to check if the flags have been set, we would need to check the node configz endpoint to validate the changes. To do that, you can follow the steps below:

1. To get the name of your worker nodes, run the following command:

$ kubectl get nodes

2. To open a connection to the API server, run the following command:

$ kubectl proxy

3. To check the node configz, open a new terminal and run the following command:

$ curl -sSL "http://localhost:8001/api/v1/nodes/<NODE_NAME>/proxy/configz" | python3 -m json.tool

Note: Replace <NODE_NAME> with your node name from the list of nodes retrieved in step 1. From the above command, you would get an output of the kubelet config that would have the following details that we set before.

In addition, if you would like to configure these flags on new instances, you can provide the kubelet arguments —-image-gc-low-threshold and —-image-gc-high-threshold in the ‘UserData’ section of the EC2 instance while launching the instance.

---- UserData script ----

#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh <cluster-name> --kubelet-extra-args '--image-gc-low-threshold=50 --image-gc-high-threshold=70'

Or, if the above does not apply, (when the EKS cluster is configured using eksctl.)

In this case, the kubelet configuration file is different.

Follow the configuration file below.

1. Edit kubelet.yaml file

$ sudo vi /etc/eksctl/kubelet.yaml

2. Add GC settings to the configuration file as shown below.

...
serverTLSBootstrap: true
imageGCHighThresholdPercent: 75 <-- add
imageGCLowThresholdPercent: 70 <-- add

3. Restart kubelet.

$ sudo service kubelet restart

4. Running kubectl proxy command.

(Of course, port 8001 must be open in the inbound group of the worker node’s firewall.)

$ kubectl proxy
Starting to serve on 127.0.0.1:8001

5. Check configz configuration changes through curl -sSL command.

$ curl -sSL "http://localhost:8001/api/v1/nodes/<NODE_NAME>/proxy/configz" | python3 -m json.tool
{
    "kubeletconfig": {
        "syncFrequency": "1m0s",
        "fileCheckFrequency": "20s",
        "httpCheckFrequency": "20s",
        "address": "0.0.0.0",
        "port": 10250,
        "serverTLSBootstrap": true,
        "authentication": {
...
...
	"imageGCHighThresholdPercent": 75, <-- check 
        "imageGCLowThresholdPercent": 70, <-- check
...
... 
(Omitted below)

Reference

Configuring kubelet Garbage Collection

Connect to your Linux instance