kubelet 1.24+ (Part 2)

4 min readAug 27, 2022

This post is part of run Kubernetes components one by one series.

This is the 2nd post(check the first one here) for running kubelet standalone, but focus on the CNI part. As we know, CNI is a framework for configuring network resources, and CNI plugin is for Kubernetes Cluster networking configuration.

Prior to Kubernetes 1.24, the CNI plugins could also be managed by kubelet using the command line options cni-bin-dir and network-plugin, but these options are removed in Kubernetes 1.24, along with the dockershim removal, so management of the CNI is no longer in scope for kubelet.

Let’s try to verify.

Btw, Kubenetes 1.25 has been released recently, so let’s download it from https://dl.k8s.io/v1.25.0/kubernetes-node-linux-amd64.tar.gz(you’ll see the kubelet binary after uncompressing the tarball).

In the 1st post, we installed cri-dockerd to provide a shim for Docker Engine that lets you control Docker via Kubernetes Container Runtime Interface, and we have also installed some CNI plugins in /opt/cni/bin, so let’s try remove those plugins first, and then start kubelet.

## remove all installed plugins
$ rm -rf /opt/cni/bin
## check cri-docker is running
$ systemctl status cri-docker
● cri-docker.service - CRI Interface for Docker Application Container Engine
   Loaded: loaded (/etc/systemd/system/cri-docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-08-25 03:26:07 GMT; 3h 29min ago
     Docs: https://docs.mirantis.com
 Main PID: 2966440 (cri-dockerd)
    Tasks: 9
   Memory: 26.4M
   CGroup: /system.slice/cri-docker.service
           └─2966440 /usr/local/bin/cri-dockerd --container-runtime-endpoint fd://
## start kubelet, you will see an error indicates the network is not ready as no CNI plugins found at /opt/cni/bin
$ ./kubelet --version
Kubernetes v1.25.0
$ ./kubelet --config=/home/opc/k8s/configs/kubeletConfigFile.yaml --container-runtime=remote --container-runtime-endpoint=unix:///run/cri-dockerd.sock
....
E0825 07:00:45.298621 3038890 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: could not locate kubenet required CNI plugins [bridge host-local loopback] at [\"/opt/cni/bin\" \"/opt/cni/bin\"]"
....

As we can see, without those CNI plugins installed, kubelet cannot start successfully, but questions coming, why kubelet use network plugin kubenet? Why the CNI plugins are bridge, host-local and loopback? Why the kubelet looks for CNI plugins from /opt/cni/bin, we don’t specify those info when starting kubelet, right? The answer is cri-dockerd.

cri-dockerd manages the CNI plugins with options cni-bin-dir and network-plugin options, you see both /opt/cni/bin and kubenet are the default values.

# cri-dockerd --help  | grep -E "cni-bin-dir|network-plugin "
      --cni-bin-dir string                      A comma-separated list of full paths of directories in which to search for CNI plugin binaries. (default "/opt/cni/bin")
      --network-plugin string                   The name of the network plugin to be invoked for various events in kubelet/pod lifecycle. (default "kubenet")

Actually, kubenet network plugin is a very basic and simple network plugin, it needs the standard CNI bridge, loopback and host-local plugins required(see a very good page kubenet for details).

Next step, we will try to update cri-dockerd to use network plugin CNI to see what are the differences.

Let’s specify network-plugin to cni for cri-dockerd startup by updating the systemd service file.

# add --network-plugin cni to the ExecStart section in file /etc/systemd/system/cri-docker.service
ExecStart=/usr/local/bin/cri-dockerd --network-plugin cni --container-runtime-endpoint fd://
# reload and restart cri-dockerd
$ systemctl daemon-reload
$ systemctl restart cri-docker

Ok, cri-dockerd is configured to use CNI, so guess will kubelet startup fine with this change, let’s try

$ ./kubelet --config=/home/opc/k8s/configs/kubeletConfigFile.yaml --container-runtime=remote --container-runtime-endpoint=unix:///run/cri-dockerd.sock
...
E0825 09:12:10.494785 3090246 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"

You see, still network plugin is not ready, but the error is cni config uninitialized this time, let’s continue explore.

To make CNI work, we need a CNI configuration file, this file must match the CNI specification, cri-dockerd has a command line option called cni-conf-dir to specify the configuration file.

Let me put the following one into /etc/cni/net.d that is the default value of cni-conf-dir.

$ cat << EOF > 01-cri-dockerd.json
{
  "cniVersion": "0.4.0",
  "name": "dbnet",
  "type": "bridge",
  "bridge": "cni0",
  "ipam": {
    "type": "host-local",
    "subnet": "10.1.0.0/16",
    "gateway": "10.1.0.1"
  }
}
EOF

OK, seems to be fine, let’s start kubelet again

$ ./kubelet --config=/home/opc/k8s/configs/kubeletConfigFile.yaml --container-runtime=remote --container-runtime-endpoint=unix:///run/cri-dockerd.sock
<no CNI initialized error this time>

OK, let’s try to create pod, just put nginx.yaml(check the content in the 1st POST ) into the static-pod path /home/opc/k8s/kubelet-static-pod

## kubelet logs show the pod is created
I0827 15:22:32.298284 3819735 topology_manager.go:205] "Topology Admit Handler"
I0827 15:22:32.386666 3819735 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-logs\" (UniqueName: \"kubernetes.io/empty-dir/9c93cee38fd11dde61debf3b9705fc69-nginx-logs\") pod \"nginx-instance-20220803-1159\" (UID: \"9c93cee38fd11dde61debf3b9705fc69\") " pod="default/nginx-instance-20220803-1159"
## and docker ps shows containers are created successfully
$ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED              STATUS              PORTS     NAMES
2b939ff5a65c   busybox                "/bin/sh -c 'while t…"   About a minute ago   Up About a minute             k8s_log-truncator_nginx-instance-20220803-1159_default_9c93cee38fd11dde61debf3b9705fc69_1
bd417fcf0a98   nginx                  "/docker-entrypoint.…"   About a minute ago   Up About a minute             k8s_nginx_nginx-instance-20220803-1159_default_9c93cee38fd11dde61debf3b9705fc69_1
4a87f1e86f35   k8s.gcr.io/pause:3.6   "/pause"                 About a minute ago   Up About a minute             k8s_POD_nginx-instance-20220803-1159_default_9c93cee38fd11dde61debf3b9705fc69_167

Let’s remove the nginx.yaml

$ rm -rf kubelet-static-pod/nginx.yaml
## kubelet logs
^@I0827 15:25:09.499271 3819735 scope.go:115] "RemoveContainer" containerID="b5a16289296b57e6c86ae6b365c9ef61dcb4323ca46a7758fe2ae48285971683"
^@
I0827 15:25:39.200599 3819735 scope.go:115] "RemoveContainer" containerID="b240edf2024b0d8bf09e4ca6b4518f2da0d3656abb742385c9c101c610759cda"
## docker ps shows nothing 
$  docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

The kubelet startup and operate successfully with CNI plugins now.

Ok, so what we have learned for now.

dockershim is removed from Kubernetes 1.24 along with the CNI management from kubelet
cri-dockerd is a third-party replacement of the built-in dockershim, it lets your use Docker Engine through OCI
Since kubelet doesn’t manage CNI plugins now, cri-dockerd need to uptake this responsibility.

Thanks for reading, hope it helps.

Check other posts of this series on Kubernetes 1.24+ components one by one series | by Elie | Medium

kubelet 1.24+ (Part 2)

Written by Elie