kubelet 1.24+ (Part 1)

6 min readAug 5, 2022

This post is part of run Kubernetes components one by one series.

This post describes the steps to run kubelet standalone(no other Kubernetes components, just a binary)using Container Runtimes Docker Engine, this is the first post that focus on to make kubelet standalone startup and create pod successfully, the second post tries to explain the CNI part involved in kubelet.

kubelet is an agent that runs on each node in the cluster. It makes sure that containers are running in a Pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy. The kubelet doesn’t manage containers which were not created by Kubernetes.

Anyway, all information here are based on my personal learning, so if there are any mistakes, feel free to point them out, I am very happy to see that!

Ok, let’s start the journey!

I wanted to build kubelet from source code and run it on my Mac, but unfortunately, kubelet doesn’t support run on Darwin for now, as you can see, it is aborted on Darwin.

foreverXT@love ~/git/github/kubernetes
$ ./hack/local-up-cluster.sh
+++ [0805 14:51:59] Building go targets for darwin/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
+++ [0805 14:52:27] Building go targets for darwin/amd64
    k8s.io/kubernetes/cmd/kubectl (non-static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
....
deployment.apps/coredns created
service/kube-dns created
coredns addon successfully deployed.
kubelet is not currently supported in darwin, kubelet aborted.
kubelet is not currently supported in darwin, kube-proxy aborted.
Create default storage class for
storageclass.storage.k8s.io/standard created
Local Kubernetes cluster is running. Press Ctrl-C to shut it down.

So we need a VM, thanks for the Cloud, we can easily get a free VM on the Cloud Providers, I am using OCI(Oracle Cloud Infrastructure) in this page, as OCI can provisioning always-free VM to use, although the VM spec is very small(just 1 CPU and 1G memory), but it is sufficient for the kubelet, so very cool!

Let’s start.

Basically, we need the following tools installed onto the VM

Docker Engine
cri-dockerd
Network plugin
kubelet

Docker Engine

We need to install Docker Engine by the following command. If you need run docker with other userid, you can just add that user to group docker

(using root)
$ yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
Adding repo from: https://download.docker.com/linux/centos/docker-ce.repo
$ yum install docker-ce 
$ systemctl enable docker
$ systemctl start docker
$ systemctl status docker
 docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-08-05 07:19:20 GMT; 3s ago
....

As you may know, Kubernetes has removed dockershim as of release 1.24, the dockershim is a component that is used for direct integration with Docker Engine, so for using Docker Engine as the container runtime for kubelet 1.24, we need to install cri-dockerd, an adapter provides a shim for Docker Engine that lets you control Docker via Kubernetes Container Runtime Interface.

The steps to install cri-dockerd are mentioned here https://github.com/Mirantis/cri-dockerd

(using root)
$ wget https://storage.googleapis.com/golang/getgo/installer_linux
$ chmod +x installer_linux
$ /installer_linux
$ source ~/.bash_profile
$ yum install git -y
$ git clone https://github.com/Mirantis/cri-dockerd
$ cd cri-dockerd
$ mkdir bin
$ go build -o bin/cri-dockerd
$ mkdir -p /usr/local/bin
$ install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd
$ cp -a packaging/systemd/* /etc/systemd/system
$ sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
$ systemctl daemon-reload
$ systemctl enable cri-docker.service
$ systemctl enable --now cri-docker.socket

In my case, the last two commands failed with error

Failed to enable unit: Unit file cri-docker.service does not exist.
Failed to enable unit: Unit file cri-docker.socket does not exist.

Fixed by command restorecon(seems some label issues in the two files, not sure?)

$ restorecon /etc/systemd/system/cri-docker.service
$ systemctl enable cri-docker.service
Created symlink /etc/systemd/system/multi-user.target.wants/cri-docker.service → /etc/systemd/system/cri-docker.service.
$ 
$ restorecon /etc/systemd/system/cri-docker.socket
$ systemctl enable cri-docker.socket
Created symlink /etc/systemd/system/sockets.target.wants/cri-docker.socket → /etc/systemd/system/cri-docker.socket.
$ systemctl start cri-docker

OK, cri-dockerd is installed successfully, we need to install a Network plugin. A Container Runtime, in the network context, is a daemon on a node configured to provide CRI(Container Runtime Interface) Services for kubelet. In particular, the Container Runtime must be configured to load the CNI(Container Network Interface) plugins to implement the Kubernetes network model.

Install Network plugin can follow the steps mentioned here https://github.com/cri-o/cri-o/blob/main/contrib/cni/README.md

$ git clone https://github.com/containernetworking/plugins
$ cd plugins/
$ git checkout v1.1.1
$ ./build_linux.sh
$ mkdir -p /opt/cni/bin
$ cp bin/* /opt/cni/bin

Very simple, right!

Ok, Container Runtime preparation is done. Now, we will install kubelet, to simplify the installation, I will download the last version of kubelet binary directly from https://dl.k8s.io/v1.24.3/bin/linux/amd64/kubelet

$ wget https://dl.k8s.io/v1.24.3/bin/linux/amd64/kubelet
$ chmod +x ./kubelet

For running kubelet, we need to create a KubeletConfiguration object, for each properties, can check https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/ for details.

$ mkdir configs
$ cd configs
$ cat kubeletConfigFile.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
enableServer: false
staticPodPath: /home/opc/k8s/kubelet-static-pod
readOnlyPort: 10250
failSwapOn: false
podCIDR: 10.241.1.0/24
authentication:
  anonymous:
    enabled: true
  webhook:
    enabled: false
authorization:
  mode: AlwaysAllow

Ok, it is time to run kubelet

./kubelet --config=/home/opc/k8s/configs/kubeletConfigFile.yaml --container-runtime=remote --container-runtime-endpoint=unix:///run/cri-dockerd.sock

You see, the container-runtime-endpoint option is pointing to the cri-dockerd we installed previously.

Now, kubelet is running and watching the directory specified by staticPodPath property, in my case, the directory is /home/opc/k8s/kubelet-static-pod, that means if we put the POD yaml into that directory, kubelet will create the POD for us automatically.

$ ps -aux | grep kubelet
root      717002  1.1  6.9 1714764 48908 pts/2   Sl+  08:37   0:11 ./kubelet --config=/home/opc/k8s/configs/kubeletConfigFile.yaml --container-runtime=remote --container-runtime-endpoint=unix:///run/cri-dockerd.sock
root      724710  0.0  0.1  12136  1188 pts/0    S+   08:54   0:00 grep --color=auto kubelet
[root@instance-20220803-1159 configs]#

Let’s try

$ pwd
/home/opc/k8s/kubelet-static-pod
$
$ ls
nginx.yaml
$
$ cat nginx.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
    volumeMounts:
    - mountPath: /var/log/nginx
      name: nginx-logs
  - name: log-truncator
    image: busybox
    command:
    - /bin/sh
    args: [-c, 'while true; do cat /dev/null > /logdir/access.log; sleep 10; done']
    volumeMounts:
    - mountPath: /logdir
      name: nginx-logs
  volumes:
  - name: nginx-logs
    emptyDir: {}

After we create the ngnix.yaml, then you will see kubelet starts to create it

^@I0805 08:40:32.231689  717002 topology_manager.go:200] "Topology Admit Handler"
I0805 08:40:32.432222  717002 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-logs\" (UniqueName: \"kubernetes.io/empty-dir/83760d52afdfe74389411f38b257d1c5-nginx-logs\") pod \"nginx-instance-20220803-1159\" (UID: \"83760d52afdfe74389411f38b257d1c5\") " pod="default/nginx-instance-20220803-1159"
^@

After some time, you will see the the containers defined in the nginx.yaml is running.

Super Amazing, isn’t it?

## kubelet GET pods api returns the pod created 
$ curl http://localhost:10250/pods/ | jq . | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2640    0  2640    0     0   429k      0 --:--:-- --:--:-- --:--:--  429k
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {},
  "items": [
    {
      "metadata": {
        "name": "nginx-instance-20220803-1159",
        "namespace": "default",
        "uid": "9c93cee38fd11dde61debf3b9705fc69",
## Pod containers were created
$ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS     NAMES
6fcd9ccf10e4   busybox                "/bin/sh -c 'while t…"   22 minutes ago   Up 22 minutes             k8s_log-truncator_nginx-instance-20220803-1159_default_83760d52afdfe74389411f38b257d1c5_0
bac90cef90f4   nginx                  "/docker-entrypoint.…"   22 minutes ago   Up 22 minutes             k8s_nginx_nginx-instance-20220803-1159_default_83760d52afdfe74389411f38b257d1c5_0
0e5c748e5628   k8s.gcr.io/pause:3.6   "/pause"                 23 minutes ago   Up 23 minutes             k8s_POD_nginx-instance-20220803-1159_default_83760d52afdfe74389411f38b257d1c5_0

If we remove the nginx.yaml from /home/opc/k8s/kubelet-static-pod, then the pod will be deleted automatically.

## kubelet log
I0902 06:09:29.587821 2013106 reconciler.go:211] "operationExecutor.UnmountVolume started for volume \"nginx-logs\" (UniqueName: \"kubernetes.io/empty-dir/9c93cee38fd11dde61debf3b9705fc69-nginx-logs\") pod \"9c93cee38fd11dde61debf3b9705fc69\" (UID: \"9c93cee38fd11dde61debf3b9705fc69\") "
I0902 06:09:29.596492 2013106 operation_generator.go:890] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c93cee38fd11dde61debf3b9705fc69-nginx-logs" (OuterVolumeSpecName: "nginx-logs") pod "9c93cee38fd11dde61debf3b9705fc69" (UID: "9c93cee38fd11dde61debf3b9705fc69"). InnerVolumeSpecName "nginx-logs". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
I0902 06:09:29.688903 2013106 reconciler.go:399] "Volume detached for volume \"nginx-logs\" (UniqueName: \"kubernetes.io/empty-dir/9c93cee38fd11dde61debf3b9705fc69-nginx-logs\") on node \"instance-20220803-1159\" DevicePath \"\""
## GET pods api call returns emtpy
$ curl http://localhost:10250/pods/ | jq . | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    64  100    64    0     0  12800      0 --:--:-- --:--:-- --:--:-- 12800
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {},
  "items": null
}
## container were removed.
$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[root@instance-20220803-1159 kubelet-static-pod]#

OK, we have a working kubelet standalone now, I will explain more on the CNI part in the next post.

Thanks for reading.

Check other posts of this series on Kubernetes 1.24+ components one by one series | by Elie | Medium

kubelet 1.24+ (Part 1)

Docker Engine

Written by Elie