Open in app

Sign in

Write

Sign in

Build a managed Kubernetes cluster from scratch — part 1

Tony Norlin
12 min readFeb 6, 2022

--

update 2022–05–14: I noted, when writing part 2, that I missed to generate the kube-apiserver certificate and that the kubelet certificates indeed also need to serve as server certificates as well. That and minor typos have been updated now.

The adventures in the Home Lab environment proceeds (when time allows), and with concepts taken from the the earlier stories I have made some improvements in the cluster earlier described. The cluster has went into a new phase and thanks to the ongoing beta with Cilium, functionality has improved!
This is some sort of a “build a managed Kubernetes service from scratch”, but that really depends on the definitions, and in this case the control plane is managed outside of Kubernetes (but none else than will manage this control plane for you).

The current cluster state, with some of the core components. Between the blue zones are firewall rules.

The cluster in its current state has some features which I intend to describe further later on, broken up into a couple of parts. It will take some time to write down all the steps, so bear with me on that one.

As earlier, the cluster features a service mesh provided by Cilium and I may have forgotten some components, but at the current stage the cluster features the following:

  • envoy ingress controller, Cilium built-in — currently in beta
  • BPF-based worker network, free from kube-proxy(!), thanks to Cilium
  • external etcd
  • Network Load Balancer provided by metalLB based Cilium built-in, which features a dedicated ASN with a dedicated IPAM Pool talking BGP from the worker nodes to my BGP peer.
  • external-dns which, after annotation, creates a record in the internal DNS so that there is a FQDN for each service and ingress.
  • Longhorn provides persistent volumes and claims from worker node local storage
  • Grafana provides dashboards
  • Prometheus server, together with Prometheus Alert Manager monitors and alerts to a dedicated Slack webhook
  • The workload container images will be provided by a local Harbor instance
  • Cert-manager talks RFC-2136 to an external DNS master and creates a valid Let’s Encrypt certificate.
  • At the moment logs are aggregated at Loki through Fluent Bit, but after some very helpful hints I am instead primary looking into a new solution with OpenTelemetry
  • Jaeger and OpenTelemetry (otel-collector) has been installed in order to provide the visibility instead of Hubble

Background

illumos

— It began with an idea I had that I wanted to find out if it actually would be possible to run Kubernetes in illumos (for those of you uninitiated — illumos is the heritage of OpenSolaris, which in turn was the opensource port of Sun Solaris) and browsing around Internet a bit I concluded that there was not many posts available on the matter and that there were little hope on make that happen.

At an early stage I wanted to stretch things a bit and try to split the Control Plane apart from the Data Plane and made an early design decision that I really wanted to try illumos as the Control Plane, for reasons being that it is the platform running my infrastructure at home.

So I began looking at the core components that makes up the cluster and looked at etcd; it turned out that someone ported an earlier version to illumos, so I took parts and managed to compile etcd 3.5.0 which then happened to run stable in my cluster for a couple of weeks. It even made my cluster more responsive (due to beeing able to run directly ‘on the metal’ as opposed to my nodes that run virtually).

I didn’t stop there and managed to port several of the kubernetes components to illumos, but I was settled back by trying to figure out how I should manage to implement vlxlan (I saw an implementation available for illumos, but I lacked a method to calculate the VNI in order to establish an VTEP).

Enter cilium — an BPF based network solution

I found out that with Cilium (https://cilium.io/) I can do without the kube-proxy and in turn it enabled other possibilities to separate the network. Also, with the current beta it showed alot of potential to me as much of the functionality is integrated into the cluster (read workers) so that the kube-apiserver remains more or less a pure API. While I would prefer that the kube-apiserver had that sole purpose, I can understand the design decisions that led to webhooks and at the same time I find it a bit annoying that the API should have outgoing communication (such as in the case of Dynamic Admission Control).

Platform

The choice of platform is of course totally optional. I will base the control plane on Solaris — as I’ve managed to port Kubernetes to solaris I’ll better use it so that there is at least one (official) user in the world. By chance, are there any more solaris/illumos kubernauts out there or am I really alone?

You can go ahead with building the platform with your favorite operating system. Plan 9 anyone?

External certificate authority (CA)

As the kubernetes project uses cfssl as its tooling for PKI, I found it fit to use it as the way to generate all the certificates. Luckily detiber/Jason DeTiberus made some excellent instructions on how to implement single node external CA a while ago, that holds still today and we just need some slight adaptions in order to fit our purpose.

Onboard a secure machine (to remain trustworthy, the root CA should be kept on a safe location), install the cfssl— either through your favorite installer or through the instructions at https://github.com/cloudflare/cfssl.

Create the root Certificate Authority

First, create the directory which will hold our root CA

mkdir root-ca
cd root-ca
cat << EOF > root-ca-config.json
{
    "signing": {
        "profiles": {
            "intermediate": {
                "usages": [
                    "signature",
                    "digital-signature",
                    "cert sign",
                    "crl sign"
                ],
                "expiry": "26280h",
                "ca_constraint": {
                    "is_ca": true,
                    "max_path_len": 0,
                    "max_path_len_zero": true
                }
            }
        }
    }
}
EOF

Then, create a Certificate Signing Request (that will hold for 10 years), and sign the request.

cat << EOF > root-ca-csr.json
{
    "CN": "my-root-ca",
    "key": {
        "algo": "rsa",
        "size": 4096
    },
    "ca": {
        "expiry": "87600h"
    }
}
EOF
cfssl genkey -initca root-ca-csr.json | cfssljson -bare ca

The following files should have now have been created

ca-key.pem
ca.csr
ca.pem
root-ca-config.json
root-ca-csr.json

As pointed out by detiber, the root CA should be protected and in our case the files should not be lifted out of our secure (encrypted) location.

Kubernetes intermediate Certificate Authority

Next, create the intermediate CA

cd ..
mkdir kubernetes-ca
cd kubernetes-ca
cat << EOF > kubernetes-ca-csr.json
{
    "CN": "kubernetes-ca",
    "key": {
        "algo": "rsa",
        "size": 4096
    },
    "ca": {
        "expiry": "26280h"
    }
}
EOF

Sign and generate the intermediate CA

cfssl genkey -initca kubernetes-ca-csr.json | cfssljson -bare kubernetes-ca
cfssl sign -ca ../root-ca/ca.pem -ca-key ../root-ca/ca-key.pem -config ../root-ca/root-ca-config.json -profile intermediate kubernetes-ca.csr | cfssljson -bare kubernetes-ca
cat << EOF > kubernetes-ca-config.json
{
    "signing": {
        "default": {
            "expiry": "168h"
        },
        "profiles": {
            "www": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "kubelet": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            }
        }
    }
}
EOF
cd ..

The above command will generate kubernetes-ca-key.pem,kubernets-ca.csr and kubernetes-ca.pem, also it will configure the kubernetes intermediate CA to allow the kubelet to act as both client and server.

In it current state, the cluster will not utilize the Front Proxy intermediate CA, but it doesn’t hurt to have it generated:

Front proxy intermediate Certificate Authority

mkdir kubernetes-front-proxy-ca
cd kubernetes-front-proxy-ca
cat << EOF > kubernetes-front-proxy-ca-csr.json
{
    "CN": "kubernetes-front-proxy-ca",
    "key": {
        "algo": "rsa",
        "size": 4096
    },
    "ca": {
        "expiry": "26280h"
    }
}
EOF
cfssl genkey -initca kubernetes-front-proxy-ca-csr.json | cfssljson -bare kubernetes-front-proxy-ca
cfssl sign -ca ../root-ca/ca.pem -ca-key ../root-ca/ca-key.pem -config ../root-ca/root-ca-config.json -profile intermediate kubernetes-front-proxy-ca.csr | cfssljson -bare kubernetes-front-proxy-ca
cfssl print-defaults config > kubernetes-front-proxy-ca-config.json
cd ..

The following files should have been generated in the kubernetes-front-proxy-cadirectory

kubernetes-front-proxy-ca-config.json
kubernetes-front-proxy-ca-csr.json
kubernetes-front-proxy-ca-key.pem
kubernetes-front-proxy-ca.csr
kubernetes-front-proxy-ca.pem

The etcd intermediate Certificate Authority

mkdir etcd-ca
cd etcd-ca
cat << EOF > etcd-ca-config.json
{
    "signing": {
        "profiles": {
            "server": {
                "expiry": "8700h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            },
            "client": {
                "expiry": "8700h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "8700h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}
EOF
cat << EOF > etcd-ca-csr.json
{
    "CN": "etcd-ca",
    "key": {
        "algo": "rsa",
        "size": 4096
    },
    "ca": {
        "expiry": "26280h"
    }
}
EOF

Sign the certificate signing request and etcd-ca-key.pem and etcd-ca.pem

cfssl genkey -initca etcd-ca-csr.json | cfssljson -bare etcd-ca
cfssl sign -ca ../root-ca/ca.pem -ca-key ../root-ca/ca-key.pem -config ../root-ca/root-ca-config.json -profile intermediate etcd-ca.csr | cfssljson -bare etcd-ca
cd ..

Now, with the intermediate CA certificates generated, it is time to generate the client, peer and server certificates for each component.

Generate the etcd certificates

Generate certificates for the etcd nodes, include each etcd node hostname and corresponding IP. The etcd doesn’t have to be in the same network zone as the other control plane components, but I have included it in the same as I don’t want to introduce more latency from a firewall than necessary. If the control plane will be hosted on illumos, and there is only one physical host to take care of the control plane, there would be a valid point to create an etherstub where the components share an internal private network.

Either split the certs up, one CSR for each server (to prefer and the way we will do it), or as a lazy option — treat them as one united group and generate the certificates to be valid on all (I’ve chosen three to form the etcd cluster)

for instance in {1..3}; do 
cat << EOF > etcd${instance}-server-csr.json
{
  "CN": "etcd${instance}",
  "hosts": [
    "etcd1",
    "etcd2",
    "etcd3",
    "10.100.0.11",
    "10.100.0.12",
    "10.100.0.13",
    "localhost",
    "127.0.0.1"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  }
}
EOF
cfssl gencert -ca=etcd-ca/etcd-ca.pem -ca-key=etcd-ca/etcd-ca-key.pem --config=etcd-ca/etcd-ca-config.json -profile=server etcd${instance}-server-csr.json | cfssljson -bare etcd${instance}
done

This will generate etcd-server.pem and etcd-server-key.pem

Generate the peer keypairs, the same principles apply here and etcd-peer.pem and etcd-peer-key.pemwill be generated

for instance in {1..3}; do 
cat << EOF > etcd${instance}-peer-csr.json
{
  "CN": "etcd${instance}",
  "hosts": [
    "etcd1",
    "etcd2",
    "etcd3",
    "10.100.0.11",
    "10.100.0.12",
    "10.100.0.13",
    "localhost",
    "127.0.0.1"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  }
}
EOFcfssl gencert -ca=etcd-ca/etcd-ca.pem -ca-key=etcd-ca/etcd-ca-key.pem --config=etcd-ca/etcd-ca-config.json -profile=peer etcd${instance}-peer-csr.json | cfssljson -bare etcd${instance}-peer
done

Generate the etcd healthcheck client keypair ( etcd-healtcheck-client.pem and etcd-healthcheck-client-key.pem )

cat << EOF > etcd-healthcheck-client-csr.json
{
  "CN": "kube-etcd-healthcheck-client",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
      {
          "O": "system:masters"
      }
  ]
}
EOF
cfssl gencert -ca=etcd-ca/etcd-ca.pem -ca-key=etcd-ca/etcd-ca-key.pem --config=etcd-ca/etcd-ca-config.json -profile=client etcd-healthcheck-client-csr.json | cfssljson -bare etcd-healthcheck-client

Generate the API Server certificates

Begin to generate the apiserver-kubelet-client.pem and apiserver-kubelet-client-key.pem

cat << EOF > apiserver-kubelet-client-csr.json
{
  "CN": "kube-apiserver-kubelet-client",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "system:masters"
    }
  ]
}
EOF
cfssl gencert -ca=kubernetes-ca/kubernetes-ca.pem -ca-key=kubernetes-ca/kubernetes-ca-key.pem --config=kubernetes-ca/kubernetes-ca-config.json -profile=client apiserver-kubelet-client-csr.json | cfssljson -bare apiserver-kubelet-client

Create the SA keypair

openssl genrsa -out sa.key 2048
openssl rsa -in sa.key -pubout -out sa.pub

As mentioned earlier, the front proxy certificates seem to be of no current use in the cluster, but let’s create them anyway just in case they’ll come handy.

cat << EOF > front-proxy-client-csr.json
{
  "CN": "front-proxy-client",
  "key": {
    "algo": "rsa",
    "size": 2048
  }
}
EOF
cfssl gencert -ca=kubernetes-front-proxy-ca/kubernetes-front-proxy-ca.pem -ca-key=kubernetes-front-proxy-ca/kubernetes-front-proxy-ca-key.pem --config=kubernetes-front-proxy-ca/kubernetes-front-proxy-ca-config.json -profile=client front-proxy-client-csr.json | cfssljson -bare front-proxy-client

Create the kube-apiserver etcd client certificates ( apiserver-etcd-client.pem and apiserver-etcd-client-key.pem ).

cat << EOF > apiserver-etcd-client-csr.json
{
  "CN": "kube-apiserver-etcd-client",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
      {
          "O": "system:masters"
      }
  ]
}
EOF
cfssl gencert -ca=etcd-ca/etcd-ca.pem -ca-key=etcd-ca/etcd-ca-key.pem --config=etcd-ca/etcd-ca-config.json -profile=client apiserver-etcd-client-csr.json | cfssljson -bare apiserver-etcd-client

Create the kube-apiserver server certificate ( apiserver.pem and apiserver-key.pem ).

cat << EOF > apiserver-csr.json
{
  "CN": "kube-apiserver",
  "hosts": [
    "apiserver",
    "10.100.0.1",
    "10.96.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  }
}
EOFcfssl gencert -ca=kubernetes-ca/kubernetes-ca.pem -ca-key=kubernetes-ca/kubernetes-ca-key.pem --config=kubernetes-ca/kubernetes-ca-config.json -profile=www apiserver-csr.json | cfssljson -bare apiserver

Create the admin (cluster-admin) kubeconfig

cat << EOF > admin-csr.json
{
  "CN": "kubernetes-admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "system:masters"
    }
  ]
}
EOF
cfssl gencert -ca=kubernetes-ca/kubernetes-ca.pem -ca-key=kubernetes-ca/kubernetes-ca-key.pem --config=kubernetes-ca/kubernetes-ca-config.json -profile=client admin-csr.json | cfssljson -bare admin
KUBECONFIG=admin.conf kubectl config set-cluster default-cluster --server=https://10.100.0.1:6443 --certificate-authority kubernetes-ca/kubernetes-ca.pem --embed-certs
KUBECONFIG=admin.conf kubectl config set-credentials default-admin --client-key admin-key.pem --client-certificate admin.pem --embed-certs
KUBECONFIG=admin.conf kubectl config set-context default-system --cluster default-cluster --user default-admin
KUBECONFIG=admin.conf kubectl config use-context default-system

Create the worker node certificates

For each worker node, create a certificate signing request as below:

for instance in {1..3}; do 
cat << EOF > worker${instance}-csr.json
{ 
  "CN": "system:node:worker${instance}",
  "key": {
    "algo": "rsa",
    "size": 2048
  },  
  "hosts": [
    "worker${instance}",
    "10.200.0.${instance}"
  ],
  "names": [
    { 
      "O": "system:nodes"
    }
  ]
}
EOF
cfssl gencert -ca=kubernetes-ca/kubernetes-ca.pem -ca-key=kubernetes-ca/kubernetes-ca-key.pem --config=kubernetes-ca/kubernetes-ca-config.json -profile=client -profile=kubelet worker${instance}-csr.json | cfssljson -bare worker${instance}
KUBECONFIG=worker${instance}.kubeconfig kubectl config set-cluster default-cluster --server=https://10.100.0.1:6443 --certificate-authority kubernetes-ca/kubernetes-ca.pem --embed-certs
KUBECONFIG=worker${instance}.kubeconfig kubectl config set-credentials system:node:ubuntu --client-key worker${instance}-key.pem --client-certificate worker${instance}.pem --embed-certs
KUBECONFIG=worker${instance}.kubeconfig kubectl config set-context default-system --cluster default-cluster --user system:node:ubuntu
KUBECONFIG=worker${instance}.kubeconfig kubectl config use-context default-systemdone

This would have generated the following files:

worker1-key.pem  
worker1.pem  
worker2-key.pem  
worker2.pem  
worker3-key.pem  
worker3.pem

Generate the kube-controller-manager certificates and kubeconfig

The controller manager need client files to connect to the api-server (controller-manager.pem , controller-manager-key.pemand controller-manager.conf )

cat << EOF > controller-manager-csr.json
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  }
}
EOF
cfssl gencert -ca=kubernetes-ca/kubernetes-ca.pem -ca-key=kubernetes-ca/kubernetes-ca-key.pem --config=kubernetes-ca/kubernetes-ca-config.json -profile=client controller-manager-csr.json | cfssljson -bare controller-manager
KUBECONFIG=controller-manager.conf kubectl config set-cluster default-cluster --server=https://10.100.0.1:6443 --certificate-authority kubernetes-ca/kubernetes-ca.pem --embed-certs
KUBECONFIG=controller-manager.conf kubectl config set-credentials default-controller-manager --client-key controller-manager-key.pem --client-certificate controller-manager.pem --embed-certs
KUBECONFIG=controller-manager.conf kubectl config set-context default-system --cluster default-cluster --user default-controller-manager
KUBECONFIG=controller-manager.conf kubectl config use-context default-system

Generate the kube-scheduler certificates and kubeconfig

This will generate the (scheduler.pem , scheduler-key.pemand scheduler.conf )

cat << EOF > scheduler-csr.json
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  }
}
EOF
cfssl gencert -ca=kubernetes-ca/kubernetes-ca.pem -ca-key=kubernetes-ca/kubernetes-ca-key.pem --config=kubernetes-ca/kubernetes-ca-config.json -profile=client scheduler-csr.json | cfssljson -bare scheduler
KUBECONFIG=scheduler.conf kubectl config set-cluster default-cluster --server=https://10.100.0.1:6443 --certificate-authority kubernetes-ca/kubernetes-ca.pem --embed-certs
KUBECONFIG=scheduler.conf kubectl config set-credentials default-scheduler --client-key scheduler-key.pem --client-certificate scheduler.pem --embed-certs
KUBECONFIG=scheduler.conf kubectl config set-context default-system --cluster default-cluster --user default-scheduler
KUBECONFIG=scheduler.conf kubectl config use-context default-system

If I got it right, the following files should have been generated

admin-key.pem
admin.conf
admin.pem
apiserver-etcd-client-key.pem
apiserver-etcd-client.pem
apiserver-key.pem
apiserver-kubelet-client-key.pem
apiserver-kubelet-client.pem
apiserver.pem
controller-manager-key.pem
controller-manager.conf
controller-manager.pem
etcd-healthcheck-client-key.pem
etcd-healthcheck-client.pem
etcd-peer-key.pem
etcd-peer.pem
etcd-server-key.pem
etcd-server.pem
front-proxy-client-key.pem
front-proxy-client.pem
etcd1-peer-key.pem
etcd1-peer.pem
etcd1-server-key.pem
etcd1-server.pem
etcd2-peer-key.pem
etcd2-peer.pem
etcd2-server-key.pem
etcd2-server.pem
etcd3-peer-key.pem
etcd3-peer.pem
etcd3-server-key.pem
etcd3-server.pem
worker1-key.pem
worker1.kubeconfig
worker1.pem
worker2-key.pem
worker2.kubeconfig
worker2.pem
worker3-key.pem
worker3.kubeconfig
worker3.pem
kubelet-key.pem
kubelet.conf
kubelet.pem
sa.key
sa.pub
scheduler-key.pem
scheduler.conf
scheduler.pem

Now that all the certificates are generated, it is time to form the cluster and we will do it in a similar matter as I have described in earlier stories. I won’t describe how to compile the components as I have described it earlier and instead focus more on where to put each certificate and how to stand up the kubernetes cluster.

Kubernetes
Cilium
Illumos
Certificate Authority
Homelab

--

--

Tony Norlin

Written by Tony Norlin

268 Followers

Homelab tinkerer. ❤ OpenSource. illumos, Linux, kubernetes, networking and security. Recently eBPF. Connect with me: www.linkedin.com/in/tonynorlin

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams