Kubernetes the hard (illumos) way

11 min readDec 12, 2021

edit: I revised the steps bit after I went through all the steps a second time. In my first version I wanted to refer as much as possible to the official Kubernetes the Hard Way guide as I appreciate it as a source, but I realize it is confusing to jump around to much between different sources — so instead I have tried to have all steps visible here directly.

This is by no means a recommended way of running kubernetes in production or even as a home lab. This is just my way on learning the different components in a fun way, with the data plane running on metal with illumos. One of the differences from the official kubernetes the hard way provided by Kelsey Hightower is that I chose to not run the data plane clustered. But it would certainly be possible and not particularly complicated, at least not for the kube-apiserver and the etcd — as the etcd loadbalances itself with multiple members and kube-apiserver is easily placed behind a loadbalancer such as haproxy. I chose to separate the data plane components to a separate node for each of etcd, kube-apiserver, kube-controller-manager and kube-scheduler.

Logical diagram of the illumos based Kubernetes control plane

Build the kubernetes binaries on illumos

As a basis for my build environment I use a simple pkgsrc branded zone with the pkgsrc (https://pkgsrc.joyent.com/) that Joyent provides.

I install the necessary binaries for building the kubernetes

pkgin install go117–1.17.1 gcc9–9.3.0 gcc9-libs-9.3.0 \
git-2.33.0 gmake-4.3nb2

I have went through quickly and hacked necessary files to complete a build at version v1.23.0 https://github.com/tnorlin/kubernetes.git (branch v1.23.0-illumos).

In order to build without issues, the GNU binaries have to be used so I set the path

export PATH=/usr/gnu/bin:$PATH

For v1.22.3 I managed to build kubelet and the add-on kube-proxy, but as I have no intention on running any workload in the data plane I skipped it this time. As a matter of fact, I will not use kube-proxy within the cluster.

GOBIN= gmake all WHAT=cmd/kube-apiserver GOFLAGS=-v 
GOBIN= gmake all WHAT=cmd/cloud-controller-manager GOFLAGS=-v
GOBIN= gmake all WHAT=cmd/kube-scheduler GOFLAGS=-v 
GOBIN= gmake all WHAT=cmd/kube-controller-manager GOFLAGS=-v 
GOBIN= gmake all WHAT=cmd/kubectl GOFLAGS=-v 
GOBIN= gmake all WHAT=cmd/kubectl-convert GOFLAGS=-v 
GOBIN= gmake all WHAT=cmd/kubemark GOFLAGS=-v 
GOBIN= gmake all WHAT=cmd/kube-controller-manager GOFLAGS=-v 
GOBIN= gmake all WHAT=cmd/kubeadm GOFLAGS=-v

Now that all the binaries are built, we are ready to follow the excellent guide by Kelsey Hightower — https://github.com/kelseyhightower/kubernetes-the-hard-way.

As we are not running in the Google Cloud Platform, we can skip the first chapter and head into Chapter 02 — “Installing the Client Tools”:

Install the cfssl and cfssljson on a shell of your choice.

Build the etcd binaries on illumos

For etcd v3.5.0, I’ve already modified the necessary files in order to build

Clone https://github.com/tnorlin/etcd.git and run

./build.sh

Bootstrap the nodes

This is an adaption of chapter 03 — Provisioning Compute Resources.

Prerequisites here is that you should pick a CIDR and VLAN of your choice, then create the zones for kube-apiserver, kube-scheduler, kube-controller-manager and etcd.

I run with the zadm command and have json-files prepared:

cat <<EOF | sudo tee /var/tmp/kube-apiserver.json
{
 "autoboot" : "true",
 "bootargs" : "",
 "brand" : "pkgsrc",
 "cpu-shares" : "1",
 "dns-domain" : "cloud.mylocal",
 "fs-allowed" : "",
 "hostid" : "",
 "ip-type" : "exclusive",
 "limitpriv" : "default",
 "net" : [
 {
 "allowed-address" : "192.168.200.1/26",
 "defrouter" : "192.168.200.62",
 "global-nic" : "aggr0",
 "physical" : "kubeapisrv0",
 "vlan-id" : "1234"
 }
 ],
 "pool" : "",
 "resolvers" : [
 "8.8.4.4",
 "8.8.8.8"
 ],
 "scheduling-class" : "",
 "zonename" : "kube-apiserver",
 "zonepath" : "/zones/kube-apiserver"
}
EOF

A example for zonecfg export would be similar to:

create -b
set zonepath=/zones/kube-apiserver
set brand=pkgsrc
set autoboot=false
set limitpriv=default
set ip-type=exclusive
add net
set allowed-address="192.168.200.1/26"
set physical="kubeapisrv0"
set vlan-id="1234"
set global-nic="aggr0"
set defrouter="192.168.200.62"
end
add rctl
set name="zone.cpu-shares"
add value (priv=privileged,limit=1,action=none)
end
add attr
set name="dns-domain"
set type="string"
set value="cloud.mylocal
end
add attr
set name="resolvers"
set type="string"
set value="8.8.4.4,8.8.8.8"
end

The etcd:

cat <<EOF | sudo tee /var/tmp/etcd.json
{
 "autoboot" : "true",
 "bootargs" : "",
 "brand" : "pkgsrc",
 "cpu-shares" : "1",
 "dns-domain" : "cloud.mylocal",
 "fs-allowed" : "",
 "hostid" : "",
 "ip-type" : "exclusive",
 "limitpriv" : "default",
 "net" : [
 {
 "allowed-address" : "192.168.200.2/26",
 "defrouter" : "192.168.200.62",
 "global-nic" : "aggr0",
 "physical" : "etcd0",
 "vlan-id" : "1234"
 }
 ],
 "pool" : "",
 "resolvers" : [
 "8.8.4.4",
 "8.8.8.8"
 ],
 "scheduling-class" : "",
 "zonename" : "etcd",
 "zonepath" : "/zones/etcd"
}
EOF

The kube-controller-manager:

cat <<EOF | sudo tee /var/tmp/kube-ctrlmgr.json
{
 "autoboot" : "true",
 "bootargs" : "",
 "brand" : "pkgsrc",
 "cpu-shares" : "1",
 "dns-domain" : "cloud.mylocal",
 "fs-allowed" : "",
 "hostid" : "",
 "ip-type" : "exclusive",
 "limitpriv" : "default",
 "net" : [
 {
 "allowed-address" : "192.168.200.3/26",
 "defrouter" : "192.168.200.62",
 "global-nic" : "aggr0",
 "physical" : "kubectrlmgr0",
 "vlan-id" : "1234"
 }
 ],
 "pool" : "",
 "resolvers" : [
 "8.8.4.4",
 "8.8.8.8"
 ],
 "scheduling-class" : "",
 "zonename" : "kube-ctrlmgr",
 "zonepath" : "/zones/kube-ctrlmgr"
}
EOF

The kube-scheduler:

cat <<EOF | sudo tee /var/tmp/kube-scheduler.json
{
 "autoboot" : "true",
 "bootargs" : "",
 "brand" : "pkgsrc",
 "cpu-shares" : "1",
 "dns-domain" : "cloud.mylocal",
 "fs-allowed" : "",
 "hostid" : "",
 "ip-type" : "exclusive",
 "limitpriv" : "default",
 "net" : [
 {
 "allowed-address" : "192.168.200.4/26",
 "defrouter" : "192.168.200.62",
 "global-nic" : "aggr0",
 "physical" : "kubesched0",
 "vlan-id" : "1234"
 }
 ],
 "pool" : "",
 "resolvers" : [
 "8.8.4.4",
 "8.8.8.8"
 ],
 "scheduling-class" : "",
 "zonename" : "kube-sched",
 "zonepath" : "/zones/kube-sched"
}
EOF

As long as the network is flat between the control plane and the workers, it shouldn’t really matter where the workers are, but I chose to run them as the bhyve branded guests. Do as many workers as needed:

cat <<EOF | sudo tee /var/tmp/worker0.json
{
 "acpi" : "on",
 "autoboot" : "true",
 "bootargs" : "",
 "bootdisk" : {
 "blocksize" : "8K",
 "path" : "dpool/bhyve/worker0/root",
 "size" : "20G",
 "sparse" : "true"
 },
 "bootorder" : "cd",
 "bootrom" : "BHYVE_CSM",
 "brand" : "bhyve",
 "cloud-init" : "/opt/kubernetes/cloud-init",
 "cpu-shares" : "4",
 "diskif" : "virtio",
 "dns-domain" : "cloud.mylocal",
 "fs-allowed" : "",
 "hostbridge" : "i440fx",
 "hostid" : "",
 "ip-type" : "exclusive",
 "limitpriv" : "default",
 "net" : [
 {
 "allowed-address" : "192.168.200.5/26",
 "defrouter" : "192.168.200.62",
 "global-nic" : "aggr0",
 "physical" : "worker0",
 "vlan-id" : "1234"
 }
 ],
 "netif" : "virtio",
 "pool" : "",
 "ram" : "4G",
 "resolvers" : [
 "8.8.4.4",
 "8.8.8.8"
 ],
 "rng" : "off",
 "scheduling-class" : "",
 "type" : "generic",
 "vcpus" : "4",
 "vnc" : "on",
 "xhci" : "on",
 "zonename" : "worker0",
 "zonepath" : "/zones/worker0"
}
EOF

Bootstrap the Linux zones, as many as you want/need (but three for this guide), and install a Linux of your choice. We will get back to the configuration later.

Provisioning a CA and Generating TLS Certificates

Head on to chapter 04 and follow the KTHW guide, with adaptions as follows:

Certificate Authority

This is untouched from the official guide

{

cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}
EOF

cat > ca-csr.json <<EOF
{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "Kubernetes",
      "OU": "CA",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

}

The Admin Client Certificate

This section is also untouched from the official guide

{
cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:masters",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOFcfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  admin-csr.json | cfssljson -bare admin}

The Kubelet Client Certificates

for instance in {0..2};  docat > worker${instance}-csr.json <<EOF
{
  "CN": "system:node:worker${instance}",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:nodes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOFOFFSET=5 # The first available IP after Data Plane nodesEXTERNAL_IP=192.168.200.$(echo ${OFFSET}+${instance}|bc) 

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=worker${instance},${EXTERNAL_IP} \
  -profile=kubernetes \
  worker${instance}-csr.json | cfssljson -bare worker${instance}done

The Kubernetes API Server Certificate

As the nodes will be split up, we need to set up the SAN to contain all the relevant hostnames.

{

KUBERNETES_PUBLIC_ADDRESS=192.168.200.1

KUBERNETES_HOSTNAMES=kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.svc.cluster.local,tetcd,kube-apiserver,kube-ctrlmgr,kube-sched

cat > kubernetes-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "Kubernetes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=10.32.0.1,192.168.200.1,192.168.200.2,192.168.200.3,192.168.200.4,${KUBERNETES_PUBLIC_ADDRESS},127.0.0.1,${KUBERNETES_HOSTNAMES} \
  -profile=kubernetes \
  kubernetes-csr.json | cfssljson -bare kubernetes

}

The Controller Manager Client Certificate

Generate the kube-controller-manager client certificate and private key:

{

cat > kube-controller-manager-csr.json <<EOF
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:kube-controller-manager",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

}

The Scheduler Client Certificate

Generate the kube-scheduler client certificate and private key:

{cat > kube-scheduler-csr.json <<EOF
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:kube-scheduler",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOFcfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-scheduler-csr.json | cfssljson -bare kube-scheduler}

The Service Account key pair

This section is also the same as in the official guide

{

cat > service-account-csr.json <<EOF
{
  "CN": "service-accounts",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "Kubernetes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  service-account-csr.json | cfssljson -bare service-account

}

Generating Kubernetes Configuration Files for Authentication

Time for chapter05, with some adaptions:

Kubernetes Public IP Address

We defined the public IP in the previous chapter:

KUBERNETES_PUBLIC_ADDRESS=192.168.200.1

The kubelet Kubernetes Configuration File

for instance in {0..2}; do
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=https://${KUBERNETES_PUBLIC_ADDRESS}:6443 \
    --kubeconfig=worker${instance}.kubeconfig

  kubectl config set-credentials system:node:worker${instance} \
    --client-certificate=worker${instance}.pem \
    --client-key=worker${instance}-key.pem \
    --embed-certs=true \
    --kubeconfig=worker${instance}.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:node:worker${instance} \
    --kubeconfig=worker${instance}.kubeconfig

  kubectl config use-context default --kubeconfig=worker${instance}.kubeconfig
done

The kube-controller-manager Kubernetes Configuration File

As we will run the controller manager separately, it needs to have a reachable IP address (from the kube-apiserver) defined:

{
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=https://${KUBERNETES_PUBLIC_ADDRESS}:6443 \
    --kubeconfig=kube-controller-manager.kubeconfig

  kubectl config set-credentials system:kube-controller-manager \
    --client-certificate=kube-controller-manager.pem \
    --client-key=kube-controller-manager-key.pem \
    --embed-certs=true \
    --kubeconfig=kube-controller-manager.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:kube-controller-manager \
    --kubeconfig=kube-controller-manager.kubeconfig

  kubectl config use-context default --kubeconfig=kube-controller-manager.kubeconfig
}

The kube-scheduler Kubernetes Configuration File

The same goes with the kube-scheduler that needs to be reachable (from the kube-apiserver):

{
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=https://${KUBERNETES_PUBLIC_ADDRESS}:6443 \
    --kubeconfig=kube-scheduler.kubeconfig

  kubectl config set-credentials system:kube-scheduler \
    --client-certificate=kube-scheduler.pem \
    --client-key=kube-scheduler-key.pem \
    --embed-certs=true \
    --kubeconfig=kube-scheduler.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:kube-scheduler \
    --kubeconfig=kube-scheduler.kubeconfig

  kubectl config use-context default --kubeconfig=kube-scheduler.kubeconfig
}

The admin Kubernetes Configuration File

As our cluster components is spread out, it seems wise to have this also on an

external reachable address (although, you could of course run it from the illumos kube-apiserver as we ported the kubectl

{
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=https://${KUBERNETES_PUBLIC_ADDRESS}:6443 \
    --kubeconfig=admin.kubeconfig

  kubectl config set-credentials admin \
    --client-certificate=admin.pem \
    --client-key=admin-key.pem \
    --embed-certs=true \
    --kubeconfig=admin.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=admin \
    --kubeconfig=admin.kubeconfig

  kubectl config use-context default --kubeconfig=admin.kubeconfig
}

Generating the Data Encryption Config and Key

this section is also exactly as in the official guide

The Encryption Key

Generate an encryption key:

ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)

The Encryption Config File

Create the encryption-config.yaml encryption config file:

cat > encryption-config.yaml <<EOF
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

Bootstrapping the etcd Cluster

The next chapter that needs adaptions is the chapter 07 as we will have different method to bootstrap the node.

zadm create -b pkgsrc etcd < etcd.json

On the client where certificates was created:

client$ tar -czf - ca.pem kubernetes.pem kubernetes-key.pem | base64

Copy over the etcdctl and etcd binaries from the build zone on the etcd zone at /opt/local/bin and paste in the base64 encoded blob as follows:

mkdir -p /etc/etcd; cd /etc/etcd; base64 -d |gtar -xzf -

Then install the service

INTERNAL_IP=192.168.200.2 # $(ipadm show-addr)

Each etcd member must have a unique name within an etcd cluster. Set the etcd name to match the hostname of the current compute instance:

ETCD_NAME=etcd # $(hostname -s)

Create the SMF method for etcd:

cat <<EOF >/lib/svc/method/etcd
#
# The contents of this file are subject to the terms of the
# Common Development and Distribution License (the "License").
# You may not use this file except in compliance with the License.
#
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets "[]" replaced with your own identifying
# information: Portions Copyright [yyyy] [name of copyright owner]
#
# CDDL HEADER END
#
#
# Copyright 2008 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
#ident    "%Z%%M%    %I%    %E% SMI"

#
# Start/Stop client LDAP service
#

. /lib/svc/share/smf_include.sh

case "\$1" in
'start')
        exec /opt/local/bin/etcd --cert-file=/etc/etcd/kubernetes.pem  --key-file=/etc/etcd/kubernetes-key.pem  --peer-cert-file=/etc/etcd/kubernetes.pem  --peer-key-file=/etc/etcd/kubernetes-key.pem  --trusted-ca-file=/etc/etcd/ca.pem  --peer-trusted-ca-file=/etc/etcd/ca.pem  --peer-client-cert-auth  --client-cert-auth  --initial-advertise-peer-urls https://${INTERNAL_IP}:2380  --listen-peer-urls https://${INTERNAL_IP}:2380  --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379  --advertise-client-urls https://${INTERNAL_IP}:2379  --initial-cluster-token etcd-cluster-0  --initial-cluster ${ETCD_NAME}=https://$INTERNAL_IP:2380  --initial-cluster-state new  --data-dir=/var/lib/etcd --name=etcd > /var/log/etcd.log 2>&1 &
    ;;

'stop')
    exec /usr/bin/pkill etcd 
    ;;

*)
    echo "Usage: \$0 { start | stop }"
    exit 1
    ;;
esac
EOFchmod +x /lib/svc/method/etcd

Create the SMF manifest for etcd

cat <<EOF | sudo tee /lib/svc/manifest/etcd.xml
<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<!-- 
    Manifest automatically generated by smfgen.
 -->
<service_bundle type="manifest" name="application-etcd" >
    <service name="application/etcd" type="service" version="2" >
        <create_default_instance enabled="true" />
        <dependency name="dep0" grouping="require_all" restart_on="error" type="service" >
            <service_fmri value="svc:/milestone/multi-user:default" />
        </dependency>
        <exec_method type="method" name="start" exec="/lib/svc/method/etcd start" timeout_seconds="30" />
        <exec_method type="method" name="stop" exec=":kill" timeout_seconds="30" />
        <template >
            <common_name >
                <loctext xml:lang="C" >ETCD 3.5.0</loctext>
            </common_name>
        </template>
    </service>
</service_bundle>
EOF

Import the SMF Manifest

svccfg import /lib/svc/manifest/etcd.xml

The application would now be started

svcs -a |grep etcd

output

online          6:10:59 svc:/application/etcd:default

I will finish writing a part 2 for the rest of the control plane components that builds up the foundation for our Kubernets cluster. At this stage you can test this zone as an external etcd for an external cluster. I’ve hosted an external illumos etcd with success for about two months in my homelab and it has been very responsive (as it runs on without a hypervisor on top) and stable to me.

Bootstrap of the Control Plane Components

In the next part I describe how the Control Plane is bootstrapped. Click here to continue to the next part.