Development setup for Kubernetes on Windows

This document is targeted at k8s developers, and describes a way of setting up a development environment to make changes to Kubernetes itself, and test your changes on your dev k8s cluster.

It leverages kubeadm on the Linux side, and flannel for the CNI. Of course, there would be many other ways of doing this, so please adjust as you see fit.

You’ll need two boxes, one Linux for the master, and one Windows for the worker. If you use cloud VMs for this, you might want to prefer Azure to AWS, since AWS Windows images have been known to have issues with Microsoft’s HCP API.

Setting up your Linux master node

Initial set up

We’ll use an Ubuntu 18.04 box for compiling k8s and serve as our dev cluster’s master. We’ll need a few things on there; first, golang:

GOLANG_VERSION=1.12.1 # change accordingly!
GOPATH=$(readlink -f ~/go) # change if you prefer it somewhere else
sudo apt-get update && sudo apt-get install -y build-essential \
  && cd wget https://dl.google.com/go/go$GOLANG_VERSION.linux-amd64.tar.gz \
  && sudo tar -C /usr/local -xzf go$GOLANG_VERSION.linux-amd64.tar.gz \
  && sudo bash -c "echo -e '\nexport PATH=\"\$PATH\":/usr/local/go/bin\nexport GOPATH=\"$GOPATH\"\nexport GOBIN=\"$GOPATH/bin\"\n' >> /etc/profile"

Then, docker:

sudo apt-get update \
  && sudo apt-get install \
      apt-transport-https \
      ca-certificates \
      curl \
      gnupg-agent \
      software-properties-common \
  && curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - \
  && sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable" \
  && sudo apt-get update \
  && sudo apt-get install -y docker-ce \
  && sudo service docker start

And we’ll also install k8s standard packages. That might seem pointless, since we’ll override the regular k8s binaries with our own; but the packages also install a number of useful scripts and put them in the right place, which will make our lives easier down the road:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - \
  && sudo bash -c "echo 'deb https://apt.kubernetes.io/ kubernetes-xenial main\n' > /etc/apt/sources.list.d/kubernetes.list" \
  && sudo apt-get update \
  && sudo apt-get install -y kubelet kubeadm kubectl

Flannel also needs bridged IPv4 traffic to iptables chains to be enabled:

sudo sysctl net.bridge.bridge-nf-call-iptables=1

Now we’re finally ready to clone and compile Kubernetes itself — this might take a few minutes:

mkdir -p $GOPATH/src/kubernetes \
  && git clone git@github.com:kubernetes/kubernetes.git \ # optionally, checkout your own dev branch
  && make

Now let’s override the system-wide binaries installed by the Kube packages with our own:

for BIN in kubelet kubeadm kubectl; do
  BIN_PATH="$(which "$BIN")"
  sudo mv -v "$BIN_PATH"{,.bckp} # keep a backup
  sudo ln -vsf "$GOPATH/src/kubernetes/_output/bin/$BIN" "$BIN_PATH"
done

Starting your Kubernetes master

Now our node is fully ready for us to actually start our master using our very own version of k8s.

If you just want it to work, you can just add my helper script start_dev_master.sh to your $PATH, and run it — you only need to provide it with the IP address you want the API to advertise on; in most cases that will be your node’s private IP. You can also use the --help flag to see the other options.

When it’s done, you should have a fully functional k8s cluster ready to go, using your own compiled version of k8s, which you can check by running kubectl get nodes .

If you don’t need to know how that script works, you can now skip to the next section.

If you’re feeling more curious, this script does 3 things:

• it locally patches the Docker images kubeadm uses to start your master with your compiled binaries
• it then calls kubeadm init with the right arguments
• once the master is started, it configures it to accept tasks, grants access to future joining nodes, and installs flannels

Setting up your Windows worker node

Now that we have a k8s master running our own version of k8s, let’s set up our Windows worker.

We’ll put everything in C:\k , which is also what Microsoft’s official doc and helper scripts assume; so it’s strongly recommended to just go with that.

Installing docker

In a powershell terminal:

Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
Install-Package -Name Docker -ProviderName DockerMsftProvider
Restart-Computer -Force

And then restart.

Compiling the client binaries for Windows

The only officially supported way to build k8s’ client binaries, at the time of writing, is cross-compiling: on your Linux master node, cd to your Kubernetes repo (the same one we used in the previous section), and run:

for BIN in kubectl kubelet kube-proxy kubeadm; do
  KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/$BIN
done

Then copy the resulting kube*.exe binaries from _output/local/bin/windows/amd64/ over to your Windows node’s C:\k directory. There are plenty of ways of doing this, the easiest two probably being setting up a Samba share on your Linux master, or else enabling SSH on your Windows node, and then using scp from the master.

You’ll also need to copy the Kube config generated by kubeadm at the previous step at ~/.kube/configon your Linux master to your Windows worker — still to theC:\k directory.

Joining the cluster

We’ll use Microsoft’s helper script for this. In a powershell console, define the $ipAddress variable to point to the local network interface you want to use (most likely your node’s private IP), and then run:

cd C:\k
wget https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/start.ps1 -o c:\k\start.ps1
$dnsServiceIp = (.\kubectl get svc/kube-dns -n kube-system | Select-String -Pattern '\s+(?:[0-9]{1,3}\.){3}[0-9]{1,3}\s+').Matches.Value
if (!$dnsServiceIp) {
  echo "Unable to retrieve the DNS service IP, aborting"
  exit 1
}
chcp 65001
.\start.ps1 -ManagementIP $ipAddress -ClusterCIDR '10.244.0.0/16' -ServiceCIDR '10.96.0.0/12' -KubeDnsServiceIP $dnsServiceIp

That’s assuming you kept the default settings when starting your master — if you changed the cluster or service CIDR, update the last line above accordingly.

When that’s done running, you should have a fully functional k8s cluster, with a Linux master and a Windows worker, both running your compiled version of k8s.

Dev iterations

Windows side-only changes

If you wish to test a new version of the Windows client binaries, re-compile them on your master, copy them to your worker, then kill the ones that are already running:

taskkill /im kubelet.exe /f ; taskkill /im kube-proxy.exe /f ; taskkill /im flanneld.exe /f

And re-run start.ps1 again. It’s always a good idea to check the output of kubectl get nodes after this operation, especially that the kubelet’s version on the Windows worker is what you expect.

Master changes

For any changes involving the master (eg the API), you’ll need to re-compile your changes (make), then stop the already running instance with this script. Finally re-run start_dev_master.sh . Don’t forget to copy the new Kube config to your Worker node, and re-join there too.