Linux and Windows in one hybrid Kubernetes cluster

Maxim Vorobjov
Jul 25, 2017 · 3 min read

Azure provides acs automation for either Linux or Windows based kube agents, though not both. We will use opensource acs-engine tool to setup single Kubernetes cluster combining both Windows and Linux agent nodes. Moreover we will generate cluster template, so that deployment can be reproduced with one command.

Setup is using az CLI to create service principal. If you do not have the tool please follow Install Azure CLI 2.0.

Make sure to have latest version of az CLI!

Instructions provided for bash. Also we will use jq to process output JSONs.

Apart of az CLI we will use ssh to setup network tunnel to connect to agent nodes which wouldn’t have public IPs.

1. Create service principal

Kubernetes cluster requires Azure service principal to manage connected Azure infrastructure, like load balancer, storage. Creation is simple via az CLI:

1$ export SUBSCRIPTION_ID=`az account show | jq ".id" -r`
1$ export res1=`az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/${SUBSCRIPTION_ID}"`
1$ export spName=$(echo $res1 | jq ".appId" -r)
1$ export spPassword=$(echo $res1 | jq ".password" -r)

2. Setup ecs-engine on local machine

Below setup considering that you have docker engine running locally, see ecs-engine in Docker for details. Otherwise refer to the ecs-engine guide. This might be run from another shell session (2$) to save time:

2$ git clone https://github.com/Azure/acs-engine
2$ cd acs-engine
2$ ./scripts/devenv.sh # run ./scripts/devenv.ps1 on Windows
2$ make bootstrap
2$ make build

While it is building feel free to go through step 3 in the shell 1$.

3. Setup your azure kubernetes ecs template

Below script will replace clientID , secret, keyData and adminPassword keys in examples/windows/kubernetes-hybrid.json and write output to examples/windows/my-kubernetes-hybrid.json. This can also be done via text editor. Please note,$spName and $spPassword were generated above, scripts supposes same shell session, hence1$. $linuxKeyData is public key to login to Linux via ssh, $windowsPassword is password to access Windows Agent via RDC:

1$ export linuxKeyData=$(< ~/.ssh/id_rsa.pub)
1$ export windowsPassword='ReplaceWithSecurePassword1234'
1$ export dnsPrefix='my-kube-hybrid'
1$ mkdir output
1$ jq <examples/windows/kubernetes-hybrid.json ".properties.servicePrincipalProfile.clientId=\"$spName\" | .properties.servicePrincipalProfile.secret=\"$spPassword\" | .properties.linuxProfile.ssh.publicKeys[0].keyData=\"$linuxKeyData\" | .properties.windowsProfile.adminPassword=\"$windowsPassword\" | .properties.masterProfile.dnsPrefix=\"$dnsPrefix\"" >_output/my-kubernetes-hybrid.json

4. Generate Azure RM template

Generation based on .json template should be performed from the acs-engine container shell (2$):

2$ ./bin/acs-engine generate _output/my-kubernetes-hybrid.json
INFO[0000] Generating assets...

It should produce all the necessary configurations to start kubernetes cluster in azure under _output/${dnsPrefix} dir:

1$ $ ls _output/${dnsPrefix}
apimodel.json azuredeploy.json ca.key kubeconfig
apiserver.crt azuredeploy.parameters.json client.crt kubectlClient.crt apiserver.key ca.crt client.key kubectlClient.key

5. Create Kubernetes cluster

Everything ready to command az CLI to start the cluster:

1$ export kubeGroup='kube-group'
1$ az group create --name $kubeGroup --location eastus
1$ export clusterInfo=`az group deployment create --name ${kubeGroup}-deployment --resource-group ${kubeGroup} --template-file "./_output/${dnsPrefix}/azuredeploy.json" --parameters "./_output/${dnsPrefix}/azuredeploy.parameters.json"`

6. Connect to master and agents

Ssh to kubernetes master and see all nodes

1$ export masterHost=`echo $clusterInfo | jq ".properties.outputs.masterFQDN.value" -r`
1$ ssh azureuser@${masterHost}
azureuser# kubectl get nodes
NAME STATUS AGE VERSION
40089acs9010 Ready 24m v1.6.6-9+8a67b481dfc2c6
40089acs9011 Ready 24m v1.6.6-9+8a67b481dfc2c6
k8s-linuxpool1-40089363-0 Ready 24m v1.6.6
k8s-linuxpool1-40089363-1 Ready 24m v1.6.6
k8s-master-40089363-0 Ready,SchedulingDisabled 24m v1.6.6
azureuser# logout

Now you can build ssh tunnel to connect directly to Linux nodes from desktop:

1$ ssh -f azureuser@${masterHost} -L 3390:k8s-linuxpool1-40089363-1:22 -N
1$ ssh azureuser@localhost -p 3390
azureuser# sudo docker ps
...
azureuser# logout

Also runnel to connect to Windows agent nodes via RDC:

1$ ssh -f azureuser@${masterHost} -L 3389:40089acs9010:3389 -N
1$ # now you can connect to agent via RDC to localhost using azureuser:${windowsPassword} as authorization

7. Connect desktop through kubectl

There is shortcut to setup kubectl working with Azure via az CLI:

1$ az acs kubernetes install-cli
1$ scp azureuser@${masterHost}:.kube/config .
1$ export KUBECONFIG=`pwd`/config
1$ kubectl proxy

Open kubernetes dashboard in browser http://localhost:8001/ui

P.S. Do not forget to specify Podspec.nodeSelector for your pods to constrain them to specific OS.

    "nodeSelector": {
"beta.kubernetes.io/os": "windows"
}

P.P.S. Additional materials, scripts and presentation available in my github repo.

Maxim Vorobjov

Written by

Save humans! Robots should do stupid work

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade