Linux and Windows in one hybrid Kubernetes cluster
Azure provides acs automation for either Linux or Windows based kube agents, though not both. We will use opensource acs-engine tool to setup single Kubernetes cluster combining both Windows and Linux agent nodes. Moreover we will generate cluster template, so that deployment can be reproduced with one command.
Setup is using az CLI to create service principal. If you do not have the tool please follow Install Azure CLI 2.0.
Make sure to have latest version of
az CLI!
Instructions provided for bash. Also we will use jq to process output JSONs.
Apart of az CLI we will use ssh to setup network tunnel to connect to agent nodes which wouldn’t have public IPs.
1. Create service principal
Kubernetes cluster requires Azure service principal to manage connected Azure infrastructure, like load balancer, storage. Creation is simple via az CLI:
1$ export SUBSCRIPTION_ID=`az account show | jq ".id" -r`
1$ export res1=`az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/${SUBSCRIPTION_ID}"`
1$ export spName=$(echo $res1 | jq ".appId" -r)
1$ export spPassword=$(echo $res1 | jq ".password" -r)2. Setup ecs-engine on local machine
Below setup considering that you have docker engine running locally, see ecs-engine in Docker for details. Otherwise refer to the ecs-engine guide. This might be run from another shell session (2$) to save time:
2$ git clone https://github.com/Azure/acs-engine
2$ cd acs-engine
2$ ./scripts/devenv.sh # run ./scripts/devenv.ps1 on Windows
2$ make bootstrap
2$ make buildWhile it is building feel free to go through step 3 in the shell 1$.
3. Setup your azure kubernetes ecs template
Below script will replace clientID , secret, keyData and adminPassword keys in examples/windows/kubernetes-hybrid.json and write output to examples/windows/my-kubernetes-hybrid.json. This can also be done via text editor. Please note,$spName and $spPassword were generated above, scripts supposes same shell session, hence1$. $linuxKeyData is public key to login to Linux via ssh, $windowsPassword is password to access Windows Agent via RDC:
1$ export linuxKeyData=$(< ~/.ssh/id_rsa.pub)
1$ export windowsPassword='ReplaceWithSecurePassword1234'
1$ export dnsPrefix='my-kube-hybrid'
1$ mkdir output
1$ jq <examples/windows/kubernetes-hybrid.json ".properties.servicePrincipalProfile.clientId=\"$spName\" | .properties.servicePrincipalProfile.secret=\"$spPassword\" | .properties.linuxProfile.ssh.publicKeys[0].keyData=\"$linuxKeyData\" | .properties.windowsProfile.adminPassword=\"$windowsPassword\" | .properties.masterProfile.dnsPrefix=\"$dnsPrefix\"" >_output/my-kubernetes-hybrid.json4. Generate Azure RM template
Generation based on .json template should be performed from the acs-engine container shell (2$):
2$ ./bin/acs-engine generate _output/my-kubernetes-hybrid.json
INFO[0000] Generating assets...It should produce all the necessary configurations to start kubernetes cluster in azure under _output/${dnsPrefix} dir:
1$ $ ls _output/${dnsPrefix}
apimodel.json azuredeploy.json ca.key kubeconfig
apiserver.crt azuredeploy.parameters.json client.crt kubectlClient.crt apiserver.key ca.crt client.key kubectlClient.key5. Create Kubernetes cluster
Everything ready to command az CLI to start the cluster:
1$ export kubeGroup='kube-group'
1$ az group create --name $kubeGroup --location eastus
1$ export clusterInfo=`az group deployment create --name ${kubeGroup}-deployment --resource-group ${kubeGroup} --template-file "./_output/${dnsPrefix}/azuredeploy.json" --parameters "./_output/${dnsPrefix}/azuredeploy.parameters.json"`6. Connect to master and agents
Ssh to kubernetes master and see all nodes
1$ export masterHost=`echo $clusterInfo | jq ".properties.outputs.masterFQDN.value" -r`
1$ ssh azureuser@${masterHost}
azureuser# kubectl get nodes
NAME STATUS AGE VERSION
40089acs9010 Ready 24m v1.6.6-9+8a67b481dfc2c6
40089acs9011 Ready 24m v1.6.6-9+8a67b481dfc2c6
k8s-linuxpool1-40089363-0 Ready 24m v1.6.6
k8s-linuxpool1-40089363-1 Ready 24m v1.6.6
k8s-master-40089363-0 Ready,SchedulingDisabled 24m v1.6.6
azureuser# logoutNow you can build ssh tunnel to connect directly to Linux nodes from desktop:
1$ ssh -f azureuser@${masterHost} -L 3390:k8s-linuxpool1-40089363-1:22 -N
1$ ssh azureuser@localhost -p 3390
azureuser# sudo docker ps
...
azureuser# logoutAlso runnel to connect to Windows agent nodes via RDC:
1$ ssh -f azureuser@${masterHost} -L 3389:40089acs9010:3389 -N
1$ # now you can connect to agent via RDC to localhost using azureuser:${windowsPassword} as authorization7. Connect desktop through kubectl
There is shortcut to setup kubectl working with Azure via az CLI:
1$ az acs kubernetes install-cli
1$ scp azureuser@${masterHost}:.kube/config .
1$ export KUBECONFIG=`pwd`/config
1$ kubectl proxyOpen kubernetes dashboard in browser http://localhost:8001/ui
P.S. Do not forget to specify Podspec.nodeSelector for your pods to constrain them to specific OS.
"nodeSelector": {
"beta.kubernetes.io/os": "windows"
}P.P.S. Additional materials, scripts and presentation available in my github repo.
