Kubernetes Adventures on Azure — Part 3 (ACS Engine & Hybrid Cluster)

Ivan Fioravanti
Aug 28, 2017 · 6 min read

My goal for today is installing a simple Hybrid Kubernetes cluster on Microsoft Azure with:

  • 1 Linux Master
  • 2 Linux Nodes in a LinuxAgentPool
  • 2 Windows Nodes in a WindowsAgentPool

This is something that can’t be accomplished out of the box with the current version of Azure Container Services, but yesterday Brendan Burns confirmed they are working on it!

So how can we install this cluster right now? The answer is ACS-Engine an Open Source project from Microsoft that takes a “cluster definition” in input and returns an ARM template with additional artifacts (i.e. kubectl export files) as output.

Let’s start!

  • Create a folder named acs to store all your working artifacts
  • Clone ACS Engine Repository: git clone https://github.com/Azure/acs-engine
  • Go in the Example folder: cd acs-engine/examples where you can find many cluster definition examples. Here you can see a windows folder, it contains the definition needed: kubernetes-hybrid.json

Important note: There is an issue with the current release 0.5.0 that leads to examples in source code not being compatible with it (they are more updated). Here we have 2 solutions:

  • Complex: Compile a new Release locally
  • Easy: Change Examples to make them compatible with old version

We will choose the complex one! Joking… I followed the complex but it’s not trivial on macOS at the moment, because you need to clone acs-engine source in $GOPATH/src and needs a separate article on its own. So let’s go for the easy solution.

  • Open kubernets-hybrid.json file in your favorite editor and change lines below

from:

"servicePrincipalProfile": {
"clientId": "",
"secret": ""
}

to:

"servicePrincipalProfile": {
"servicePrincipalClientID": "",
"servicePrincipalClientSecret": ""
}

Wait don’t close the file! We want to use latest Kubernetes version in our cluster right? So add line in bold:

"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorVersion": "1.7.2"
},

Now we can follow steps in Deploy a Kubernetes Cluster that describes a short and long way to deploy your cluster.

Short way drawbacks

  • DNS prefix: a random number will be added to prefix passed as arguments to make it unique
  • A new application (ServicePrincipal) will be created on you subscription each time you run deploy command and will be left there when you delete ResourceGroup because it’s not part of it
  • A new SSH key pair will be created for deploy

Solution 1: We can avoid all these drawbacks creating artifacts once (application, SSH key pair, DNS) and adding their value in template json file used for deployment. I will do this for SSH.

Solution 2: At the end of the first deploy, acs-engine generates a _output folder with generated artifacts. Here there is a file called azuredeploy.parameters.json that contains all values for parameters that can be copied in the cluster definition file used. That’s all.

Deployment

"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": "YOURKEY starts with ssh-rsa"
}
]
}
},

For deploy you need subscription id you are working on that can be easily retrieved with az account show

./acs-engine deploy — subscription-id YOURSUBSCRIPTIONIDHERE — dns-prefix ivank8stest — location westeurope — auto-suffix — api-model ~/acs/acs-engine/examples/windows/kubernetes-hybrid.json

Note: If you want to avoid passing dns-prefix as argument you can add it to template.json file as we did for keyData same trick can be used for any elements in the file. acs-engine is smart enough to skip steps if data are present. i.e. ServicePrincipalProfile creation.

If everything went well you should see something like:

WARN[0002] apimodel: missing masterProfile.dnsPrefix will use “ivank8stest-59a13555”
WARN[0002] apimodel: ServicePrincipalProfile was missing or empty, creating application…
WARN[0004] created application with applicationID (11111111-048e-420f-afad-1fe450036077) and servicePrincipalObjectID (22222222-b0fb-4395–9576–6bb18481f88f).
WARN[0004] apimodel: ServicePrincipalProfile was empty, assigning role to application…
INFO[0030] Starting ARM Deployment (myAcsTest-1222046555). This will take some time…
Coffee break is needed!

INFO[0526] Finished ARM Deployment (myAcsTest2–676730522).

We did it! Our Hybrid Kubernetes Cluster seems up and running! Let’s test it now.

Connect to your Kubernetes cluster

It contains artifacts generated by acs-engine and kubeconfig is really useful for us as described here in step 9. I will try to make it super simple here.

From your terminal run following command point to your _output folder

export KUBECONFIG=~/acs/acs-engine/_output/ivank8stest-59a13ac6/kubeconfig/kubeconfig.westeurope.json
kubectl get nodes
NAME STATUS AGE VERSION
23586acs9010 Ready 1s v1.7.2-4+b0c9ea2463aba4
23586acs9011 Ready 3s v1.7.2-4+b0c9ea2463aba4
k8s-linuxpool1-23586643-0 NotReady 2s v1.7.2
k8s-linuxpool1-23586643-1 NotReady 5s v1.7.2
k8s-master-23586643-0 NotReady,SchedulingDisabled 7s v1.7.2

It’s up and running and we can use it with our local kubectl!

Everything is up and running, let’s try to deploy some Linux and Windows containers!

Linux Containers

Here is the content of the azure-vote.yaml file with changes highlighted:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: azure-vote-back
spec:
replicas: 1
template:
metadata:
labels:
app: azure-vote-back
spec:
containers:
- name: azure-vote-back
image: redis
ports:
— containerPort: 6379
name: redis
nodeSelector:
beta.kubernetes.io/os: linux

---
apiVersion: v1
kind: Service
metadata:
name: azure-vote-back
spec:
ports:
— port: 6379
selector:
app: azure-vote-back
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: azure-vote-front
spec:
replicas: 1
template:
metadata:
labels:
app: azure-vote-front
spec:
containers:
— name: azure-vote-front
image: microsoft/azure-vote-front:redis-v1
ports:
— containerPort: 80
env:
— name: REDIS
value: "azure-vote-back"
nodeSelector:
beta.kubernetes.io/os: linux
---
apiVersion: v1
kind: Service
metadata:
name: azure-vote-front
spec:
type: LoadBalancer
ports:
— port: 80
selector:
app: azure-vote-front

You can now deploy it using: kubectl create -f azure-vote.yaml

Leave Kubernetes working and let’s move to…

Windows Containers

apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: iis
spec
replicas: 2
template:
metadata:
labels:
app: iis
spec:
containers:
- name: iis
image: microsoft/iis
ports:
- containerPort: 80
name: iis
nodeSelector:
beta.kubernetes.io/os: windows
---
apiVersion: v1
kind: Service
metadata:
name: iis
spec:
ports:
- port: 80
selector:
app: iis

Again we can deploy it using: kubectl create -f iisdeploymentfull.yaml

Now we need to wait few minutes to have everything up and running.

Check status from Kubernetes Dashboard

From here feel free to navigate in the various sections to check status of your cluster, pods, deployments and services.

From Services you will be able to retrive External Endpoints that you can use to test both iis and azure-vote deployments.

Let’s close this article with a quick scale of both deployments to push our cluster:

kubectl scale deployments/azure-vote-front --replicas 100

kubectl scale deployments/iis --replicas 4

Final result:

Kubernetes Hybrid Cluster running on Microsoft Azure

As always you can delete everything with a simple Azure CLI 2 command: az group delete --name myAcsTest --yes --no-wait

We did it again! Now we are Azure Kubernetes Masters and we can easily deploy whatever we want!

My adventure will continue with ingress distributed on Linux and Windows nodes, Monitoring, Autoscale on Azure and much more.

HackerNoon.com

how hackers start their afternoons.

Ivan Fioravanti

Written by

Co-founder and CTO of @4wardsrl, creator of @4ward365, Developer, MongoDB, Kubernetes, Azure, Mensa member, Innovation lover and insatiable Reader.

HackerNoon.com

how hackers start their afternoons.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade