Deploy the Kubernetes Cluster Autoscaler into Azure AKS from an ADO Pipeline
Recently Microsoft released documentation describing how to deploy the Kubernetes Cluster Autoscaler onto AKS in Azure. To ease the deployment of this, I created an Azure DevOps(ADO) pipeline.
Secrets
The documentation has you run a bash script to generate the secrets YAML that will get deployed. You can generate an SPN as the example does, or use already existing credentials.
You can store these credentials in secret ADO Variables or Variable Groups, or pull them directly from a KeyVault. In my example I pull some directly from the KeyVault and convert them to base64 values, and some I store in secret build variables as their base64 values.
Task 1: Read values from an Azure KeyVault
I use the Azure Key Vault task to read the Client ID and Client Secret. I’ll convert these to base64 values in task 2.
- task: AzureKeyVault@1
displayName: 'Azure Key Vault: MYKEYVAULT'
inputs: azureSubscription: 'My Sub'
KeyVaultName: MYKEYVAULT
SecretsFilter: 'CLIENT-ID, CLIENT-SECRET'Task 2: Create the secrets YAML file
I use a command line task to convert the Client ID and Secret variables to base64 values and cat that into the secret YAML file. I also include the base64 values for the other entries from secret ADO variables.
- script: |
client_id=$(echo -ne "$(CLIENT-ID)" | base64)
client_secret=$(echo -ne "$(CLIENT-SECRET)" | base64)
cat << EOT > aks-cluster-autoscaler-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: cluster-autoscaler-azure
namespace: kube-system
data:
ClientID: $client_id
ClientSecret: $client_secret
ResourceGroup: $(resource_group_base64)
SubscriptionID: $(subscription_id_base64)
TenantID: $(tenant_id_base64)
VMType: QUtTCg==
ClusterName: $(cluster_name_base64)
NodeResourceGroup: $(node_resource_group_base64)
EOT
displayName: 'Create aks-cluster-autoscaler-secret.yaml'Task 3: Create the autoscaler YAML file
Task 3 is another command line task to generate the autoscaler YAML. I provide a link to the full example below. You can also get the content from the documentation page linked above.
- script: |
cat <<EOT > aks-cluster-autoscaler.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
name: cluster-autoscaler
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
...
EOT
displayName: 'Create aks-cluster-autoscaler.yaml'In my full example I have hard coded the min/max nodes, but you could create these as variables in your build pipeline.
Tasks 4 and 5: Deploy to Kubernetes
In the final two tasks I use the “Deploy to Kubernetes” task to apply the YAML files to the AKS cluster. These tasks take advantage of the subscription endpoints you can configure within ADO.
- task: Kubernetes@1
displayName: 'kubectl apply aks-cluster-autoscaler-secret.yaml'
inputs:
azureSubscriptionEndpoint: 'My Team' azureResourceGroup: MY_AKS_RG kubernetesCluster: MY_AKS_CLUSTER useConfigurationFile: true configuration: 'aks-cluster-autoscaler-secret.yaml'- task: Kubernetes@1
displayName: 'kubectl apply aks-cluster-autoscaler.yaml'
inputs:
azureSubscriptionEndpoint: 'My Team' azureResourceGroup: MY_AKS_RG kubernetesCluster: MY_AKS_CLUSTER useConfigurationFile: true configuration: 'aks-cluster-autoscaler.yaml'
Full Example
You can find the full example in this Github Gist.
The Easy Button
After I had gotten the ADO Pipeline running I discovered that there is a new “aks-preview” extension for Azure CLI that will make this a much easier process.
It’s likely the Cluster Autoscaler feature will get built into the Terraform AzureRM Provider and other tools in the near future as it comes out of preview and support for it gets added to the Azure SDK.
This was an interesting exercise and demonstrates some of the flexibility of Azure DevOps Pipelines.
