Trying out Jenkins X on Azure Kubernetes Service(AKS)

When I was at Jenkins World 2018, everybody seemed to be talking about Jenkins X. An opinionated Kubernetes and Jenkins with automated CI/CD pipelines that allow you to focus on writing awesome applications for your customers. So I decided to jump on the high-speed rail and experience how Jenkins X is working on AKS.

Unfortunately, like a few of you, my train ride was not that smooth. I was not able to build the pipeline with the Docker registry created in my AKS cluster (GitHub issue.) So I switched the built-in Docker registry out with Azure Container Registry(ACR) and things started to work smoothly. While my team work on creating a PR to default to ACR for AKS, here’s how you can workaround the issue.

Switch to ACR and add secrets to Jenkins X and Kubernetes

  • Jenkins X did a great job documenting how to get started. I followed the steps here to install the jx binary on Windows. The jx version I have is 1.3.435. I also ran all jx, az cli, kubectl commands from my laptop.
Tip: if you have more than one subscription, use az account set --subscription XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXXto default to the right one
  • Next, I installed Jenkins X on an existing AKS cluster by using the jx install command
Tip: to create an AKS cluster with 6 nodes, use: az aks create -g yourResourceGroup -n yourAKS --node-count 6
  • Run jx console to open and sign into Jenkins
  • Switch to ACR instead of the built in Docker registry. Go to Manage Jenkins → Configure System and update the DOCKER_REGISTRY environment variable to your ACR.
Update global properties in Jenkins
Tip: to create an admin enabled ACR in your Azure subscription, use: az acr create --name yourACR --resource-group yourResourceGroup --sku Standard --admin-enabled true
  • You need to add ACR credentials to the jx namespace so your Jenkins agent can push images to ACR. Run:

jx create docker auth --host "yourACR.azurecr.io" --user "yourACRUser" --email "yourEmail@xxx.com" --secret "yourACRSecret" create

Tip: you can go to Azure portal or use az acr credential show --name yourACR to get login credentials for ACR.
  • Because two other namespaces are created by Jenkins X, you need to add the credentials to jx-staging and jx-production as well so that you can pull the Docker images from ACR. Run the following commands with your ACR credentials:

kubectl create secret docker-registry jenkins-docker-cfg --docker-server="yourACR.azurecr.io" --docker-username="yourACRUser" --docker-password="yourACRSecret" --docker-email="youremail@xxx.com" --namespace=jx-staging

kubectl create secret docker-registry jenkins-docker-cfg --docker-server="yourACR.azurecr.io" --docker-username="yourUser" --docker-password="yourACRSecret" --docker-email="youremail@xxx.com" --namespace=jx-production

Create a Spring Boot app and test end to end

  1. Run jx create spring -d web -d actuator to create a new Spring boot app and add a test home page. i.e., just follow steps in this demo.
  2. Commit and push the test home page to the master branch.

My staging environment is successfully updated!

Notes:

If you create a PR, you will run into once again, the same authentication issue.

When a PR is created, Jenkins X creates a new namespace for the PR. You can list all the namespaces by running kubectl get ns

Namespaces in my AKS cluster

Suppose jx-XXXX-pr-X is your PR namespace, run kubectl describe pod --namespace=jx-XXXX-pr-X to find out what's wrong:

Authentication error in pod

To work around, manually add secret to the PR namespace.

kubectl create secret docker-registry jenkins-docker-cfg --docker-server="yourACR.azurecr.io" --docker-username="yourACRUser" --docker-password="yourACRSecret" --docker-email="youremail@xxx.com" --namespace=jx-XXXX-pr-X

Update (Oct 30 2018)
Because secret stored in jx namespace is not available to other application namespaces, you need to grant jx, jx-staging, jx-production and jx-XXX-pr-X access to pull image from ACR. Instead of running kubectl create secret for each namespace (including the PR namespace) like above, you can run the following commands to assign your AKS a reader role to ACR.
CLIENT_ID=$(az aks show --resource-group <yourAKSResourceGroup> --name <yourAKSCluster> --query "servicePrincipalProfile.clientId" --output tsv)
ACR_ID=$(az acr show --name <yourACR> --resource-group <yourACRResourceGroup> --query “id" --output tsv)
az role assignment create --assignee $CLIENT_ID --role Reader --scope $ACR_ID

I also ran into an issue when the (build) agent pod in AKS got stuck in pending state due to insufficient CPU. I solved the issue by scaling my AKS cluster: az aks scale --name yourAKSCluster --resource-group yourResourceGroup --node-count X (where x is the number of nodes.)

As mentioned, we are working on a PR to default to ACR when running Jenkins X on AKS. You won’t need to do the above manual steps and will be able to do as Kohsuke said “just go to the station and take the high-speed rail to your end destination.” Stay tuned!

Update (Jan 8 2019)
The PR to default to ACR is merged. Just tried with jx version 1.3.694. It is not necessary to perform these manual steps anymore.

We are always looking for feedback, please leave a comment here, or in Jenkins X Slack channel, or email Azure Jenkins Support.