From Monolith to Microservice Architecture on Kubernetes, part 5— Deployment Automation & Continuous Delivery

In this blog series we’ll discuss our journey at Cupenya of migrating our monolithic application to a microservice architecture running on Kubernetes. In the previous parts of the series we’ve seen how the core components of the infrastructure, the Api Gateway and the Authentication Service, were built, how we converted our main application to a microservice running in Kubernetes and how we dealt with logging, monitoring & tracing. In this post we’re going to see how we automated deployment by setting up our CI/CD pipeline.

Parts

• Part 1: The Api Gateway
• Part 2: Authentication with JWT
• Part 3: Deploying our Scala app as a microservice
• Part 4: Monitoring, health checks, logging & tracing
• Part 5: Deployment automation & continuous delivery (this part)

It’s been a while since my previous post in this series so meanwhile I gained some new insights and learned about alternative approaches with regards to the topic of deployment automation. I’ll try to incorporate those insights and learnings when explaining our setup at Cupenya.

Disclaimer: Here be dragons

There will be a lot of rather dirty scripting code snippets. I tried to keep them minimal for the purpose of this post, but I wanted to show (parts of) the real scripts we used to make our deployment automation work. I’m sure there’s plenty of great tools around nowadays that we might not have been aware of which can be used to accomplish a similar result in a cleaner way.

CI/CD

As you probably already know CI/CD stands for continuous integration and continuous delivery. Continuous integration is the practice of integrating, or merging when working with VCS branches, code which is under development in the shared mainline (e.g. master branch) continuously. In an ideal world this should occur on every change (push), but it should be at least multiple times a day. Continuous delivery is about actually deploying your changes in production. The idea is that every integrated code change can and will be deployed to production.

Let’s look at the steps involved getting our Scala app deployed that we discussed in part 3

1. Compiling and packaging
2. Running automated tests
3. Building a docker image
4. Publishing the docker image to Google Container Registry
5. Writing a Kubernetes deployment descriptor
6. Rolling out the deployment using kubectl

Packaging your app in a docker image

All these steps need to be executed in our CI/CD pipeline. As seen in the previous part step 1 till 3 is covered by SBT and the docker plugin of the SBT Native Packager. This plugin generates a very simple Dockerfile which can build a docker image capable of running a packaged java application. It looks something like this:

FROM openjdk:11-jre-slim
WORKDIR /opt/my-app
ADD opt /opt
RUN ["chown", "-R", "daemon:daemon", "."]
USER daemon
ENTRYPOINT ["bin/my-app"]
CMD []

To build this docker image and publish it to your local docker server run:

$ sbt docker:publishLocal

Publishing to Google Container Registry

The docker plugin is also capable of publishing to a docker registry such as Google Container Registry (GCR). As explained already this is achieved by configuring the docker repository and package name in build.sbt. See the relevant section below

packageName in Docker := "eu.gcr.io/my-repo/" + name.value
dockerRepository := Some("eu.gcr.io")

Authorising against GCP

To be able to publish you need to authorise against GCP. Unfortunately it’s not possible to specify authentication from within the SBT build at the time of writing. You’ll have to explicitly login using the local docker binary. You can authorise using a user account or a service account. For automated deployments authorising using a service account is recommended. To procure a key for an existing service account run

$ gcloud iam service-accounts keys create [FILE] —-iam-account [SERVICE_ACCOUNT]@[PROJECT].iam.gserviceaccount.com

This will store a JSON formatted key in FILE. You can use this file to login your local docker binary

$ docker login -u _json_key -p $(cat FILE)

Now you are able to publish to GCR through another SBT task

$ sbt docker:publish

Building and publishing your app without the docker plugin

Alternatively, taking benefit from the nice multi-stage build feature of docker, you could also incorporate the whole build pipeline in a docker container. You can use Heiko Seeberger’s excellent scala-sbt as a base image and manually create a runnable container by executing SBT build tasks. The benefit of this approach is that it’s easily portable between CI systems. A downside is that you’ll have to maintain a custom Dockerfile which is a little more complex than the one generated by the docker plugin of SBT Native Packager. See below for a rather minimal example

Because of multi-stage you can keep your runner image small while having all build tools you need in your builder image. To build and tag the docker image simply run:

docker build -t eu.gcr.io/my-repo/my-app:$(git rev-parse --short @) .

Providing you are authorised against GCP (see section above) you can publish the image by running:

docker push eu.gcr.io/my-repo/my-app:$(git rev-parse --short @)

Generating a Kubernetes Deployment Descriptor

After successfully publishing to GCR using either method you can start writing descriptors for your Kubernetes resources. From part 3 we remember the minimum setup looks something like below

The spec.template.spec.containers[0].image variable contains the path to the published image and an imagePullSecret is specified which contains the credentials of a service account authorised to pull the image.

Since we want to automate the CI/CD pipeline we need to be able to generate this descriptor file and script the rollout (and rollback!) process. At the time we weren’t aware of Helm, a package manager for Kubernetes which allows you to configure, release, version, rollback and inspect the deployment of Kubernetes resources, so we rolled out our own solution. There’s an excellent post on doing CI/CD with Helm which would’ve saved us a lot of time. I’ll outline our custom solution here, but would advice to avoid it if possible.

We built our solution around the Mustache template system. We made a template for Kubernetes deployment and service descriptor files. These templates look like the ones below.

K8s deployment descriptor template

K8s service descriptor template

The templates in a Helm chart would probably look similar to these. Where the input for the variables in the template with Helm is defined in values.yaml we are embedding a microservice.json file in each microservice repository with the same purpose. An example:

To render the templates one can simply run

$ mustache microservice.json service.mustache > service.yaml
$ mustache microservice.json deployment.mustache > deployment.yaml

Almost. You might have noticed that the deployment.mustache template mentions a variable GIT_REV_PLACEHOLDER which is indeed a placeholder for the git revision. So we need to substitute it

$ mustache microservice.json deployment.mustache | sed"s#GIT_REV_PLACEHOLDER#$(git rev-parse --short HEAD)#" > deployment.yaml

That wasn’t too hard. This way, for each microservice we generate a k8s service and deployment descriptor ready to be rolled out. We don’t need any additional ingress setup, because our Api Gateway will register our microservice using its metadata (remember from the first part of this series?).

Rolling out the deployment using kubectl

Again. Don’t do this. Use Helm. But if you can’t use Helm:

$ kubectl apply -f service.yaml
$ kubectl apply -f deployment.yaml

This will apply our microservice deployment configuration to the live environment or create a new one if none exists. To watch the rollout process until completion run:

$ kubectl rollout status -f deployment.yaml

Caveat: Kubectl apply doesn’t do a full upsert; it will perform a three way merge between the old configuration, the new configuration and the live configuration. As a result only modified properties will override the live configuration! Helm uses a more simplistic two way merge approach which will disregard any modifications on the live config. I’d argue that this is probably the right approach.

Bringing microservice deployment to your CI system

Now that we’ve seen all the steps to deploy our microservice to Kubernetes it’s time to integrate this in our CI system. At the time we were using Jenkins, but I’ll also briefly show how to do this with Gitlab since that’s a very popular choice nowadays.

CI/CD with Jenkins

To implement a CI/CD pipeline in Jenkins the Jenkins Pipeline suite can be used. You can model your CI/CD pipeline “as code” via the DSL syntax. You will declare your pipeline in a text file called Jenkinsfile and commit this to your microservice repository, so it’s versioned and reviewed like any other code.

To declare our build pipeline in the Jenkinsfile we first specify a node to determine where in the Jenkins environment our pipeline will be executed. Then we define your stages (e.g. compile, test, deploy) which contain the logical build steps of your pipeline.

For certain steps, such as publishing to a repository, we will need to provide credentials. You can use the Credentials Plugin to manage credentials and the Credentials Binding Plugin to bind them to variables available in your pipeline.

If we incorporate all the build and deploy steps from above, our Jenkinsfile will look similar to the one below.

Our microservice CI/CD pipeline using Jenkins Pipeline DSL

The Jenkinsfile should be available in all your repositories where you want to enable Jenkins Pipeline CI/CD. If you’re a fan of DRY principles you might consider moving (part of) this file to a Shared Library. This allows you to wrap your common scripts into groovy functions and make it available to any Pipeline job.

CI/CD with Gitlab CI

Whereas Jenkins Pipeline has the Jenkinsfile as its foundation, Gitlab CI is configured through a YAML file called .gitlab-ci.yml. I’ll give an example of how our CI/CD pipeline will look, but this time using our custom multi stage Dockerfile that we’ve defined earlier. We will make use of Protected Variables to hold our credentials, so we don’t need to do anything special.

Our CI/CD pipeline using Gitlab CI

Managing environments

Great! We are able to deploy our dockerized microservice in Kubernetes in an automated way. The pipelines above assume we only have one Kubernetes environment, however. In a more realistic setup we would need to be able to deploy to a full DTAP setup, where configuration of our deployment is environment specific. Let’s have a look how we managed to do this.

First of all you’ll have to decide whether you would setup a dedicated Kubernetes cluster per environment or use namespaces within a cluster or a hybrid of that. For the former you’d have to maintain credentials per cluster and your deployment script needs a switch to pull the corresponding one for your environment. For the latter you just need a switch around namespace labels (and pass —-namespace my-env to kubectl).

With Helm you’d probably setup a values.yaml per environment (dev, test, acc, prod) and have a switch in your deployment script. Gitlab CI makes it easy to trigger certain stages for specific branches only (e.g. only: [ master ]), so if you’re using git flow you can map branches to environments and it would be trivial to deploy a certain configuration for a specific environment.

Without Helm and Gitlab CI you’ll have to figure out something yourself, though, as we had to do. As explained before we maintained all our deployment scripts and templates in a Shared Library and our microservice specific deployment configuration in a microservice.json file in its corresponding repository. In addition to the deployment scripts and templates we also kept a mapping of environments to branches to Kubernetes clusters. We maintained a service account per environment and prefixed the corresponding credentials file (in Jenkins) with the name of the environment, so it was easy to resolve by the deployment script.

def deployForEnv(env) {
  stage('deploy') {    
    withCredentials([file(credentialsId: "${env}-gce-creds", variable: 'GOOGLE_AUTH_JSON')]) {
    // do deploy
    }
  }
}

In a similar fashion we were handling environment specific configuration. We maintained an environment specific microservice.json in the Shared Library in a folder with a name corresponding to the environment. Then before rendering the final deployment config template we merged the generic microservice.json with the environment specific one if one was defined.

def getServiceConfigContent(env, svc) {
  def content = null
  try {
    content = libraryResource("${env}/${svc}.json")
  } catch (err) {
    content = libraryResource("default/${svc}.json")
  }
  return content
}
echo 'Generating config file'
writeFile(file: 'msvc.json', text: confForEnv(env, svc))
sh "jq -s \".[0] + .[1]\" ${env}/microservice.json msvc.json > microservice-config.json"
echo 'Merged config'
// render templates and deploy

Feature branch environments

Nice, this allows us to deploy to our DTAP setup smoothly. What would be really nice if we could also support feature branches to quickly demo new features before they merged without breaking our test environments. Ideally we want to be able to spin up a new environment from scratch for a feature branch and tear it down when we’re done demoing.

Setting up a dedicated cluster would be a bit too much for this, so we can just use the dev cluster and rely on namespaces. We need to extend our mechanism from above to

• dynamically create environments based on a feature branch name and store
• provision those new environments with basic components (e.g. database endpoints) and secrets
• deploy branched version of services touched by our new feature
• deploy main line(master) version of services that are not touched by our new feature
• setup ingress / load balancer for our new namespace to make it accessible
• expose a cleanup job to tear down the environment for when we’re done

This also means we have some lifecycle management to do for our new environment. A developer might want to initially create a new clean environment for his feature branch, deploy a few updates to it and finally tear it down when he’s done. That means our deployment script needs to keep track of these environments. Our mapping from environments to branches to kubernetes clusters that we maintain in our Shared Library, as explained earlier, now needs to be dynamic. We actually stored this mapping in a simple CSV file called envs.csv where each line contains a mapping for a (feature) branch. This file is updated all the time by committing changes into git.

A developer could manually kick off a Jenkins job which would allow him to manage environments. We chose to rely on user input for setting up / tearing down environments for branches:

def actions = ["add-env", "remove-env"].join('\n')
def envNames = getEnvNames().join('\n') \\ parse from envs.csv
def params = [
            choice(choices: actions, name: 'action'),
            choice(choices: envNames, name: 'env'),
            string(name: 'branchName')
          ]
def userInput = input(
          id: 'userInput',
          message: 'Add or remove environment?',
          submitterParameter: 'submitter',
          parameters: params
        )
def envConf = getEnvByName(userInput.env) // find env in envs.csv
if (userInput.action == "add-env") {
  sh ./init-env-from-branch.sh ${userInput.branchName}
} else if (userInput.action == "remove-env") {
  sh ./delete-env.sh ${envConf.namespace} ${envConf.branch}
}

We use a fixed pattern for feature branches: feature/${JIRA_TICKET_ID}-some-post-fix which makes it easy to relate a feature branch to a user story. We can use this pattern to consistently name our environments and make it easy to understand which environment belongs to which feature and user story. For adding an environment the following shell script is called:

Once an environment is setup we are almost ready to deploy our microservice to it. After all, the DNS is already setup and we even have a public IP already so we can share the $JIRA_TICKET_ID.my-domain.com url with anyone interested to preview our feature. However, the environment is still pretty empty so we need to first provision it with

• our core components, such as api-gateway and auth service
• the version of our services touched by our new feature
• the main line version of the services which are NOT touched by our new feature

To accomplish this we need to keep a registry of the repositories of our core components and services. We also stored this in our Shared Library under a file call services.json. This file contained a mapping of service/component name to their git repository. Based on this file we could provision our newly created environment

The script checks for each service if there’s a feature branch available and deploys that OR falls back to master if it isn’t.

Now that our feature environment is fully provisioned we can keep updating it with new versions of our microservices that are touched by the feature. After each push on our microservice repository we would check if and where we should deploy this update by figuring out if there’s a or multiple deployment mapping(s) for the branch that we’re in:

def envs = getEnvs() // parse envs.csv
envs.each {
  def envConf = it
  def branchForEnv = envConf.branch
  def currentBranch = env.BRANCH_NAME // get branch from Jenkins var
  if(branchForEnv == currentBranch) {
    deployForEnv(envConf.namespace) // deploy 
  } else {
    echo "Skipping ${envConf.namespace}" for deploy 
  }
}

Eventually, after we’re done playing around, we need to clean up after ourselves. For tearing down we call the following simple script:

Well, that’s all there is to share about our CI/CD pipeline for microservices. We’ve covered deploying a scala based microservice to Kubernetes using custom templates or with a tool like Helm. We’ve also seen how to bring this to CI systems like Jenkins or Gitlab CI. Finally, I showed how we managed deployment to multiple (feature) environments.

Thanks for reading my series. If you like this post, have any questions or think my code sucks, please leave a comment!