DraftKings Kubernetes Workshop Part II: Hands-on Learning with Helm(with Video Walkthrough)

This is part two of a three-part series getting hands-on with Kubernetes and Helm. Part one is here.

Why Helm?

As we learned in the previous workshop, Kubernetes is very configurable, and you can use it to easily create a micro-service with a load balancer. When you start dealing with entire applications, the amount of configuration to deploy all the different pieces can be hard to manage. YAML configuration, while flexible, is also hardcoded and not command-line overridable.

Enter Helm

Helm has a lot of capabilities, many of which we’ll explore in the following two workshops. Its main purpose is to be a “package manager” for Kubernetes-deployed “things.” For example, if you want to deploy an entire sports betting application, you would no longer need to individually deploy every micro-service. You can set them as dependent packages, and Helm will install them if they’re not already installed.

It also has an extensive templating engine that enables you to make common YAMLs and pass in configuration or overrides for specific cases.

Terminology

Packages or applications in Helm are called Charts.

I think they’re called charts because Helm is short for Helmsman, and a helmsman used to chart courses for boats, but there’s no official documentation stating that.

Helm installs charts as Releases of an application. Every instance of a chart that’s installed (which is not really a single instance but a collection of resources, services, pods, etc.) is considered a release of that chart. You can install the same chart many times under different release names, e.g., many different PostgeSQL databases. Releases are also versioned.

Releases also have a benefit: they enable you to manage and track a large number of objects deployed into K8s together as one unit.

Chart

Difference With K8s YAMLs (Templating)

Helm allows you to parameterize your YAMLs. You can put variables in them and have them replaced by Helm. Helm does this from two main mechanisms:

1. values.yaml This file declares all the variables that can appear in your templates, as well as their default values.
2. On the command line, you can override anything declared as a variable in values.yaml

Example

Don’t worry if the above YAML doesn’t make sense yet; we’ll be walking through all the concepts used in it throughout the workshop.

Complex Variable Options

In addition to simple variable replacement, you can also control things like indentation (which is important in a YAML structure) and other complex functionality. You can include other templates (as seen below) and then run functions on that using the pipe (|) character like in Linux.

The Dot

You’ll see the dot character used throughout Helm chart templates. It’s a relatively simple concept: think of it as a JSON path marker from the root object down to the actual properties.

• When it’s in front of a variable name, it means “find this object or property on the root object,” e.g. .Values.name essentially it means MyHelmChart.Values.name.
• When it’s used on its own inside the curly brackets, e.g. {{ . }}, this will write out the root object as a value
• When it appears after a function, it will be passed as an argument to the function, e.g. {{ include “myfunction" . }}
• You can also pass any object or value under the root as an argument as well, e.g. {{ include “myfunction" .Values }}
• When it’s used in the middle of a variable “path” it’s a lookup similar to Json lookups, e.g. .Values.spec.metadata.namespace would walk down that tree. This would correspond to a YAML structure like:

Workshop Overview

This is a workshop that we’ve run with some of our engineers to help demystify Helm and introduce some of the concepts.

In this session, you’ll learn how to:

• Convert the Kubernetes tutorial to be deployed via Helm
• Use the base Helm capabilities (easy 1-step install, package, etc.)
• Use a chart repository
• Deploy with command-line value overrides
• Understand Helm functions
• Use dependent charts, including conditionally
• Helm versioning, upgrades, rollbacks

How To Use The Workshop

All the files you’ll create are viewable in their final state at https://github.com/dmusicant-dk/helm-workshop.

Throughout this workshop, you’ll see a lot of new concepts introduced. Whenever they’re inside code, for example, in a YAML file, we’ll include comments in the code file itself that explain the concepts. Having the documentation close to the code should make it clearer for you.

We also want to call out that this is not a production-quality application but for this workshop. For example, you’ll deploy a database into K8s without stateful, persistent storage. That is beyond the scope of this exercise.

Below you’ll walk through the following steps:

1. Step 1: Setup (Pre-Requisites, Helm and Chart Museum)
2. Step 2: Clear Out Your Cluster (From Previous Workshop)
3. Step 3. Make the Database Chart
4. Step 4. Package and Publish The Chart
5. Step 5. Install/Update/Rollback/Uninstall A Chart
6. Step 6. Use a Dependent Chart
7. Step 7. Adding the REST Application

For each, you’ll see a “Hands-On Steps” to indicate where your work begins.

Video Walkthrough

You can also follow along with all the steps in this article by watching the video below:

Step 1: Setup

Pre-Requisites

It’s recommended that you complete the first workshop before doing this one.

You can run this exercise in any Kubernetes cluster on your laptop. We recommend using Minikube, but you can also use Docker Desktop’s Kubernetes as well. We’ll call out all places where there are differences.

Kubernetes Pre-Requisites

See the Pre-Requisites section of the previous workshop to install K8s, Docker, and other non-Helm-related requirements.

Helm Pre-Requisites

Install Helm

As with the previous tutorial, we’re going to be using the Chocolatey package manager on Windows. For other platforms, see here.

Install ChartMuseum

We’ll need a repository to hold our charts so Helm can download and install the right versions.

Connect Helm and ChartMuseum

Windows Users Note: You must use git bash to run this command on Windows as the ChartMuseum developers only made a “.sh” install script.

Later we’ll show how we can push our chart to the repository and have it automatically update the index. You can also read some of the options in the official README.

Some commands you can use if you want to verify anything or fix something in ChartMuseum:

Step 2: Clear Out Your Cluster (From Previous Workshop)

You only need this step if you participated in the previous workshop.

Step 3. Make The Database Chart

Since the database will be our dependency for the web application, we’ll make this chart first. The recommended way to get started with Helm is to use the Create command. This will give us the structure we want and a lot of unneeded files and file contents we’ll remove.

Create the Helm Chart

Make sure you’re in a good working directory and use Helm’s Create command:

helm create db-application

Now change into the newly created directory: db-application. You should see the following file structure:

Hands-On Steps

Let’s start by deleting some of the files we don’t need:

• templates/deployment.yaml
• templates/hpa.yaml
• templates/ingress.yaml
• templates/NOTES.txt
• templates/service.yaml
• templates/serviceaccount.yaml
• templates/tests/test-connection.yaml

Application MetaData

We’re going to be making this chart fit the MySql deploy we created in the last workshop.

Hands-On Steps

1. To get started, open Chart.yaml and change it to be:

Database Secret

If you remember from the last workshop, we used a secret to hold our MySql user and password. We’re going to templatize that now.

Hands-On Steps

1. Create a file db-application/templates/db-credentials.yaml
2. Paste in this code:

You’ll notice on lines 22, 23 we’re not just setting values for YAML properties but constructing an entire name: value pair. You can construct entire YAML structures in your functions.

3. Open the db-application/templates/_helpers.tpl file and fully replace it with this code (we’ll explain it in a bit):

4. Open db-application/values.yaml and change it completely to be:

5. At the command line in the directory that contains your db-application chart directory (not inside the chart’s directory) run the command:

You should see the db-credentials.yaml fully printed out with all correct values and not the template variables.

A few important notes:

• Helm functions are written in a “sort-of” Go language. It’s not a perfect match, and there are some interesting differences between a Helm template and a Go template
• Although we chose to give a hierarchical structure to our values.yaml properties, there’s no requirement for this (and as we’ll soon see, it’s usually a bad practice). We could have used names like secretKeyUserNameValue instead.
• We are specifying a namespace in our document. This is not usually the best practice, but we’re doing it for simplicity for this tutorial.

Functions Explained

As mentioned earlier, Helm uses a Go-based language for its functions, or what it calls helpers. All “code” appears between double curly brackets {{}}. So even comments have to appear between them.

Make sure you read the comments in the below snippet to understand the code fully.

As mentioned in the comments above, you can only pass a single argument to any function, but we often need to pass multiple things. To handle this, we create and pass a dictionary (dict) holding those items.

{{ include "db-application.getSecretData" (dict "source" . "index" 0) }}

The above snippet passes a dictionary with two keys:

• source holds the entire root object .
• index holds an integer 0

As touched on above, functions don’t have return values but instead are continually printing output to the output stream. This is why Helm uses the include keyword for calling functions; you’re “including” its output in your YAML.

DB Deployment

Let’s now take the Deployment YAML from the last tutorial and convert it to be templated as well.

Hands-On Steps

1. Create a file db-application/templates/db-deployment.yaml
2. Paste in this code:

3. Open the db-application/values.yaml and completely change it to be:

4. Open db-application/templates/_helpers.tpl

5. Find the function db-application.selectorLabels and replace it with:

6. Now run the following command to see everything, including the secrets properly created:

Refer to our previous K8s Workshop, Step 1: Create a Secret, to see how the Base64 encoded user/pass is created.

The Pitfalls of Nested Values

You’ll notice that while the above command worked perfectly, it has some important drawbacks.

1. Overriding the values is messy and easy to get wrong
2. We have to write complex, hard to follow functions
3. Our templates require complex dictionaries to be passed
4. In Helm, when overriding a single value in a list/array/dictionary, you must override every single item as well

The fourth point above is not obvious but has big implications. If you were to only override the secretKey.data fields and not the secretKey.name field, you’d get an error. Helm would think you’re overriding the entire envVariables array to have only one item secretKey with a single property data.

When creating properties in your values.yaml, you should think about how the values will be overridden. Using a complex nested array or map structure is not recommended. We did it here to explore different templating capabilities and feel the pain. That pain is why Helm recommends favoring flat structures over lists.

Best Practice Interlude: Removing Lists in Values

Let’s simplify our structure so we can have an easier time in the rest of the workshop.

Hands-On Steps

1. Change your db-application/values.yaml completely to be:

2. Change db-application/templates/db-credentials.yaml completely to be:

3. Let’s remove all the unneeded functions by changing db-application/templates/_helpers.tpl completely to be:

Notice we removed the functions:

• “db-application.getSubKey”
• “db-application.getSecretName”
• “db-application.getSecretData”

4. Now check it with this one-line command:

DB Service

We have everything we need to add the Service for the database as well.

Hands-On Steps

1. Create the file db-application/templates/db-service.yaml and set the contents to:

2. Run the same command as before:

NOTE: If you want to check just a single file, you can do that with --show-only. For example, helm template ./db-application --set credentials.username=cm9vdA== --set credentials.password=ZGJwYXNzd29yZDE= --show-only templates/deb-service.yaml. However, you should be careful as it might hide changes to other files.

DB External Name

This file is conditionally deployed and won’t be installed in the db namespace. It will only be created by Helm and installed in Kubernetes if you enable it.

Hands-On Steps

1. Create the file db-application/templates/db-external-name.yaml and set the contents to:

2. Update the db-application/values.yaml completely look like:

3. Run with the command from before:

Notice there’s no db-external-name.yaml printed out!

4. Now try running it with it set enabled:

Step 4. Package and Publish the Chart

We’re now going to publish the chart to our Chart Repository, ChartMuseum, which we set up with Helm in the Setup section above. The version information is located in the db-application/Chart.yaml file.

Hands-On Steps

Run the following commands:

Step 5. Install/Update/Rollback/Uninstall a Chart

Now for the fun part: we’re going to use Helm to install our application’s deployable objects!

What is a Helm Install (Release)?

When you install a Helm Chart, you are creating an instance of that chart in Kubernetes. This instance is tied to what Helm calls a release name. You can install the exact same chart multiple times — likely with different values or in different namespaces — but create different instances of it simply by changing the release name.

There are multiple ways Helm can find the chart we want to install:

1. From a repository via a chart reference which will look in the named repository in its list of repositories for the chart. Format: <repo-name>/<chart-name>, e.g. local-development/db-application
2. From a repository via a repo URL and chart reference, e.g. helm install --repo https://charts.dkinternal.com my-release db-application (NOTE: It installs the latest stable version unless you specify the flag --version)
3. From a local packaged chart helm install mynginx ./nginx-1.2.3.tgz
4. From a local unpacked chart directory helm install mynginx ./nginx
5. From an absolute URL helm install mynginx https://example.com/charts/nginx-1.2.3.tgz

We’ll be using the first option with a chart reference since we installed it in our local Chart Museum repository.

Install The Chart

Hands-On Steps

Make sure you have the proper namespaces (we created these in the last workshop):

1. Run the Helm install command:

2. Now verify it installed properly:

Updating A Chart

In Helm, updating a release is called a Release Upgrade. You can make changes by altering the actual chart files or by simply using the helm upgrade command and passing different values while also increasing the chart version number

To do this, we’ll need first to make and publish some changes to our chart. We’ll make a change to increase our version number, and we’ll add an extra label.

Hands-On Steps

Let’s first see what labels we have right now:

1. Open db-application/templates/db-deployment.yaml and change the template.metedata.labels section to look like the snippet below(you’re only adding one new line beneath the call to the function with the value extra-label: helm-updated):

2. Now open db-application/Chart.yaml and change the version number to be 0.2.0:

3. Push the change to the repo and tell Helm to update its list:

4. Upgrade the chart in K8s:

5. Validate it upgraded it properly:

Rollback a Chart

You can always roll back to a previous version if anything looks wrong. Simply provide the release name (db-app in our case) and the revision number.

Notice it’s not the version number. That’s because you can force upgrade (and force publish) a chart with the same version number, despite passing overridden values or even YAML changes.

Hands-On Steps

Notice how a rollback creates a new revision. You don’t have one revision, but three! This is because Helm is keeping an audit trail to understand every change made to the deployed objects.

Uninstall a Chart

Remember in the last workshop how many different apply and delete commands we had to run? Let’s see how easy uninstalling a bunch of Kubernetes resources is with Helm. Despite our resources living across multiple namespaces, Helm we can uninstall a single command:

It’s important to note that Helm no longer has any history about our application. All revisions are gone, and the audit trail with them. What would you do if you wanted them temporarily uninstalled but wanted to keep that history? You can use a few things to do that:

1. Make sure your YAMLs have conditionals on them
2. Run an upgrade for the release and override the conditionals to be disabled, e.g. ingress.enabled=false (We’ll take a look at conditionals in a few steps)

That will create a new revision, or Helm “instance” in K8s, but with no resources at all (assuming you put that conditional on all your YAML templates).

Step 6. Use a Dependent Chart

How Dependencies Work

When you specify a dependency, it needs to be downloaded and installed into your chart before packaging. It does not get downloaded on the fly by Helm when installing.

Adding A Simple Dependency

We’ll start by making an empty chart that simply depends on our already created chart for MySQL.

Hands-On Steps

Make sure you’re in the base working directory (and not inside the db-application directory) and use Helm’s create command:

Just like before, let’s delete all the unneeded files:

• templates/deployment.yaml
• templates/hpa.yaml
• templates/ingress.yaml
• templates/NOTES.txt
• templates/service.yaml
• templates/serviceaccount.yaml
• templates/tests/test-connection.yaml

You should be left with just an empty templates and templates/tests directory, _helpers.tpl, .helmignore, Chart.yaml and values.yaml files.

1. Change the values.yaml to be empty for now
2. Change the Chart.yaml to look like this:

3. Now update your REST app for the helm dependency on the db-app (see below for why this is the only way)

4. Change values.yaml to look exactly like:

5. Now publish and install this application:

Passing Values to a Sub Chart

Notice in the values.yaml, we are using the exact name of the dependent chart (mysql-db-app) as a top-level property. We then put overrides for that chart underneath that property. Helm automatically passes those overrides through to the dependent chart. The dependent chart’s templates will get them without the chart name in the property path. For example, the sub chart mysql-db-app will see .Values.credentials.username instead of .Values.mysql-db-app.credentials.username.

Why We Can’t Add Dependencies (By Repo Name) When Pushing

The Helm push command is supposed to have the ability to also add dependencies on the fly, but due to this bug, it throws an error (it doesn’t remove the @ symbol). Once this is fixed, you will be able to use helm cm-push rest-application/ local-development --dependency-update. Note that you can add it that way if you use a URL.

Using Conditionals

We’re going to try two conditionals:

• Disabling the entire dependency
• Disabling the app-layer’s external name service.

Hands-On Steps

Disabling the Dependency Chart

Disabling the External Name

First, uninstall the empty rest-app:

helm uninstall rest-app

Let’s now install the chart, but without the external name:

Step 7. Adding the REST Application

Most of this is not introducing any new concepts, so we’ll just add and update the files from our REST application’s YAMLs in the last workshop.

Hands-On Steps

1. Add the file rest-application/templates/app-db-credentials.yaml and set its contents as:

2. Add the file rest-application/templates/app-rest-ingress.yaml and set its contents as:

3. Add the file rest-application/templates/app-rest-deployment.yaml and set its contents as:

4. Add the file rest-application/templates/app-rest-service.yaml and set its contents as:

5. Update values.yaml to be:

6. Update your Chart.yaml to be (notice the new version 0.2.0):

7. Package, push, and install it:

8. Add the domain to your hosts file (or update it) for the IP following the instructions in the last workshop (Section “Step 7.C: Add a “Domain” To Your Localhost”)

9. Visit http://draftkingsk8s.com/, and you should see “Got: first value!”

Congratulations, you have a fully working templated application packaged and deployed from Helm into a Kubernetes cluster!

Key Takeaways

There are a few important best practices to keep in mind when making Helm charts.

Naming

• Charts: lower case names are required but can have dashes
• Values: must be camelCased

Labels

Follow the best practices from the Helm team: https://helm.sh/docs/chart_best_practices/labels/.

YAML Structure

• Use a flat structure as it makes references easier to do safely, as well as conditionals and overrides
• e.g., if you’re referencing .Values.server.name, you need first to check that .Values.server exists. If you instead use a flat property, you don’t need to check anything: .Values.serverName

Overrides With Simple vs. Complex Structure

Do not do this:

Instead, do this:

Next Up

In Part 3 of the Kubernetes Workshop series, we’ll look at linting, schemas, integration testing, and health checks with Helm, Kubernetes, and a few other tools.

Thanks to Garrett Edwards for his help in testing, validation and the video walk-through!