Deploying on Kubernetes #6: Application Secrets

4 min readApr 8, 2018

This is the sixth in a series of blog posts that hope to detail the journey deploying a service on Kubernetes. It’s purpose is not to serve as a tutorial (there are many out there already), but rather to discuss some of the approaches we take.

Assumptions

To read this it’s expected that you’re familiar with Docker, and have perhaps played with building docker containers. Additionally, some experience with docker-compose is perhaps useful, though not immediately related.

Necessary Background

So far we’ve been able:

Secrets

There is some information that is inherently sensitive. When managing authentication applications must be able to uniquely prove that they’re authorised to access a given resource (username/password), or otherwise prove their identity (PKI). Should that information be compromised, the guarantees inherent in providing identity to a limited set of users are lost.

Kubernetes provides a primitive that is designed especially for secret management. It is very similar to ConfigMap as created previously, exception:

It’s a different object, so access can be limited via Role Based Access Control (RBAC)
The information is encoded in base64, so odd secret types (for example, pgp keys) can be embedded easily
It is possible to encrypt them such that only applications with the required identity can read them.

The secrets we need

In the previous post, we added a large amount of configuration to the fleet binary. However, we deliberately omitted secret information:

# templates/configmap.yaml:14-1814     mysql:
         address: {{ default "kolide-fleet-mysql:3306" .Values.fleet.mysql.address }}
         username: {{ default "kolide" .Values.fleet.mysql.username }}
         # Handled as a secret in an environment variable
         # password: kolide

This information should be handled as “safely” as possible. The configuration (represented as dot notation) is:

mysql.password
redis.password
auth.jwt_key

Reusing existing secrets

Both MySQL and Redis already have provision for managing secrets. Indeed, they automatically generate secrets when the applications are loaded. Unfortunately, it is a little difficult to reach those secrets. Additionally, there are problems with those secrets as time goes on — with each release, they are regenerated. This might be fine if something takes care of updating those secrets, but in doing preliminary testing this does not appear to be the case.

However, we can store these secrets in something like thevalues.yml file where they will reused consistently.

Stubbing Configuration

First, we need to stub the values for these secrets. Earlier, we added redis and mysql as dependencies to this chart. We are able to influence the configuration in those charts in our chart, simply by adding the variables to our charts values.yml and prefixing it with the appropriate package name:

# values.yml:6-12mysql:
  # This is a required value.
  mysqlPassword: ""redis:
  # This is a required value.
  redisPassword: ""

The passwords are currently empty. That’s fine — we don’t want to add the same passwords to the default chart, which we will later submit to kubernetes/charts for all to use. We’ll come to that shortly.

We now need to reference this configuration in our secret object:

# templates/secret.yml:1-11---
apiVersion: "v1"
kind: "Secret"
metadata:
  labels:
    app: {{ template "fleet.fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    heritage: "{{ .Release.Service }}"
    release: "{{ .Release.Name }}"
  name: {{ template "fleet.fullname" . }}
data:

We add the configuration as follows:

# templates/secret.yml:11-14data:
  fleet.mysql.username: {{ required "A valid MySQL Username is required" .Values.mysql.mysqlUser | b64enc }}
  fleet.mysql.password: {{ required "A valid MySQL Password is required" .Values.mysql.mysqlPassword | b64enc }}
  fleet.auth.jwt_key: {{ required "A valid JWT key is required" .Values.fleet.auth.jwt_key | b64enc }}

There’s a few things to note there:

All the secrets are wrapped in a required function. It allows us to denote to the user that this will automatically handled — please take care of it.
All secrets are base64 encoded, as denoted by the specification

Perfect! Now, we need to actually supply some secrets. Luckily, helm lets us do this with multiple values.yml files:

$ cat <<EOF > values.secret.yaml
---
mysql:
  mysqlUser: "kolide"
  mysqlPassword: $(pwgen 32 1)redis:
  redisPassword: $(pwgen 32 1)fleet:
  auth:
    jwt_key: $(pwgen 32 1)EOF

(Presuming you have the pwgen utility installed) This will create a file with the appropriate passwords. How you handle this is up to you.

Once the file is created, we can tell helm to use both the normal values.yml file and the additional values.secret.yml file as source values for this chart:

$ helm upgrade --install kolide-fleet --values values.yaml --values values.secret.yaml .

Awesome! Our values are being used… well, sortof. They’re in the configuration, but not being consumed yet by the application.

Consuming Configuration

Kolide/Fleet allows specifying configuration via environment variables. We’ll take advantage of that to inject our secret information.

We need to edit the deployment object:

# templates/deployment.yml:50-68      containers:
        - name: fleet
          env: 
            - name: "KOLIDE_AUTH_JWT_KEY"
              valueFrom:
              secretKeyRef:
                  name: {{ template "fleet.fullname" . }}
                  key: "fleet.auth.jwt_key"
            - name: "KOLIDE_MYSQL_USERNAME"
              valueFrom:
                secretKeyRef:
                  name: {{ template "fleet.fullname" . }}
                  key: "fleet.mysql.username"
            - name: "KOLIDE_MYSQL_PASSWORD"
              valueFrom:
                secretKeyRef:
                  name: {{ template "fleet.fullname" . }}
                  key: "fleet.mysql.password"
        image: {{ .Values.pod.fleet.image | quote }}

The configuration above injects the environment variables into the fleet container, which should be consumed by the fleet application.

In summary

That’s it! If we check the fleet logs, we can see it’s successfully able to connect to MySQL:

$ kubectl logs kolide-fleet-fleet-7c5f4999d7-9j6bt
Using config file:  /etc/fleet/config.yml
################################################################################
# ERROR:
#   Your Fleet database is not initialized. Fleet cannot start up.
#
#   Run `fleet prepare db` to initialize the database.
################################################################################

Hmm. Looks like it’s not setup. Let’s cover that in the next chapter:

https://medium.com/@andrewhowdencom/deploying-on-kubernetes-7-application-installation-f9ce876c51a4