A Complete Guide to Deploying Elixir & Phoenix Applications on Kubernetes — Part 4: Secret Management

At Polyscribe, we use Elixir and Phoenix for our real-time collaboration and GraphQL API backends and Kubernetes for our deployment infrastructure. In this series, I walk through the setup we used from start to finish to create a system that supports the following:

• Automatic clustering for Elixir and Phoenix channels
• Auto-scaling to respond to spikes in demand
• Service discovery for microservices, including those in other frameworks like Node.js
• Maintaining the exact same environment between staging and production and easily deploying from staging to production
• Relatively easy to setup and manage

Other posts in this series — Part 1: Setting up Distillery, Part 2: Docker and Minikube, Part 3: Deploying to Kubernetes, Part 5: Clustering Elixir and Phoenix Channels

By now we have a configuration that deploys our app onto our Kubernetes cluster. Here’s the configuration as it is now:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: myapp-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: myapp
          image: myapp:release
          ports:
            - containerPort: 8000
          args: ["foreground"]
          env:
            - name: HOST
              value: "example.com"
            - name: SECRET_KEY_BASE
              value: "highlysecretkey"
            - name: DB_USERNAME
              value: "postgres"
            - name: DB_PASSWORD
              value: "postgres"
            - name: DB_NAME
              value: "myapp_dev"
            - name: DB_HOSTNAME
              value: "10.0.2.2"

One issue with our setup however is that our secrets — database password, secret key base used by phoenix to generate secret tokens and cookies, etc. — are visible in plain-text in our config file. This is a config file we’ll want to commit into our version control, so having our secrets in there for everyone to see isn’t a good idea. Instead, we’ll use Kubernetes Secrets to securely store our secrets and pass them to our app as environment variables.

There are a few ways to create secrets in Kubernetes, the simplest being using kubectl create secret from the command line. However, rather than keeping track of CLI invocations, I prefer to keep everything in configuration files so the process is easily repeatable and somewhat self-documenting. This requires a little more manual work but is still quite easy. Create a file called k8s/myapp-secrets.yaml with the following contents:

apiVersion: v1
kind: Secret
metadata:
  name: secrets
type: Opaque
data:
  secret_key_base: aGlnaGx5c2VjcmV0a2V5
  db_password: cG9zdGdyZXM=

As usual, we start the config file by with the apiVersion, kind, and some metadata giving this secret a name. type: Opaque indicates that this secret contains arbitrary data (as opposed to, for example, our Docker registry auth). For data, we simply specify a key-value map. Note that the values must be base64-encoded. For example, to convert our database password “postgres” to its base64 representation, I executed echo -n "postgres" | base64 from my command line and pasted the result into the config file.

Next we need to modify our deployment configuration to use our secrets. Here’s the new configuration (the changed lines are in bold):

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: myapp-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: myapp
          image: myapp:release
          ports:
            - containerPort: 8000
          args: ["foreground"]
          env:
            - name: HOST
              value: "example.com"
            - name: SECRET_KEY_BASE
              valueFrom:
                secretKeyRef:
                  name: secrets
                  key: secret_key_base
            - name: DB_USERNAME
              value: "postgres"
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: secrets
                  key: db_password
            - name: DB_NAME
              value: "myapp_dev"
            - name: DB_HOSTNAME
              value: "10.0.2.2"

For each of the environment variables we want to pull from our secret store, we tell Kubernetes that the value has to come from a secret. The name of the reference is the name we gave our secret in our secret configuration’s metadata, and we use the key from the key-value pairs we defined in our secret’s data section.

Let’s deploy this change. First, tell Kubernetes to create the secret with kubectl create -f k8s/myapp-secrets.yaml. Then, apply the new deployment configuration with kubectl apply -f k8s/myapp-deployment.yaml. Note that this won’t cause Kubernetes to redeploy your current pods — the change will only apply to the next time your pod is created. To force the change to occur immediately, you can use the Kubernetes dashboard to delete the current pods so Kubernetes will create new ones.

If you need to change the value of a secret, simply update myapp-secrets.yaml with the new value (base64-encoded of course) and apply the change with kubectl apply -f k8s/myapp-secrets.yaml. Again, keep in mind that the change won’t apply till the pods are recreated, so if you want to instantly reflect the change make sure you destroy your pods so Kubernetes creates new ones. You can also easily share your secrets with whoever needs access by sharing the myapp-secrets.yaml. Secrets can also be accessed by using kubectl get secret secrets -o yaml.

There’s more you can do with Kubernetes Secrets, including storing files (useful for SSL certs for example) and making secrets only accessible to a single pod. Take a look at the documentation when you want to learn more.

In the next part, we’ll take a look at how we can cluster our Elixir applications so we can use Phoenix channels and the rest of Elixir’s fantastic cross-node communication abilities.