Streamlining Application Secrets in Kubernetes with Pipelines

The “automagic” between Vault Secrets Engine, Kubernetes, and GitLab Pipelines.

Introduction

If you ever tried to modernize a large company’s process and infrastructure, you will probably come across some insufficient configuration/secrets management or lack thereof. Think of a .env file in the project root with shared database credentials in production, sound familiar? If not, then you are in a good place.

I have been both a web developer and currently a DevOps engineer. As a software developer, I only want to focus on my application and shouldn’t care about external tools. As a DevOps engineer, I want to automate the process and keep secrets secure. If secrets are compromised, mitigate risk as much as possible.

This is my opinionated way of deploying applications with rotating secrets into a scalable infrastructure automatically.

Requirements

There is a separation of responsibilities of developers to DevOps. In this context of configurations and secrets, I have a few requirements to follow:

• My application should read configuration and secret variables from a file. Keep it simple and straightforward.
• Only my application will go into a Dockerfile. No configuration management tools will be included in the Dockerfile.
• When a better secret management tool comes out, I should be able to swap with ease.

Tooling

Container orchestration:

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

CI/CD Pipelines

Gitlab Pipelines are the top-level component of continuous integration, delivery, and deployment.

Secrets and configuration management

Hashicorp Vault: The shift to managing secrets and protecting data in dynamic infrastructure.

Secrets and configuration retriever: confd (preference) or consul-template. I will show both implementations.

No logos to show here 🙁

confd is a lightweight configuration management tool focused on keeping local configuration files up-to-date using data.

consul-template: This project provides a convenient way to populate values from Consul into the file system using the consul-templatedaemon.

Mitigation of Risk

Path of Credential Retrieval

1. The Kubernetes service account JWT is passed to Vault’s Kubernetes Auth Method.
2. Once Vault validates the JWT via the token reviewer API, a Vault token with a TTL is returned. This vault token has the ability to generate new database credentials and renew its own TTL.
3. Vault token is used to generate new database credentials with a TTL and written to .env for application to consume.

Strategies

Progressing from secure to more secure in order:

One Container

The application and tool used to retrieve secrets are packaged into one container.

Pros: The application is unaware of any configuration management tool and reads off the .env file.

Cons: If the container is compromised — the database credentials, Vault address, Vault token, and service account JWT will also be compromised. Secondly, this violates one of the requirements. The Dockerfile is not just the application, it has an added tool to retrieve secrets.

App Container with Sidecar

This improves the One Container approach. The application and the tool used to retrieve secrets are separated. They only share a volume containing the credentials.

Pros: The application container only contains the application. If the application container is compromised, only the database credentials are at risk until the TTL expires. The Vault token and service account JWT are not compromised because they are only in the sidecar.

Cons: The sidecar still holds the service account JWT for no reason. If an attacker gets hold of the JWT, he/she can keep generating Vault tokens.

InitContainer, App Container, and Sidecar (preferable)

The initContainer would use the service account JWT to authenticate, pass the Vault token to a shared volume for the sidecar to consume, and die. Then the application container and sidecar container would startup. The sidecar container would have the Vault token and be responsible for retrieving the credentials and updating the shared volume once the TTL has expired.

Pros: Everything that is compromisable has a TTL.

This approach was originally demonstrated by Seth Vargo in his Vault Kubernetes Workshop.

Prerequisites

Vault needs to be up and running. It doesn’t matter what backend you use and doesn’t matter where your instance is located. I have Vault set up on GKE with an internal load balancer with a private DNS record. If you don’t have an instance, you can follow my write up for setting up Vault.

Deploying Vault with etcd backend in Kubernetes

You will need a Kubernetes cluster that can access your Vault instance. I don’t have a write up for that yet. My preference is to use Terraform to spin up a GKE cluster. Infrastructure as code is good!

Spin up a MySQL instance anywhere. As long as Vault and your cluster can hit it, you’re good. I use GCP’s CloudSQL and works great.

Configure Vault with database secrets engine, policy, and Kubernetes authentication method. You can follow my write up.

Vault: Kubernetes Auth and Database Secrets Engine

Lastly, a GitLab project. I am using a self-hosted GitLab but gitlab.com works the same. I will supply you with the source code.

Implementation

These are all of the pieces of the puzzle, the order doesn’t matter much. Once everything is in place, all pipeline runs will utilize the flow.

The Application

Feel free to use your own application, we’re only interested in the configuration file anyways. It is important to have the repository in GitLab. Here is a sample nodejs application that I wrote up to demonstrate the implementation:

jackalus / node mysql test

This project expects a .env file in the project root and it should contain the credentials for our database connection. Example:

MYSQL_URL=mysql://username:password@demodb.stage.jackalus.dev/mysql

GitLab Kubernetes Integration

Spin up your Kubernetes instance and configure your GitLab project to communicate to your cluster. Follow the GitLab docs for Adding an Existing Kubernetes Cluster. If you followed my guide for setting up Kubernetes Auth in Vault, ensure the namespace is demo.

Add in the base domain to your public domain.

GitLab offers managed applications to install into your cluster. You are not required to install any of these applications via GitLab, you can install Helm Tiller manually instead. I installed the following via GitLab:

• Helm Tiller — Package manager for Kubernetes.
• Ingress — Installs nginx ingress controller and requests for a LoadBalancer service. Wait for the ingress endpoint IP and point your DNS to the address. You may need to wait a few minutes for the DNS to update before the next application.
• Cert-Manager — Native Kubernets certification management controller.
• Prometheus — Open-source monitoring.

The Pipeline

We will utilize GitLab’s AutoDevOps and our project will not require a .gitlab-ci.yml. AutoDevOps is nothing more than a set of predefined templates and jobs according to an opinionated set of stages. If AutoDevops is not enabled by default, follow the GitLab Docs. At the time of this writing, the option is located here: Settings > CI/CD > Auto DevOps

AutoDevops uses CI/CD environment variables to enable/disable certain jobs. This can be found here: Settings > CI/CD and expand Variables. Let’s go ahead and disable everything we don’t need for now.

• TEST_DISABLED=1
• PERFORMANCE_DISABLED=1
• CODE_QUALITY_DISABLED=1
• CONTAINER_SCANNING_DISABLED=1
• DEPENDENCY_SCANNING_DISABLED=1
• DAST_DISABLED=1
• SAST_DISABLED=1
• LICENSE_MANAGEMENT_DISABLED=1
• POSTGRES_ENABLED=false

A complete list of the AutoDevops environment variables can be found here.

Custom auto-deploy-app

GitLab’s AutoDevOps uses a generic helm chart called auto-deploy-app. There are a few features the chart is missing which is the ability to add initContainers/sidecar containers, shared process namespace, and volume mounts. I do have a merge request for GitLab, but just doesn’t seem to be a use case for many. Go ahead and push for it if you find it useful.

added initContainers and additionalContainers (!2) · Merge Requests · GitLab.org / charts /…

In the meantime, I have a forked version with the needed functionality. To use my helm repository instead of the GitLab repository, add the following CI/CD variable:

• AUTO_DEVOPS_CHART_REPOSITORY=https://jackalus.gitlab.io/charts

For transparency, here is the custom chart:

jackalus / helm-charts / auto-deploy-app

… and my make-shift helm repository:

jackalus / charts

Helm Values

We will need to update some values of the auto-deploy-app chart. Instead a bunch of inline --set, we will add a file called helm_values.yaml and add the following CI/CD variable:

• HELM_UPGRADE_EXTRA_ARGS=-f helm_values.yaml

Here is the good stuff. Contents of helm_values.yaml.

consul-template vs confd

consul-template was created by Hashicorp. It only supports Consul’s KV store and Vault. confd was created by Kelsy Hightower. It supports way more services but cannot do more than one source in a configuration. The only implementation difference is in the sidecar. I will show you both. If a more thorough breakdown is needed, message me or leave a comment.

helm_values.yaml using consul-template

initContainers:
- name: vault-authenticator
  image: sethvargo/vault-kubernetes-authenticator:0.2.0
  imagePullPolicy: IfNotPresent
  volumeMounts:
  - name: vault-token
    mountPath: /var/run/secrets/vaultproject.io
  - name: vault-tls
    mountPath: /etc/vault/tls
  env:
  - name: VAULT_ADDR
    valueFrom:
      configMapKeyRef:
        name: vault
        key: vault_addr
  - name: VAULT_CACERT
    value: /etc/vault/tls/ca.pem
  - name: VAULT_ROLE
    value: demo
additionalContainers:
  - name: consul-template
    image: hashicorp/consul-template:0.20.0-light
    imagePullPolicy: IfNotPresent
    securityContext:
      capabilities:
        add: ['SYS_PTRACE']
    volumeMounts:
    - name: secrets
      mountPath: /secrets
    - name: vault-tls
      mountPath: /etc/vault/tls
    - name: vault-token
      mountPath: /var/run/secrets/vaultproject.io
    env:
    - name: VAULT_ADDR
      valueFrom:
        configMapKeyRef:
          name: vault
          key: vault_addr
    - name: VAULT_CACERT
      value: /etc/vault/tls/ca.pem
    - name: CT_LOCAL_CONFIG
      value: |
        vault {
          vault_agent_token_file = "/var/run/secrets/vaultproject.io/.vault-token"
          ssl {
            ca_cert = "/etc/vault/tls/ca.pem"
          }
          retry {
            backoff = "1s"
          }
        }
        template {
          contents = <<EOH
        {{- with secret "database/creds/demo-role" }}
        MYSQL_URL=mysql://{{ .Data.username }}:{{ .Data.password }}@demodb.stage.jackalus.dev/mysql
        {{ end }}
        EOH
          destination = "/secrets/.env"
          command     = "/bin/sh -c \"kill -HUP $(pidof node index.js) || true\""
        }
volumes:
- name: secrets
  emptyDir: {}
- name: vault-tls
  secret:
    secretName: vault-tls
- name: vault-token
  emptyDir:
    medium: Memory
application:
  command: [/bin/sh, "-c", "ln -s /secrets/.env /app/.env; node index.js"]
  volumeMounts:
  - name: secrets
    mountPath: /secrets/
service:
  internalPort: 8080
  externalPort: 8080
livenessProbe:
  path: version
readinessProbe:
  path: version
shareProcessNamespace: true
serviceAccountName: vault

helm_values.yaml using confd

initContainers:
- name: vault-authenticator
  image: sethvargo/vault-kubernetes-authenticator:0.2.0
  imagePullPolicy: IfNotPresent
  volumeMounts:
  - name: vault-token
    mountPath: /var/run/secrets/vaultproject.io
  - name: vault-tls
    mountPath: /etc/vault/tls
  env:
  - name: VAULT_ADDR
    valueFrom:
      configMapKeyRef:
        name: vault
        key: vault_addr
  - name: VAULT_CACERT
    value: /etc/vault/tls/ca.pem
  - name: VAULT_ROLE
    value: demo
additionalContainers:
  - name: confd
    image: jackalus/confd
    imagePullPolicy: Always
    securityContext:
      capabilities:
        add: ['SYS_PTRACE']
    command: ["/bin/sh"]
    args:
    - -c
    - |
        export VAULT_TOKEN=$(cat /var/run/secrets/vaultproject.io/.vault-token)
        echo "backend = \"vault\"
        auth_type = \"token\"
        auth_token = \"$VAULT_TOKEN\"
        interval = 600
        nodes = [
          \"$VAULT_ADDR\"
        ]" > /etc/confd/confd.toml &&
        echo '[template]
        src = "test.conf.tmpl"
        dest = "/secrets/.env"
        keys = [
            "/database/creds/demo-role",
        ]' > /etc/confd/conf.d/test.toml &&
        echo '{{ $data := json (getv "/database/creds/demo-role") }}
        MYSQL_URL=mysql://{{ $data.username }}:{{ $data.password }}@demo2db.stage.jackalus.dev/mysql'> /etc/confd/templates/test.conf.tmpl
        while sleep 300s; do vault token renew; done &
        confd
    volumeMounts:
    - name: secrets
      mountPath: /secrets
    - name: vault-tls
      mountPath: /etc/ssl/certs
    - name: vault-token
      mountPath: /var/run/secrets/vaultproject.io
    env:
    - name: VAULT_ADDR
      valueFrom:
        configMapKeyRef:
          name: vault
          key: vault_addr
    - name: VAULT_CACERT
      value: /etc/ssl/certs/ca.pem
volumes:
- name: secrets
  emptyDir: {}
- name: vault-tls
  secret:
    secretName: vault-tls
- name: vault-token
  emptyDir:
    medium: Memory
application:
  command: [/bin/sh, "-c", "ln -s /secrets/.env /app/.env; node index.js"]
  volumeMounts:
  - name: secrets
    mountPath: /secrets/
service:
  internalPort: 8080
  externalPort: 8080
livenessProbe:
  path: version
readinessProbe:
  path: version
shareProcessNamespace: true
serviceAccountName: vault

Note: I am using my forked version of confd with the Vault client binary. This is so I can do my token renewal within the same container. Alternatively, I can leave out the Vault client from the confd container and add a Vault token renewal container. I figure that would make it more confusing.

forked confd Dockerfile:

FROM golang:1.10.2-alpine

ARG CONFD_VERSION=0.16.0

ADD https://github.com/kelseyhightower/confd/archive/v${CONFD_VERSION}.tar.gz /tmp/

RUN apk add --no-cache \
    bzip2 \
    make && \
  mkdir -p /go/src/github.com/kelseyhightower/confd && \
  cd /go/src/github.com/kelseyhightower/confd && \
  tar --strip-components=1 -zxf /tmp/v${CONFD_VERSION}.tar.gz && \
  go install github.com/kelseyhightower/confd && \
  rm -rf /tmp/v${CONFD_VERSION}.tar.gz && \
  mkdir -p /etc/confd/conf.d /etc/confd/templates 

COPY --from=vault /bin/vault /bin/vault
CMD confd

What Do We Have?

The Kubernetes service account JWT is only accessible in the initContainer. Once a Vault token has been generated, the initContainer dies and the JWT cannot be compromised.

Each pod will have its own unique Vault token with expiration. Tokens are automatically renewed to ensure tokens live as long as the pod. Tokens are not exposed to the application container.

Each instance of the application will have its own set of database credentials. Credentials are rotated on a time interval. If credentials were somehow compromised, access is not indefinite. We know the pod that was compromised and can revoke the set of credentials.

Pipeline

Output

2019-08-23T00:16:43Z review-confd-pad63a-86c459744b-942jr confd[21]: INFO Target config /secrets/.env out of sync                                                                                                                          
2019-08-23T00:16:43Z review-confd-pad63a-86c459744b-942jr confd[21]: INFO Target config /secrets/.env has been updated                                                                                                                     
Key                                       Value                                                                                                                                                                                            
---                                       -----                                                                                                                                                                                            
token                                     s.VMUglVotKPB3xxjvstK8pLn5                                                                                                                                                                       
token_accessor                            qNXDV2bF8rdPvle23Cbg9KIR                                                                                                                                                                         
token_duration                            1h                                                                                                                                                                                               
token_renewable                           true                                                                                                                                                                                             
token_policies                            ["default" "demo-db-r"]                                                                                                                                                                          
identity_policies                         []                                                                                                                                                                                               
policies                                  ["demo-db-r"]                                                                                                                                                                                    
token_meta_service_account_secret_name    vault-token-xcbzv                                                                                                                                                                                
token_meta_service_account_uid            6f552115-c527-11e9-b42d-42010aa8012d                                                                                                                                                             
token_meta_role                           demo                                                                                                                                                                                             
token_meta_service_account_name           vault                                                                                                                                                                                            
token_meta_service_account_namespace      demo                                                                                                                                                      
2019-08-23T00:26:43Z review-confd-pad63a-86c459744b-942jr confd[21]: INFO /secrets/.env has md5sum 928696bacbd204577416b14afbad5c52 should be 0dd32ddc558166e3a50e91a5a412d04a                                                             
2019-08-23T00:26:43Z review-confd-pad63a-86c459744b-942jr confd[21]: INFO Target config /secrets/.env out of sync                                                                                                                          
2019-08-23T00:26:43Z review-confd-pad63a-86c459744b-942jr confd[21]: INFO Target config /secrets/.env has been updated

Check the application

The /mysql endpoint connects to the mysql database and runs the following query: SELECT @@version Hitting this endpoint proves we have a successful connection.

Next Steps

This write up is only meant to be a starting point. There are many improvements to be made here.

To secure this even further, we can throw in Istio and SPIFFE into the mix. This will only allow access to the database if your route has been prescribed even if you have the database credentials. If there is enough interest, I’ll do a write-up.