Vault Kubernetes Auth Method for OpenShift

Kawsar Kamal
Jul 28, 2019 · 7 min read

HashiCorp Vault can enable a secure and automated authentication workflow for applications. Solving this challenge is an important part of adopting a DevSecOps framework which seeks to remove manual processes and allow operations in a low/zero-trust network. For more details, I recommend a talk by Armon Dadgar on building secure development lifecycles and zero-trust networks.

In Vault’s workflow, successful authentication is a pre-requisite step for applications (or humans) to store/retrieve secrets and perform cryptographic operations. These days apps are more heterogeneous than ever and run in many places. Hence, Vault has a range of Auth Methods to address any application running on any platform.

If your portfolio includes applications managed by an orchestration engine such as Kubernetes, below are some likely Auth Method candidates.

This guide provides a walkthrough for implementing the Kubernetes Auth Method, then testing it using the Vault CLI and an example Golang app. The application and corresponding scripts are available in the vault-openshift-demo git repo.

In a future blog post, I will cover how the Vault Agent tool can be used in conjunction with the above Auth Methods.

Pre-requisites

export VAULT_ADDR=https://vault.example.org:8200
export VAULT_TOKEN=<admin-or-root-token>
vault status
vault token lookup
  • Please ensure that the Vault server is able to reach the OpenShift cluster. You can test this by running the following commands on the Vault server.
# Substitute OpenShift API server endpoint
curl -kv https://<openshift_api_server>:<port>/healthz
  • Similarly, the OpenShift server should be able to reach Vault. Please test this by running the following commands from one of the OpenShift cluster nodes.
# Substitute Vault server's API endpoint
curl -kv https://<vault_server>:8200/v1/sys/health

Both of the above health checks should come back with “HTTP/2 200”.

Workflow

In the diagram below, steps 1–3 represent the authentication workflow. Upon successful completion, Vault will return a token with pre-configured policies attached. From step 4 onwards the application will use the token to retrieve secrets from Vault’s Key/Value secrets engine. We will implement these steps using CLI tools.

Note that an alternative approach for configuring Vault and Kubernetes is to use an Infrastructure-as-code (IaC) approach with Terraform’s Vault provider and Kubernetes provider. Roger Berlind has published a fully working example of this approach in terraform-guides repository under the k8s-services-openshift folder.

Vault Setup 1

Create an application policy and example secret. VAULT_ADDR and VAULT_TOKEN should be set from before.

# Create application policy for app1
cat <<EOF > app1-policy.hcl
path "secret/app1" {
capabilities = ["read", "list"]
}
path "database/creds/app1" {
capabilities = ["read", "list"]
}
EOF

OpenShift Setup

In this example, the OpenShift API server ca.crt file is placed at path: /etc/origin/master/ca.crt

# Create OC project and token reviewer JWT:
oc login -u system:admin
oc new-project vault-demo
oc projects
oc create sa vault-auth

Vault Setup 2

Use the commands below to enable and configure the Kubernetes Auth Method in Vault. Adjust the kubernetes_host parameter and path to ca.crt as needed. VAULT_ADDR and VAULT_TOKEN should be set from before.

sudo cp /etc/origin/master/ca.crt .
reviewer_jwt="$(cat reviewer_sa_jwt.txt)"

Test the authentication method using CLI

Use the commands below to log into Vault. Both CLI and curl examples are shown. A successful login will include a set of key/value pairs including a token and policies array. The policies array will contain “app1-policy” which will allow us to fetch the example secret. In the following command, substitute the <token-from-login> value with the token Vault returned. The VAULT_ADDR environment variable should be set from before.

# Login to app1 role - vault CLI
vault write "auth/ocp/login" role="app1-role" \
jwt="$(oc sa get-token app1)"

Using the same token, if you try to read “secret/app2” you will get an authorization failure. Similarly, if you try to read “secret/app1” with app2 JWT, you will get a “service account name not authorized” error.

# error: "permission denied"
VAULT_TOKEN="<token-from-login>" vault read secret/app2

Test the authentication method from an example Golang application

We will now deploy an example Golang application that uses Vault’s Go Client to login using the Kubernetes Auth Method. The application code and a corresponding Dockerfile is published in the vault-openshift-demo GitHub repo.

In the steps below, please adjust the “image” parameter if you pushed your own version of this application into a private Docker registry. The VAULT_ADDR environment variable should be set from before.

# Create a deployment.yaml file:

Once the “basic-example” pod is running, we can check application logs for authentication and secrets retrieval. If you can access the OpenShift administrative console, you can click on the “basic-example” Pod, then obtain the same output from the “Logs” tab.

oc get pods
# You will see a pod name starting with "basic-example-"
export pod=<pod_name>
oc logs ${pod}

The application reads the JWT for app1 service account from the path “/var/run/secrets/kubernetes.io/serviceaccount/token” and uses it to send a login request to Vault. You can see this in the application’s main.go file.

On the OpenShift administrative console’s Pod “Details” tab, you can see this secret JWT mounted:

Optionally, you can start a shell in this container using “oc” CLI (or use the console’s Pod “Terminal” tab) and issue an equivalent curl command. This may be helpful for troubleshooting.

oc exec ${pod} -it /bin/sh

Cleanup

Ensure that the environment variables VAULT_ADDR and VAULT_TOKEN are set properly, then run the commands below. Important: Please run these commands carefully to ensure you don’t accidentally delete

# Delete Auth Method and Secrets Engine
vault auth disable ocp
vault secrets disable secret

Summary

In this post, we have set up an automated and secure workflow for an OpenShift application to authenticate and fetch secrets from Vault. To achieve this, we set up a project and 2 service accounts in OpenShift, and configured the Kubernetes Auth Method in Vault. Then we tested the Auth Method using CLI and an example application.

Congratulations on getting this far! Thank you for reading and please leave any feedback or ideas for future expansion of this guide.

HashiCorp Solutions Engineering Blog

A Community Blog by the Solutions Engineers of HashiCorp and Invited Guests

Thanks to Jake Lundberg and Roger Berlind

Kawsar Kamal

Written by

HashiCorp Solutions Engineering Blog

A Community Blog by the Solutions Engineers of HashiCorp and Invited Guests

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade