Automated Kubernetes secret injection with 1Password Secret Automation and HashiCorp Vault

Introduction

Riskfuel trains neural network models on datasets consisting of hundreds of millions of data points. Running at such a scale presents its own set of unique challenges. We have more and more pods which need to access tokens for storage, metrics, you name it. Previously, we already have our secrets in 1password but to use them in deployments, we had to manually include them as plaintext in our YAMLs. This slowed our developers’ efficiency and posed a potential security flaw.

In April 2021, 1password rolled out its Secret Automation workflow to address its CLI limited capability in distributed systems. This workflow includes plugins into backend systems like Hashicorp Vault, Kubernetes operator, Ansible collection and more. This allows Riskfuel to expand our existing usage of 1password and utilize it as a single source of truth for our Hashicorp Vault secrets. We are now able to automatically inject secrets into our deployments with ease.

There is documentation for this plugin with a manual Vault server workflow. This blog will be focusing a holistic deployment of the 1password-Vault plugin in Kubernetes with Helm.

For our scope, I’m going to assume the following:

• You have a 1password business account
• You have a Kubernetes cluster ready
• You are familiar with Vault, Kubernetes and Helm CLI

Getting Started

Overview

Step 1. Obtain 1password-credentials.json and access token

Each team plan will get 3 free credits for Secret Automation with a subscription for more.

Only an owner or admin of the 1password business account can set up Secret Automation and obtain the configuration json and access token. Follow the on screen instruction here to do so.

Step 2. Deploy Connect server

If you have used Helm before, this step is straightforward; use the credentials file you just obtained.

helm repo add 1password https://1password.github.io/connect-helm-charts/ 
helm install connect 1password/connect --set-file connect.credentials=1password-credentials.json

Step 3. Configure plugin in Vault’s Helm chart

We want to bake in this plugin, so let’s modify Vault’s Helm chart. We’re using /usr/local/libexec/vault in this example but this would be the path to your plugin binaries.

• If you are using vault-helm default chart, in your values.yaml under standalone or ha depending on your mode, under config, add:

plugin_directory = “/usr/local/libexec/vault”

• If you have a slimmed down vault Helm chart like we do, you would want to add this to the server-config-configmap.yaml under extraconfig-from-values.hcl

Step 4. Add plugin installation initContainer

We will install the plugin binaries and designate a file to be the config file later, via an initContainer . Don’t forget to modify the release version to match with your system and create a volume match the volumeMount here. Similar from above:

• If you are using vault-helm default chart, in the extraInitContainers in the values.yaml file, add:

• If you are using your own Helm chart, add this under initContainer in server-statefulset.yaml.

Step 5. Configure plugin inside Vault pod

Once the changes above are made, deploy the chart and exec into the vault-0 pod. We will then configure Vault for sidecar injection with k8s auth, register and enable the plugin.

Step 6. Insert access tokens

Staying inside the vault-0 pod, we now need to configure the plugin with the access token from step 1 to be able to make API request to the connect server. Let’s add to the /usr/local/libexec/vault/op-connect-config.json file we created in the initContainer the following:

{
"op_connect_host": "http://<IP_address_of_Connect_server_step_2>:8080",
"op_connect_token": "<access_token_from_step_1>"
}

Be mindful of the http and port number here, it is important. Then write this file to Vault.

vault write op/config @op-connect-config.json

Step 7. Test Connection

Now that the plugin is set up, we should be able to send and retrieve data from 1password.

• Get a list of Vault your token has access to: vault list op/vaults
• List items in a 1password vault: vault list op/vaults/<1PW_vault_name>/items
• Read item: vault read op/vaults/<1PW_vault_name>/items/<item_name>

Step 8. Deployment Usage

Before we can inject these secrets into our deployment, please ensure that your Vault policy is tied to your 1password path (i.e op/vaults/<1PW_vault_name>/items/*)and Vault role is configured with the proper service account and namespace.

In your deployment.yaml , under spec, add the annotation:

spec:
  ...
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: '<Vault_role>'
        vault.hashicorp.com/agent-inject-secret-config: 'op/vaults/<1PW_vault_name>/items/<item_name>'

Step 9. Profit

If you made it this far, congratulations! Once the deployment with annotation is running, you can exec into the pod and go to /vault/secret/config to see the injected contents. You can also leverage the Consul template annotation to have your secrets as environment variables.

Voila!

Conclusion

While the product is in its infancy, it boasts a powerful pipeline to connect 1password’s reliability with Hashicorp Vault’s infrastructure integration.

If you are interested in furthering your secret workflow, I also recommend checking out IBM’s argocd-vault-plugin. This allows us to streamline Kubernetes secrets by setting desired secrets states and have ArgoCD manage them, without revealing the value of these secrets in Github.

A small plug, Riskfuel is innovating the capital markets industry using cutting-edge machine learning and hardware. If you like solving problems like this, check out our careers page.

That’s it! Thanks for reading :)

Reference

Deploy a 1Password Connect server on Kubernetes

GitHub - 1Password/vault-plugin-secrets-onepassword: Hashicorp Vault plugin that allows for the…

Injecting Secrets into Kubernetes Pods via Vault Agent Containers | Vault - HashiCorp Learn