Automated Kubernetes secret injection with 1Password Secret Automation and HashiCorp Vault
Introduction
Riskfuel trains neural network models on datasets consisting of hundreds of millions of data points. Running at such a scale presents its own set of unique challenges. We have more and more pods which need to access tokens for storage, metrics, you name it. Previously, we already have our secrets in 1password but to use them in deployments, we had to manually include them as plaintext in our YAMLs. This slowed our developers’ efficiency and posed a potential security flaw.
In April 2021, 1password rolled out its Secret Automation workflow to address its CLI limited capability in distributed systems. This workflow includes plugins into backend systems like Hashicorp Vault, Kubernetes operator, Ansible collection and more. This allows Riskfuel to expand our existing usage of 1password and utilize it as a single source of truth for our Hashicorp Vault secrets. We are now able to automatically inject secrets into our deployments with ease.
There is documentation for this plugin with a manual Vault server workflow. This blog will be focusing a holistic deployment of the 1password-Vault plugin in Kubernetes with Helm.
For our scope, I’m going to assume the following:
- You have a 1password business account
- You have a Kubernetes cluster ready
- You are familiar with Vault, Kubernetes and Helm CLI
Getting Started
Step 1. Obtain 1password-credentials.json and access token
Each team plan will get 3 free credits for Secret Automation with a subscription for more.
Only an owner or admin of the 1password business account can set up Secret Automation and obtain the configuration json and access token. Follow the on screen instruction here to do so.
Step 2. Deploy Connect server
If you have used Helm before, this step is straightforward; use the credentials file you just obtained.
helm repo add 1password https://1password.github.io/connect-helm-charts/ helm install connect 1password/connect --set-file connect.credentials=1password-credentials.json
Step 3. Configure plugin in Vault’s Helm chart
We want to bake in this plugin, so let’s modify Vault’s Helm chart. We’re using /usr/local/libexec/vault
in this example but this would be the path to your plugin binaries.
- If you are using vault-helm default chart, in your
values.yaml
understandalone
orha
depending on your mode, underconfig
, add:
plugin_directory = “/usr/local/libexec/vault”
- If you have a slimmed down vault Helm chart like we do, you would want to add this to the
server-config-configmap.yaml
underextraconfig-from-values.hcl
Step 4. Add plugin installation initContainer
We will install the plugin binaries and designate a file to be the config file later, via an initContainer
. Don’t forget to modify the release version to match with your system and create a volume
match the volumeMount
here. Similar from above:
- If you are using vault-helm default chart, in the
extraInitContainers
in thevalues.yaml
file, add:
- If you are using your own Helm chart, add this under
initContainer
inserver-statefulset.yaml
.
Step 5. Configure plugin inside Vault pod
Once the changes above are made, deploy the chart and exec
into the vault-0
pod. We will then configure Vault for sidecar injection with k8s auth, register and enable the plugin.
Step 6. Insert access tokens
Staying inside the vault-0
pod, we now need to configure the plugin with the access token from step 1 to be able to make API request to the connect server. Let’s add to the /usr/local/libexec/vault/op-connect-config.json
file we created in the initContainer
the following:
{
"op_connect_host": "http://<IP_address_of_Connect_server_step_2>:8080",
"op_connect_token": "<access_token_from_step_1>"
}
Be mindful of the http
and port number here, it is important. Then write this file to Vault.
vault write op/config @op-connect-config.json
Step 7. Test Connection
Now that the plugin is set up, we should be able to send and retrieve data from 1password.
- Get a list of Vault your token has access to:
vault list op/vaults
- List items in a 1password vault:
vault list op/vaults/<1PW_vault_name>/items
- Read item:
vault read op/vaults/<1PW_vault_name>/items/<item_name>
Step 8. Deployment Usage
Before we can inject these secrets into our deployment, please ensure that your Vault policy is tied to your 1password path (i.e op/vaults/<1PW_vault_name>/items/*
)and Vault role is configured with the proper service account and namespace.
In your deployment.yaml
, under spec
, add the annotation:
spec:
...
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/role: '<Vault_role>'
vault.hashicorp.com/agent-inject-secret-config: 'op/vaults/<1PW_vault_name>/items/<item_name>'
Step 9. Profit
If you made it this far, congratulations! Once the deployment with annotation is running, you can exec
into the pod and go to /vault/secret/config
to see the injected contents. You can also leverage the Consul template annotation to have your secrets as environment variables.
Voila!
Conclusion
While the product is in its infancy, it boasts a powerful pipeline to connect 1password’s reliability with Hashicorp Vault’s infrastructure integration.
If you are interested in furthering your secret workflow, I also recommend checking out IBM’s argocd-vault-plugin. This allows us to streamline Kubernetes secrets by setting desired secrets states and have ArgoCD manage them, without revealing the value of these secrets in Github.
A small plug, Riskfuel is innovating the capital markets industry using cutting-edge machine learning and hardware. If you like solving problems like this, check out our careers page.
That’s it! Thanks for reading :)