Introduction to Vault to provide secret management in your Kubernetes cluster
One of the core Kubernetes resources is a Secret. However, these Secrets are not actually secure, as anyone with access to the cluster may be able to read and update the secret. This article introduces Vault into the cluster to securely manage secrets.
Vault
Before we install Vault, it is probably worth understanding what it is.
Vault is a secrets management application produced and maintained by Hashicorp. Secrets it can manage include:
- Key/Value pairs
- Passwords
- Certificates
- Encryption keys
It has the ability to manage the insertion of these secrets into applications, even when the application is not Vault aware. It can also automatically rotate credentials and can create dynamic, time limited credentials.
Management is by way of a web UI, a Command Line Interface (CLI) and/or API.
Hashicorp offer it as a free community version, a managed web-based service and as a fully supported enterprise solution. For this article we will use the free community version.
Deployment Options
Vault can be deployed in a number of ways:
- A standalone instance for development and exploration purposes
- A single instance backed by a Persistent Volume
- As a cluster of instances (normally 3 or 5 nodes)
If you have been following my articles, you would have a three node Kubernetes cluster. As Vault has an anti-affinity setup to ensure that all its instances are on worker nodes, it is not possible to install the 3 or 5 instance Vault cluster on 2 worker nodes. Although you can change the taints and tolerations to do so if you wish. In this article we will install a cluster with just 2 instances to show how it works.
Installing Vault
We will install Vault from Helm. If you need to install Helm, you can find instructions in this article.
First we add the required Helm repositories to our Helm configuration. Run this wherever you run kubectl (I run this on my k8s-master node):
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo updateYou can check you now have access the required chart with:
helm search repo hashicorp/vaultYou should see the repository we are going to use. At the time of writing this gave me:
NAME CHART VERSION APP VERSION DESCRIPTION
hashicorp/vault 0.27.0 1.15.2 Official HashiCorp Vault Chart Next we will create a namespace for Vault to hold our Vault resources:
kubectl create namespace vaultConnections
There are several connections to the Vault cluster, including a user interface, an API and inter-connections.
It is vital that all these connections are secured using TLS connections and appropriate certificates.
As TLS adds complexity to the deployment, we will first deploy without TLS and in a separate article, we will add TLS.
Configuring Vault
Before we install the Helm chart, there are changes that we need to make to configure the install for our environment. Create the following configuration file:
vault-config.yml
global:
enabled: true
tlsDisable: true
namespace: vault
ui:
enabled: true
serviceType: NodePort
serviceNodePort: 30802
externalIPs:
- 10.240.0.19
server:
dataStorage:
storageClass: nfs-client
ha:
enabled: true
replicas: 2
raft:
enabled: true
setNodeId: true
config: |
ui = true
cluster_name = "vault-integrated-storage"
storage "raft" {
path = "/vault/data/"
}
listener "tcp" {
address = "0.0.0.0:8200"
cluster_address = "0.0.0.0:8201"
tls_disable = "true"
}
service_registration "kubernetes" {}Let’s break this down into sections:
global:
enabled: true
tlsDisable: true
namespace: vault
This sets up global variables for the deployment that are used as defaults.
ui:
enabled: true
serviceType: NodePort
serviceNodePort: 30802
externalIPs:
- 10.240.0.19I would recommend you enable the User Interface (UI) so you can manage Vault through a browser.
There are two places you have to enable the UI. This is one. The other is in the HCL configuration section, which I will describe later.
In this section you enable the service that will provide access to the UI. It is created as a NodePort service on port 30802 (this is a value I have chosen, you may choose another). I have also attached this to my k8s-master, which is at address 10.240.0.19. You will have a different value for this.
server:
dataStorage:
storageClass: nfs-clientThis section tells Vault to use a storageClass of nfs-client. If you have been following my articles, you will know that this will mean that when the Persistent Volume Claims (PVCs) are created, a Persistent Volume will be automatically created within the Network File Systems (NFS) server. You can read more about NFS operators here
ha:
enabled: true
replicas: 2This section starts the definition of the High-Availability configuration. It is enabled (rather than being a standalone deployment) and there will be 2 replicas as I have two worker nodes.
raft:
enabled: true
setNodeId: true
config: |
ui = true
cluster_name = "vault-integrated-storage"
storage "raft" {
path = "/vault/data/"
}
listener "tcp" {
address = "0.0.0.0:8200"
cluster_address = "0.0.0.0:8201"
tls_disable = "true"
}
service_registration "kubernetes" {}This section manages the synchronisation of the integrated storage solution between the Vault instances using a protocol called Raft. Raft elects a leader and all writes to the leader are then logged to the other instances, distributing the storage across the nodes.
After enabling Raft, it sets the ID of each Raft instance based on the node name. The config section is defined as a multi-line value that is actually defined in HashiCorp Configuration Language (HCL). It configures the Vault application. The cluster_name is useful if you are reporting performance via Prometheus.
You can see that the UI is enabled.
If you do not enable this here, you will receive 404 page not found errors when you try to access the Vault UI.
After this, the Raft cluster is set up. It is given a cluster name and then adds the Raft location at /vault/data/.
The configuration then continues by setting up the listener at port 8200 (without TLS being enabled). This port actually serves both the API and the UI.
Finally, service_registration allows you to define how Pod and Namespace names are defined. {} indicates that the default will be used unless overridden.
Deploying Vault
Now we have configured Vault, we now need to deploy it. This is done using Helm with the following command:
helm install vault hashicorp/vault -f vault-config.yml -n vaultThis installs the Helm chart with the configuration file we just created.
It can take a few minutes to deploy, Check with:
kubectl get pods -n vaultYou will then see:
NAME READY STATUS RESTARTS AGE
vault-0 0/1 Running 0 8s
vault-1 0/1 Running 0 7s
vault-agent-injector-55748c487f-6bv2q 1/1 Running 0 11sNote that the status is Running but not ready (0/1). If they are pending there is a problem and you should look at the Pods with:
kubectl describe pod vault-0 -n vault
kubectl describe pod vault-1 -n vaultYou can also check the Services with:
kubectl get svc -n vaultWhich should get you something like:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
vault ClusterIP 10.109.110.229 <none> 8200/TCP,8201/TCP 8h
vault-active ClusterIP 10.97.237.232 <none> 8200/TCP,8201/TCP 8h
vault-agent-injector-svc ClusterIP 10.100.252.34 <none> 443/TCP 8h
vault-internal ClusterIP None <none> 8200/TCP,8201/TCP 8h
vault-standby ClusterIP 10.111.83.128 <none> 8200/TCP,8201/TCP 8h
vault-ui NodePort 10.110.144.217 <none> 8200:30802/TCP 8hYou will see the last Service is the UI. If you are connected to your cluster, for example, by a VPN, then you should be able to go to your browser and enter:
http://<node ip address>:30802/uiIf you get a 404 page not found then you may not have enabled the UI in the HCL config section.
If you have a UI, congratulations, you have installed Vault on your Kubernetes cluster.
Starting Vault
You may be wondering why, when we look at the Vault Pods, that none of them are ready. This is because a new Vault needs to be initialised and unsealed before it can be used.
Unsealing Vaults
Like any safe, you cannot get in unless it is first unlocked. When we talk about Vault, this is known as unsealing the Vault. If you lock the Vault, you are sealing it.
Vault encrypts everything. This ensures that, should anyone get access to the contents of the vault, say through the filesystem, they still cannot read the contents.
If everything is encrypted, you may be wondering how Vault decrypts it so we can use it. There are three levels of encryption.
- Your data is encrypted and the encryption key is stored in Vault’s keyring along with all other keys.
- The keyring is encrypted with a root key.
- The root encryption key is encrypted with the unseal key.
In order for us to access our data, we need the unseal key, which gives us access to the root key, which gives us access to, what I will call, the final key that decrypts our data!
It seems like a lot of keys but this is how it works.
When you unseal Vault, you give it the unseal key. This allows it to unlock the root key, which it then stores in memory. You are not given the root key. Now, whenever data is requested, Vault uses the root key to gain access to the final key, which it can then use to access your data.
Note that you only get access to the data if you have the right privileges.
Of course, if the service or server is restarted, it needs to be unsealed with the unseal key before it can be used. If the service has been restored from backup, it will need to be unsealed with the unseal key used originally when the back up was made.
If you are still with me, you will realise that our newly created Vault is sealed and we need the unseal key.
Now things get a little more complex. Although there are other options, for a production Vault service, you will probably be using a Shamir key as the unseal key.
Shamir Keys
Whilst the subject of Shamir keys is highly mathematical, their purpose is quite easy to understand. By the way, Shamir is the name of its founder, Adi Shamir.
Shamir keys is a set of key that unlock an encryption. However, not all the keys are required, only more than the threshold.
For example, a set of 5 Shamir keys may have a threshold of 3, which means that at least 3 need to be provided to unlock the encryption.
Vault requires you to decide on the number and threshold of Shamir keys that you are going to use to unseal the Vault.
This is the next step in the installation process.
Setting up Vault
Go to the user interface. You will now be asked if you want to join an existing Raft cluster or to create a new one.
Select Create a new Raft cluster and click Next.
Now select the number of Shamir keys and the threshold require to unseal the Vault. If you are working by yourself, 1 and 1 are acceptable. If you have multiple people who need to come together to unseal the Vault, choose appropriate numbers. I will select 1 and 1.
Click Initialize.
You will now be shown your initial Root Token (more about that shortly) and your list of Shamir keys. In the example above, there is just 1.
Click Download keys.
Keep these very secure. Anyone with access to this file has access to your Vault.
This file includes base64 versions of the keys which you will need if you are working from the Command Line Interface (CLI).
Click Continue to Unseal.
Now you will be asked for the threshold number of Shamir keys. In my case, I will enter the key I was given (not the token).
The Vault is now unsealed. Let’s have a look at our pods:
kubectl get pods -n vaultNot what you expected?
NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 12m
vault-1 0/1 Running 0 12m
vault-agent-injector-55748c487f-4jqfs 1/1 Running 0 12mThere is still one Pod that is not ready. When using Shamir keys, you need to unseal each and every instance in the cluster. As this is a manual process, it does not automate well. There are services that will unseal your Vault automatically for you. These are trusted services that rarely fail. We will unseal our remaining instance manually.
If you do not unseal an instance, it will not be able to take part in the cluster as it cannot access its underlying secrets.
As the UI only attaches to one instance, the others have to be done via the Command Line Interface (CLI).
First attach to the instance to be unsealed, eg:
kubectl exec -it vault-1 -n vault -- sh From here you can access your Vault server using the vault command, which acts as a client.
valut --helpThis will show you what you can do from the CLI. We will now unseal this Vault instance. First we must join this instance to our Raft cluster:
kubectl -n vault exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200If successful, you should see:
Key Value
--- -----
Joined trueNow we can unseal the Vault:
vault operator unsealYou will be asked to enter the required number of Shamir keys (1 in my instance). This must be in base64, which you can find in the json file you downloaded.
If successful, you will see this:
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.15.2
Build Date 2023-11-06T11:33:28Z
Storage Type raft
Cluster Name vault-integrated-storage
Cluster ID c7b6d160-a160-38c4-458e-c1ed449b712a
HA Enabled true
HA Cluster https://vault-0.vault-internal:8201
HA Mode standby
Active Node Address http://192.168.121.149:8200
Raft Committed Index 82
Raft Applied Index 82If it does not unseal, try again.
Repeat this for each sealed instance.
Log in to Vault
Now you have unsealed Vault, you can now start using it.
There are multiple ways that you can authenticate to Vault, too many to describe here. We will log in using an authentication token. At this point in time, there is only one token, the Root Token you were given earlier.
Enter it here.
Vault will now log you in. It will give you a warning about using the Root Token. The Root Token that was provided is really only designed to get you started. It is recommended that you cancel the root token after getting the Vault set up and then generate one-time use tokens as and when required.
Paths and Secrets Engines
Every secret in Vault has a path. Think of it as a path on your computer’s filesystem. There can be multiple secrets at any point in the path.
Each path resides inside a secrets engine. There are different types of secrets engine, including:
- Key/value pairs
- PKI certificates
- SSH keys
- Databases
And many more.
To see how this works, we will create a hello/world secret as a key/value pair.
In the UI, go to Secrets Engines. Click on Enable new engine. Choose kv under Generic and click Next.
For the path enter my-kv-secrets. This will form the top level leaf for your secrets. Click Enable Engine.
You should now see this:
You can now create a secret by clicking on Create secret. You will be asked to provide a path for the secret, within the secret engine, for example: my-secret-place. It can have many leaves if you want.
Enter hello for the key and world for the secret.
Click Save.
Congratulations, you have just created a secret in your Vault under your root token!
Adding Users
Doing things under the root token is not advisable. You should actually create a user.
Go to Access on the main menu. Click Enable new method. Choose Username & Password under Generic and click Next. Change the name of this authentication method, if required, and click Enable Method. Click View method >.
Now you can create users using the Username & Password authentication method. Click Create user. Enter a username and password and click Save.
You can now log out and log back in as that user. Each user is given a cubbyhole secrets engine where you can add your own secrets.
Notice that the main menu now has limited options as you are no longer logged in under the root token.
There is a lot more to be said about users, tokens and access control but fo rthis article I just wanted to get you up and running so you can explore.
Starting Again
Just a final note on what happens if you want to start again. If you simply delete and reinstall Vault, it will assume that you are accessing the original Vault and will ask you for the unseal keys.
If you truely want to start from scratch (for instance when you want to explore different features), you need to do the following:
helm delete vault -n vault
kubectl get pvc -n vault
...
kubectl delete pvc data-vault-0 -n vault
kubectl delete pvc data-vault-1 -n vault
...The first line removes the Vault Pods. This does not delete the PVCs, which must then all be deleted individually. If you do not do this, then when Vault is reinstalled, it will pick up from where you left off!
Now reinstall with:
helm install vault hashicorp/vault -f vault-config.yml -n vaultYou will now need to initial and unseal your new Vault.
Summary
In this article we looked at waht Vault is and why you might use it. We then used Helm to install a 2 instance cluster on our 3 node Kubernetes cluster, crating the necessary configuration to create an HA cluster backed by a Raft managed NFS storage layer.
We enabled the UI so we could start the installation and unseal process for our new Vault. Following on from that, we unsealed our other instance using the CLI.
Finally we looked at creating secrets and users.
If you found this article of interest, please give me a clap as that helps me identify what people find useful and what future articles I should write. If you have any suggestions, please add them in the comments section.
