Auto-unseal Vault using GCP Cloud KMS

When a Vault server is started, it starts in a sealed state. In this state, Vault is configured to know where and how to access the physical storage, but doesn’t know how to decrypt any of it. Unsealing is the process of obtaining the plaintext root key necessary to read the decryption key to decrypt the data, allowing access to the Vault.

Why Unsealing is Required?

The data stored by Vault is encrypted. Vault needs the encryption key in order to decrypt the data. The encryption key is also stored with the data (in the keyring), but encrypted with another encryption key known as the root key.

Therefore, to decrypt the data, Vault must decrypt the encryption key which requires the root key. Unsealing is the process of getting access to this root key. The root key is stored alongside all other Vault data, but is encrypted by yet another mechanism: the unseal key.

Shamir seals

The default Vault config uses a Shamir seal. Instead of distributing the unseal key as a single key to an operator, Vault uses an algorithm known as Shamir’s Secret Sharing to split the key into shares. A certain threshold of shares is required to reconstruct the unseal key, which is then used to decrypt the root key.

This is the unseal process: the shares are added one at a time (in any order) until enough shares are present to reconstruct the key and decrypt the root key.

Unsealing

The unseal process is done by running vault operator unseal or via the API. This process is stateful: each key can be entered via multiple mechanisms from multiple client machines and it will work. This allows each shares of the root key to be on a distinct client machine for better security.

Note that when using the Shamir seal with multiple nodes, each node must be unsealed with the required threshold of shares. Partial unsealing of each node is not distributed across the cluster.

Once a Vault node is unsealed, it remains unsealed until one of these things happens:

1. It is resealed via the API.
2. The server is restarted.
3. Vault’s storage layer encounters an unrecoverable error.

Auto Unseal

Auto Unseal was developed to aid in reducing the operational complexity of keeping the unseal key secure. This feature delegates the responsibility of securing the unseal key from users to a trusted device or service. At startup Vault will connect to the device or service implementing the seal and ask it to decrypt the root key Vault read from storage.

There are certain operations in Vault besides unsealing that require a quorum of users to perform, e.g. generating a root token. When using a Shamir seal the unseal keys must be provided to authorize these operations. When using Auto Unseal these operations require recovery keys instead.

Just as the initialization process with a Shamir seal yields unseal keys, initializing with an Auto Unseal yields recovery keys.

It is still possible to seal a Vault node using the API. In this case Vault will remain sealed until restarted, or the unseal API is used, which with Auto Unseal requires the recovery key fragments instead of the unseal key fragments that would be provided with Shamir. The process remains the same.

Recovery Key

When Vault is initialized while using an HSM or KMS, rather than unseal keys being returned to the operator, recovery keys are returned. These are generated from an internal recovery key that is split via Shamir’s Secret Sharing, similar to Vault’s treatment of unseal keys when running without an HSM or KMS.

Auto-Unseal via KMS on GCP

Vault supports opt-in automatic unsealing via cloud technologies: AliCloud KMS, AWS KMS, Azure Key Vault, Google Cloud KMS, and OCI KMS. This feature enables operators to delegate the unsealing process to trusted cloud providers to ease operations in the event of partial failure and to aid in the creation of new or ephemeral clusters.

This blog demonstrates an example of how to use Terraform to provision an instance that can utilize an encryption key from GCP Cloud KMS to unseal Vault.

Provision the Cloud Resources

1. Clone the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.

$ git clone https://github.com/hashicorp/vault-guides.git

2. Be sure to set your working directory to where the /operations/gcp-kms-unseal folder is located.

$ cd vault-guides/operations/gcp-kms-unseal

The working directory should contain the provided Terraform files.

$ tree
.
├── README.md
├── main.tf
├── terraform.tfvars.example
├── variables.tf
└── versions.tf

The main.tf generates the following:

Service account with Cloud KMS IAM to be used by Compute Engine instances

Compute Engine instance with Vault installed

(Optional) Cloud KMS key ring and crypto key

3. Provide necessary GCP account information in the terraform.tfvars.example and save it as terraform.tfvars. Overwrite the default variable values (variables.tf) as needed.

Example:

terraform.tfvars

gcloud-project = "gcloud-vault-test"
account_file_path = "/Users/james/gcp/gcloud-vault-test.json"

gcloud-region = "us-central1"
gcloud-zone = "us-central1-c"
key_ring = "my-key-ring-name"
crypto_key = "my-crypto-key-name"
# keyring_location = "global"

4. Perform a terraform init to pull down the necessary provider resources.

$ terraform init

5. Perform terraform plan to verify your changes and the resources that will be created.

$ terraform plan

6. If all looks good, perform a terraform apply to provision the resources.

$ terraform apply -auto-approve
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Test the Auto-unseal Feature

SSH into the generated compute instance.

To verify that Vault has been installed, run vault status command.

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       gcpckms
Initialized              false
Sealed                   true
Total Recovery Shares    0
Threshold                0
Unseal Progress          0/0
Unseal Nonce             n/a
Version                  1.8.1
Storage Type             file
HA Enabled               false

Notice that Initialized is false.

Run the vault operator init command to initialize the Vault server.

$ vault operator init

Example output:

Recovery Key 1: iz1XWxe4CM+wrOGqRCx8ex8kB2XvGJEdfjhXFC+MA6Rc
Recovery Key 2: rKZETr6IAy686IxfO3ZBKXPDAOkkwkpSepIME+bjeUT7
Recovery Key 3: 4XA/KJqFOm+jzbBkKQuRVePEYPrQe3H3TmFVmdlUjRFv
Recovery Key 4: lfnaYoZufP0uhooO3mHDAKGNZB5HLP9HYYb+LAfKkUmd
Recovery Key 5: L169hHj3DMpphGsOnS8TEz3Febvdx3vsG3Xr8kGWdUtW

Initial Root Token: s.AWnDagUkKNNbvkENiL72wysn
Success! Vault is initialized
Recovery key initialized with 5 key shares and a key threshold of 3. Please
securely distribute the key shares printed above.

After a successful initialization, check the Vault server status.

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.8.1
Storage Type             file
Cluster Name             vault-cluster-23602eff
Cluster ID               0068b53a-42df-73ee-784b-d6753f08ced7
HA Enabled               false

Notice that the Vault server is already unsealed (Sealed is false).

In the service log, you should find a trace where GCP KMS key is being fetched to unseal the Vault server.

$ sudo journalctl --no-pager -u vault

 ==> Vault server configuration:
       GCP KMS Crypto Key: vault-test
         GCP KMS Key Ring: test
          GCP KMS Project: gcloud-vault-test
           GCP KMS Region: global
                Seal Type: gcpckms
                      Cgo: disabled
               Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_re
quest_size: "33554432", tls: "disabled")
                Log Level: (not set)
                    Mlock: supported: true, enabled: false
                  Storage: file
                  Version: Vault v1.8.1
              Version Sha: c19cef14891751a23eaa9b41fd456d1f99e7e856
 ==> Vault server started! Log data will stream in below:
...

Restart the Vault server to ensure that Vault server gets automatically unsealed.

$ sudo systemctl restart vault

Verify the Vault status.

$ vault status

Review the Vault configuration file (/test/vault/config.hcl).

$ cat /test/vault/config.hcl

The configuration file looks as below.

storage "file" {
  path = "/opt/vault"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = "true"
}

seal "gcpckms" {
  project     = "gcloud-vault-test"
  region      = "global"
  key_ring    = "test"
  crypto_key  = "vault-test"
}

disable_mlock = true

Notice the Vault configuration file defines the gcpckms stanza which sets the GCP Cloud KMS key ring name, crypto key name, and its location as well as your GCP project ID.

Rotating the Unseal Key

When Vault is sealed with Shamir’ keys, execute the vault operator rekey command to generate a new set of unseal keys. With Auto-unseal enabled, you can simply rotate the Cloud KMS key used to unseal Vault. One of the benefits of using Cloud KMS is its automatic key rotation feature which eliminates the need for a manual operation.

Automatic rotation

To enable automatic rotation of a key, set the rotation schedule using the gcloud command-line tool:

$ gcloud kms keys update <KEY_NAME> \
         --location <LOCATION> \
         --keyring <KEYRING_NAME> \
         --rotation-period <ROTATION_PERIOD> \
         --next-rotation-time <NEXT_ROTATION_TIME>

Alternatively, you can manage it through GCP Console.

1. From the navigation menu, select Security > Cryptographic Keys.
2. Select the key ring used for Vault Auto-unseal.
3. Select the key to rotate.
4. Select EDIT ROTATION PERIOD.

5. Select the desired rotation period, and then click SAVE.

Manual rotation

To create a new key version and make it primary using the gcloud command-line tool:

$ gcloud kms keys versions create --location <LOCATION> \
         --keyring <KEYRING_NAME> \
         --key <KEY_NAME> --primary

Using the GCP Web Console, simply select ROTATE.

Clean Up

Once completed, execute the terraform destroy commands to clean up the cloud resources.

$ terraform apply -destroy -auto-approve

You can delete the Terraform state files.

$ rm -rf .terraform terraform.tfstate*