HashiCorp Vault as an external key manager for Cloud Volumes ONTAP

How HashiCorp Vault can secure Data at Rest on enterprise storage in the Cloud. This blog post will provide a Terraform demo setup for testing.

Published in

HashiCorp Solutions Engineering Blog

8 min readApr 18, 2023

Prerequisites: HashiCorp Vault Enterprise License, Cloud Volumes ONTAP License, NetApp Support Site Credentials and Terraform binary on your local machine for executing Terraform Code.

Organizations adopting cloud data storage face challenges in securing data, particularly in managing encryption keys for cloud-hosted data. This is crucial when you must comply with data protection regulations. This blog post discusses how HashiCorp Vault can help you in addressing data-at-rest encryption in cloud data storage environments and provide you with a setup for testing.

The Root of Trust

In regulated industries such as healthcare, finance and government, ensuring the secure storage and management of encryption keys is crucial for compliance with data protection regulations like HIPAA, PCI-DSS, and GDPR. Encryption keys must be securely stored and accessed only by authorized target systems, with a root of trust serving as the foundational element for secure systems. When it comes to securing storage systems like NetApp ONTAP, it’s crucial not only to manage external keys outside of the storage system, but also to maintain a root of trust within your own data center. HashiCorp Vault enables organizations to achieve these goals in a standardized way by providing KMIP Integration and HSM support.

KMIP (Key Management Interoperability Protocol) is an industry-standard protocol that enables secure communication between a key management system and a target system e.g. NetApp’s Cloud Volumes ONTAP. It provides a standardized way for key management systems to communicate with different target systems. In many cases, organizations need to perform cryptographic operations for their services and applications, such as transparent database encryption or full disk encryption. To manage encryption keys securely, you have to delegate the key management task to an external provider using KMIP. This tutorial explains how to configure the KMIP Engine in HashiCorp Vault, manage encryption keys and integrate with other systems using KMIP protocol.

Cloud Volumes ONTAP

NetApp’s Cloud Volumes ONTAP is a cloud-based storage solution that allows customers to manage their data in the cloud similarly to traditional on-premises data center storage. It is software-based and offers a range of data protection capabilities, including snapshots, backup and recovery as well as easy data migration. We will focus on the Storage Encryption capabilities. NetApp Aggregate Encryption (NAE) & NetApp Volume Encryption (NVE) are software-based and enable FIPS 140–2 compliant data-at-rest encryption of aggregates and volumes. It is worth noting that each Cloud Volumes ONTAP environment has its own Storage Virtual Machine (SVM), which appears as a single dedicated storage server to user clients and has its own set of encryption keys for encrypting its aggregates and volumes.

Unlocking HashiCorp Vault as external Key Manager

One solution to the described challenges is to use HashiCorp Vault as an external key manager for NetApp’s Cloud Volumes ONTAP (CVO). HashiCorp Vault is a popular tool used by enterprises to secure sensitive information such as API keys, token, and encryption keys. While cloud storage systems like Cloud Volumes ONTAP have built-in key managers, compliance regulations often require organizations to store their keys separately from the storage system. That’s where HashiCorp Vault comes in, providing a centralized and secure location for managing and protecting sensitive secrets. When integrating with NetApp’s Cloud Volumes ONTAP, HashiCorp Vault can be used as a KMIP Server. Vault can also be used in a multi-tenant environment, where different teams or customers can have their own isolated encryption environments, providing flexibility and scalability.

Architecture Deployment

For demonstration purposes, Vault is deployed as a single node setup. However, in an Enterprise setup a Vault cluster must be deployed using a highly available and fault-tolerant configuration. This recommended architecture improves data security and maintains continuous service of your encrypted volumes in the event of node failures. It requires a HashiCorp Vault Enterprise License and NetApp Cloud Volumes ONTAP PayGo License.

1. Configuring your Vault

During Terraform apply the scripts, vault_setup.sh and vault_kmip.sh will be copied to the remote host. The vault_setup.sh installs and configures Vault on an Amazon Linux machine. It installs the required dependencies, sets up the necessary directories, and creates the configuration file. In the configuration file, Vault is configured to use Integrated Storage as the Storage Backend with a listener on port 8200.

NOTE: The enterprise key for HashiCorp Vault must be supplied for Vault configuration. This enables testing and configuration of KMIP Secrets Engine, an enterprise-level advanced data protection feature.

The vault_setup.sh script contains a systemd service, sets the necessary permissions, and starts the service on boot.

#!/bin/bash
… 
$ echo "here_vault_license_key" | sudo tee /etc/vault.d/license.hclic


$ sudo cat << EOF > /etc/systemd/system/vault.service
[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
…
[Service]
User=ec2-user
Group=ec2-user
… 
EOF

$ sudo systemctl enable vault.service --now

Initialize Vault, write output to a file then unseal Vault with the keys from vault.txt.

vault operator init -key-threshold=1 -key-shares=1 -format=json > vault.txt
vault operator unseal $(jq -r '.unseal_keys_b64[0]' vault.txt)

export VAULT_TOKEN=$(cat token.txt)
vault login $(jq -r .root_token vault.txt)

This will authenticate you into Vault as the root user, allowing you to access and configure the KMIP Secrets Engine.

Note: The root user is used for demonstration purposes.

2. Vault KMIP Setup

The kmip-setup.sh script will be automatically provided on the vault instance. This script will help you to set up HashiCorp Vault as a KMIP server. Run the script and execute the following command. This command generates the client certificate and writes the generated serial number to the credential_cvo.json file.

vault write -format=json kmip/scope/cvo/role/administration/credential/generate format=pem > credential_cvo.json

The following command uses jq to extract the serial number of the CVO key from the credential_cvo.json file and passes it as an argument to the vault read command.

vault read kmip/scope/cvo/role/administration/credential/lookup serial_number=$(jq -r '.serial_number' credential_cvo.json)

The output of the command includes the necessary certificate chain in the correct order to ensure successful configuration of your external Key Manager for NetApp’s Cloud Volumes ONTAP. For more information on generating client certificates using the KMIP Secrets Engine in Vault, refer to the official Vault documentation here.

The certificate order from top to bottom is as follows:

Root CA
Intermediate CA
Client CA
Root CA
Intermediate CA

3. Configure external Key Management Server on Cloud Volumes ONTAP

Since CVO is behind a private IP and not directly accessible from the public internet, access it through a jump host. Connect to the jump host via SSH and access the CVO instance.

This setup may not be suitable for every organization, as it depends on the specific requirements and network architecture. Some may have different security measures in place, such as using a VPN or dedicated access servers, to secure private network access. If you are interested in how to access remote hosts without storing SSH keys and securing private keys from outside of your enterprise network, you should check out HashiCorp Boundary.

3.1. Installing the Client Certificate

svm:> security cert install -vserver <svm> -type client -subtype kmip-cert 

>Paste the client certificate (3) here and press enter. 

>Enter your Private Key. You can find your private key on your vault instance from 
 $ jq -r '.data.private_key' < credential_cvo.json > cvo_key.pem

At this point, copy Root CA (4) and intermediate CA (5) in a block.

>Enter Intermediate Certificate: Press <Enter> when done. 

Message: Do you want to continue entering root and/or intermediate certificates {y|n}: n

3.2. Installing Server CA root certificate

svm:> security cert install -vserver <svm> -type server-ca -subtype kmip-cert

>Paste the Root CA (4) here

This command will install the Server CA root (4) certificate and specify the type of certificate as server-ca. The security install command is used to install certificates on NetApp Storage Systems.

3.3. Install Server CA intermediate

svm:> sec cert install -vserver <svm> -type server-ca -subtype kmip-cert 

>Paste the intermediate certificate (5) here.

3.4. Enable external KMIP Server

Display the certificate values

svm:> security certificate show -type client
svm:> security certificate show -cert-name vault*

Vserver    Serial Number   Certificate Name                       Type
-------------------------------------------------------------------------
lomar_svm  2E55AA5C7D806C3528D948E3BDD46C5E4232F07D 
                           b7q6h_2E55AA5C7D806C3528D948E3BDD46C5E4232F07D 
                                                                  client
  Certificate Authority: vault-kmip-default-intermediate
  Expiration Date: Sat Apr 01 16:46:16 2023

Enable KMIP Server

svm:> security key-manager external enable -vserver <svm> -key-servers <private_ip_adress_vault:5696> -client-cert <value_of_certificate_name> -server-ca-certs vault-kmip-default-intermediate, vault-kmip-default

svm:> security key-manager external show-status 

Node   Vserver   Primary Key Server                Status
-----------------------------------------------------------------
svm-1  lomar_svm  10.0.1.142:5696                   available

4. Enable Aggregate and/or Volume Encryption

NAE Encryption: Volumes created on NAE encryption enabled aggregates will be encrypted by default. With our Terraform deployment, an SVM and aggregate is already provisioned.

# Show encrypted Aggregates
svm:> storage aggregate show -fields encrypt-with-aggr-key

# Modify existing Aggregates to have NAE enabled
svm:> storage aggregate modify -aggregate aggr1 -encrypt-with-aggr-key true

Before you can enable aggregate encryption you have to convert all existing volumes within an aggregate, but for now encrypting our encryption_volume is enough.

svm:> volume show -encrypt false
Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------------------------------------------------------------------------------
svm1-01   vol0         aggr0_svm1_01 
                                    online     RW      72.71GB    62.03GB   10%
svm_svm1  encryption_volume 
                       aggr1        online     RW         10GB     9.50GB    0%
svm_svm1  svm_svm1_root 
                       aggr1        online     RW          1GB    972.1MB    0%

Modify existing volumes to have NVE enabled

svm:> volume encryption conversion start -vserver <svm> -volume <volume_name> 

# check status of volume encryption conversion
svm:> volume encryption conversion show -volume encryption_volume -vserver <svm>

> Vserver Name: svm_name
   Volume Name: encryption_volume
   Start Time: 3/24/2023 09:05:23
   Status: running

# Status changed from running in not currently going on 

> Vserver Name: svm_name
  Volume Name: encryption_volume
  Start Time: -
  Status: Not currently going on

Check volume encryption

svm:> volume show -encrypt true 

Vserver   Volume       Aggregate    State     Type   Size  Available Used%
----------------------------------------------------------------------------
svm_svm1  encryption_volume 
                       aggr1        online    RW     10GB     9.50GB    0%

If you are receiving the error message below, then you have not added your subscription to your BlueXP account or your NetApp Support Site Credentials Account have not been activated yet.

>Error: command failed: Package "VE" is not licensed in the cluster.

5. Mount your encrypted Volume

Visit your BlueXP Management Console, choose your Working Environment in Cloud Volumes ONTAP, select your volume and select the Mount Command.

# Copy and paste mount command on client machine
ubuntu:> mount 10.0.1.28:/encryption_volume <destination_directory>

Conclusion

This approach provides centralized key management and a strong root of trust, which is crucial for meeting data protection regulations, particularly in regulated industries such as healthcare, finance, and government institutions. Using an external key manager like HashiCorp Vault can be a beneficial solution for managing encryption keys in cloud data storage environments such as NetApp’s Cloud Volumes ONTAP.

To learn more about HashiCorp Vault and NetApp’s Cloud Volumes ONTAP, check out the following resources: