Open in app

Sign in

Write

Sign in

How to Backup a HashiCorp Vault Integrated Storage Cluster with MinIO

Nicolas Ehrman
HashiCorp Solutions Engineering Blog
Published in
7 min readOct 7, 2020

Disclaimer: This article talks about an Open Source Snapshot Agent which is not supported by HashiCorp. Since Vault 1.6, the official Automated Integrated Storage Snapshot is available and should be used for Production environment.

Since HashiCorp Vault 1.4, a new feature that we call Integrated Storage became GA. With Integrated Storage you don’t have to rely on external storage by using the servers’ own local filesystems for storing data, making Vault operations even easier (In the past, Consul KV servers were used for most Vault deployments). But, even if Integrated Storage is highly available, it is still recommended that you have another security layer by providing backup.

In this blog post, we will see how to use Integrated Storage and how to take advantage of the open source tooling to use an automated snapshot backup for your Vault cluster.

Before going deeper into the CLI commands and code, maybe it could be interesting to understand how Integrated Storage is implemented in HashiCorp Vault.

HashiCorp Vault Integrated Storage

First of all, Integrated Storage uses a consensus protocol that is based on raft for replicating data between each Vault server in the cluster but also for providing consistency with regards to CAP’s definition.

As you can see in the following picture, a Vault cluster works with an elected leader (you can start with only 1 node) and follower nodes that participate in the quorum. The quorum requires at least (N+1/2) members to be able to elect a leader.

For example, if you have 3 nodes in the cluster, you can lose only 1 node.

Note: If you want to go in-depth on how Integrated Storage works, you can go to our website.

Minimum supported deployment for HA

Challenges

Even if the storage is highly available and consistent, you still need to backup Vault data as you must be protected against deletion, corruption, or sabotage in your cluster. (DR included)

Fortunately, HashiCorp Vault provides the ability to snapshot at the raft level and it can be automated by using script and cron for instance. It is not yet integrated with the consul snaphot agent.

That’s where the power of HashiCorp’s open source community comes to play as Robert Lippens (thanks to him for his very good work) wrote an excellent Vault raft snapshot agent in Go that can automate snapshots and send them to local, S3, Azure, or GCP storage.

But, what about our existing and future Enterprise-version customers who deployed HashiCorp Vault on-premise and need to backup Vault with Integrated Storage on their S3 compatible storage instead of in the cloud?

Again, that’s the power of the community. I made some modifications to Robert Lippens’ code to enable the support of S3 compatible storage and did a pull request on his repo to make it available for everyone.

How to Test It

Grab your keyboard of choice and let’s run some CLI commands.

Test infrastructure

What Do You Need?

  • At least one Vault server up and running, configured to use raft as the storage backend
  • MinIO Docker container, a lightweight S3 compatible object storage utility, up and running with a configured bucket
  • The Vault raft snapshot agent built for your OS (Linux, Windows or even macOS)

Configure Your Demo Environment

First, download the Vault binary, configure Vault, and finally start the Vault server. To do so, use these commands:

$> curl -o vault_1.5.3_linux_amd64.zip https://releases.hashicorp.com/vault/1.5.3/vault_1.5.3_linux_amd64.zip
$> unzip vault_1.5.3_linux_amd64.zip
$> sudo mv vault /usr/local/bin/
$> tee config.hcl << EOF
cluster_name = "vault"storage "raft" {
    path = "/var/vault_raft"
    node_id = "vault01"
}listener "tcp" {
    address = "192.168.0.3:8200",
    tls_disable = true
}api_addr = "http://192.168.0.3:8200"
cluster_addr = "http://192.168.0.3:8201"
ui = true
EOF
$> vault server -config=config.hcl

Note: This is not a recommended deployment for production. This is for testing purposes only. Our production reference architecture can be found here.

Now, you have to initialize the Vault cluster (even if it’s only 1 node) and verify that your Vault server is really the raft leader.

$> export VAULT_ADDR="http://192.168.0.3:8200"
$> vault operator init -key-shares=1 -key-threshold=1
Unseal Key 1: zDXvOhOcyAM63R3IwezNDmd75SIfg6Z1eLjVO1Ri9H6=Initial Root Token: s.HqdXjeeaQv0wAbSKHpKtmbnzVault initialized with 1 key shares and a key threshold of 1. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 1 of these keys to unseal it
before it can start servicing requests.Vault does not store the generated master key. Without at least 1 key to
reconstruct the master key, Vault will remain permanently sealed!It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
$> vault operator unseal 
$> vault login 
$> vault operator raft list-peers 
Node       Address              State     Voter
----       -------              -----     -----
vault01    192.168.0.3:8201     leader    true

Now the Vault server is up and running with Integrated Storage and is the leader. Now we can deploy MinIO object storage with this simple command:

$> docker run -p 9000:9000 \
  -e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" \
  -e "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \
  minio/minio server /data

Validate that and you can access the MinIO server and create a new bucket. Mine is called “mybucket”:

MinIO Server with bucket already created

Manual Backup Process

With the Vault CLI (or via API) you can save your current state to a file with snapshot. The command is quite simple:

$> vault operator raft snapshot save mysnapshot.snap

Then, you can send your snapshot file to an external storage location like a bucket or a file share hosted on-prem.

Configure Automated Backup with the Agent

We are ready to download, configure, and run our Vault raft snapshot agent.

Note: The Vault raft snapshot agent can be installed on each Vault server in the cluster but only the raft leader can be used to create a snapshot.

From your computer, download the version 0.1.0 of the Vault raft snapshot agent and send it to the Vault server (or update your image if you use Packer).

$> wget https://github.com/Lucretius/vault_raft_snapshot_agent/releases/download/v0.1.0/vault_raft_snapshot_agent_linux_amd64

From the Vault server, deploy the Vault raft snapshot agent and create the service as shown below:

$> chmod +x vault_raft_snapshot_agent_linux_amd64
$> sudo mv vault_raft_snapshot_agent_linux_amd64 /usr/local/bin/vault_raft_snapshot_agent
$> sudo tee /etc/systemd/system/vault-snapshot.service <<EOF
[Unit]
Description="An Open Source Snapshot Service for Raft"
Documentation=https://github.com/Lucretius/vault_raft_snapshot_agent/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/snapshot.json[Service]
Type=simple
User=vault
Group=vault
ExecStart=/usr/local/bin/vault_raft_snapshot_agent
ExecReload=/usr/local/bin/vault_raft_snapshot_agent
KillMode=process
Restart=on-failure
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF

In order to make the Vault raft snapshot agent work properly, we need to configure an AppRole on the Vault server to be able to authenticate and give access to the snapshot API endpoint. Follow these commands to do so:

$> tee snapshot_policy.hcl <<EOFpath "/sys/storage/raft/snapshot"
{
  capabilities = ["read"]
}EOF
$> vault policy write snapshot_agent snapshot_policy.hcl
Success! Uploaded policy: snapshot_agent
$> vault write auth/approle/role/snapshot token_policies="snapshot_agent"
Success! Data written to: auth/approle/role/snapshot
$> vault read auth/approle/role/snapshot/role-id
Key        Value
---        -----
role_id    9946cdef-51c3-d69a-5354-589d3168378f
$> vault write -f auth/approle/role/snapshot/secret-id
Key                   Value
---                   -----
secret_id             c3064f35-c5bd-175b-0848-b59f59a7aa83
secret_id_accessor    ff5b0931-0161-bdc7-c235-1dc9b1af370e

Good, now we can configure the vault raft snapshot agent to meet our needs and enable and start the service:

$> sudo tee /etc/vault.d/snapshot.json <<EOF
{
   "addr":"http://192.168.0.3:8200",
   "retain":5,
   "frequency":"30s",
   "role_id": "9946cdef-51c3-d69a-5354-589d3168378f",
   "secret_id":"c3064f35-c5bd-175b-0848-b59f59a7aa83",
   "aws_storage":{
      "access_key_id":"AKIAIOSFODNN7EXAMPLE",
      "secret_access_key":"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
      "s3_region":"us-east-1",
      "s3_bucket":"mybucket",
      "s3_endpoint":"http://192.168.0.4:9000",
      "s3_force_path_style":true
   }
}
EOF$> systemctl enable vault-snapshot 
$> systemctl start vault-snapshot

Finally, we can check that the service is running and that we have backups on our S3 compatible storage.

$>systemctl status vault-snapshot
● vault-snapshot.service - "An Open Source Snapshot Service for Raft"
     Loaded: loaded (/etc/systemd/system/vault-snapshot.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-09-17 07:36:38 UTC; 5 days ago
       Docs: https://github.com/Lucretius/vault_raft_snapshot_agent/
   Main PID: 1286590 (vault_raft_snap)
      Tasks: 9 (limit: 4621)
     Memory: 27.9M
     CGroup: /system.slice/vault-snapshot.service
             └─1286590 /usr/local/bin/vault_raft_snapshot_agentSep 22 12:46:29 vault01 vault_raft_snapshot_agent[1286590]: 2020/09/22 12:46:29 Successfully created aws snapshot to http://192.168.0.4:9000/mybucket/raft_snapshots/raft_snapshot-1600778789671583854.snap
Sep 22 12:46:59 vault01 vault_raft_snapshot_agent[1286590]: 2020/09/22 12:46:59 Successfully created aws snapshot to http://192.168.0.4:9000/mybucket/raft_snapshots/raft_snapshot-1600778819718810064.snap
Sep 22 12:47:29 vault01 vault_raft_snapshot_agent[1286590]: 2020/09/22 12:47:29 Successfully created aws snapshot to http://192.168.0.4:9000/mybucket/raft_snapshots/raft_snapshot-1600778849769717951.snap
Sep 22 12:47:59 vault01 vault_raft_snapshot_agent[1286590]: 2020/09/22 12:47:59 Successfully created aws snapshot to http://192.168.0.4:9000/mybucket/raft_snapshots/raft_snapshot-1600778879834950807.snap
Sep 22 12:48:29 vault01 vault_raft_snapshot_agent[1286590]: 2020/09/22 12:48:29 Successfully created aws snapshot to http://192.168.0.4:9000/mybucket/raft_snapshots/raft_snapshot-1600778909884117828.snap
Sep 22 12:48:59 vault01 vault_raft_snapshot_agent[1286590]: 2020/09/22 12:48:59 Successfully created aws snapshot to http://192.168.0.4:9000/mybucket/raft_snapshots/raft_snapshot-1600778939936101759.snap

From the MinIO’s UI:

And we’re done. Now you have a fully configured Vault with an Integrated Storage cluster backed up automatically by the Vault raft snapshot agent.

Conclusion

In this blog post, you saw how to use MinIO to leverage features that are not already implemented out-of-the-box (but it will be soon) to enhance HashiCorp Vault and make it even better.

References:

Vault
Hashicorp
Hashicorp Vault
Secops

--

--

Nicolas Ehrman
HashiCorp Solutions Engineering Blog

Written by Nicolas Ehrman

59 Followers
Writer for

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams