Backup a Consul cluster using `consul snapshot` command

When Consul is a Vault storage backend and runs on Kubernetes

Published in

Wealth Wizards Engineering

6 min readMay 9, 2018

Taking full Consul backups for DR scenarios it’s actually quite straightforward: you just need to run consul snapshot save backup.snap. Job done! But what if my Consul cluster is a storage backend for Vault, running on Kubernetes and I want to encrypt my backup and ship it to an AWS S3 bucket? How can I go about running my backup tool of choice on a Kube pod and minimise secrets management?

Bash to the rescue! And Docker of course, and Vault’s awesome features👏!!

This post describes how to use consul snapshot save command inside a Kubernetes pod and use Vault Kubernetes auth backend alongside AppRole and AWS secrets engine to backup a consul Cluster.

The goal is to provide a quick and easy way to restore Vault, mainly in case of a DR, but not limited to this. The compute part of both Consul and Vault are defined in code, stored in git and can be brought back online in no time by running a single bash script for each product. Backing up Vault’s data, which is actually stored in Consul is achieved by using consul snapshot command.

As described in consul snapshot docs, the snapshot command has subcommands for saving, restoring, and inspecting the state of the Consul servers for disaster recovery. These are atomic, point-in-time snapshots which include key/value entries, service catalog, prepared queries, sessions, and ACLs. This command is available in Consul 0.7.1 and later.

There are some utilities out there for backing-up a Consul cluster, and the best one I found so far is https://github.com/pshima/consul-snapshot which can ship backups to AWS S3, runs as a daemon and has integrated monitoring and backup 👌. The drawback for using it , is that it cannot restore Consul ACLs 😞, which makes the restore of our Vault service slightly more complicated and a bit lengthier process. Being able to restore Consul ACLs gives the fastest and simplest restore process: bring up a new Consul cluster and then just run consul snapshot restore backup.snap: all data is restored at this point. Now just start Vault pods, unseal and this is it!

So I’ve created a Docker container called consul-backup that runs some Bash wrapper scripts which allow me to take Consul backups and has a restore functionality as well. Since Consul is a storage backend for Vault and is running on Kubernetes, this is how it all hangs together:

Consul-backup pod uses Vault Kubernetes Auth Method to obtain an initial Vault token
Use the token to authenticate against Vault using AppRole Auth Method
Use AWS Secrets Engine to get a dynamically generated set of AWS keys that allow access to the S3 backup bucket
Get a secret from Vault to use for encrypting the backup before pushing it to S3.
Get a Consul management token so it can used when running `consul snapshot save`. Since Consul snapshots actually contain ACL tokens, the Snapshot API requires a management token for snapshot operations and does not use a special policy.

Going into details, below are the steps on how it was all set up assuming these names were used for the clusters:

Kubernetes: kube.mydomain
Vault: vault.mydomain
Consul consul.mydomain

These are Vault configuration steps:

Configure Consul management tokens lease:

vault write consul/roles/management token_type=management lease=30m

2. Configure Vault AppRole Auth Method:

vault write auth/approle/role/consul-backup \
secret_id_ttl=5m \
token_num_uses=4 \
token_ttl=5m \
token_max_ttl=5m \
secret_id_num_uses=1 \
bound_cidr_list=”10.22.88.0/23" \
bind_secret_id=”true” \
policies=”consul-backup”

bound_cidr_list is optional and if set, specifies blocks of IP addresses which can perform the login operation.

3. Create the policy for the above AppRole:

cat consul-backup.hcl
path “aws-s3-consul-backup/creds/s3-consul-backup” {
  capabilities = [“read”]
}
path “secret/infrastructure/consul-backup” {
  capabilities = [“read”]
}
path “consul/creds/management” {
  capabilities = [“read”]
}vault policy write consul-backup consul-backup.hcl

4. Configure Vault Kubernetes Auth Method:

Enable Kubernetes Auth:
vault auth enable -path=kube -plugin-name=kubernetes kubernetes
Create Kubernetes certificate chain:

kubectl get secrets default-token-j3291 -o json | jq -r ‘.data[“ca.crt”]’ | base64 -D > kubernetes.pemopenssl s_client -connect api.kube.mydomain:443 2> /dev/null < /dev/null | openssl x509 > kubernetes-master.pemcat kubernetes-master.pem kubernetes.pem > kube.pem

Configure Vault to talk to Kubernetes:

vault write auth/kube/config \
 kubernetes_host=https://api.kube.mydomain \
 kubernetes_ca_cert=”@kube.pem`

Create a named role:

vault write auth/$cluster/role/consul-backup \
 bound_service_account_names=consul-backup \
 bound_service_account_namespaces=infrastructure \
 policies=consul-backup-login \
 ttl=300 \
 max_ttl=300 \
 num_uses=3`

Configure Kubernetes cluster — service Accounts used in this auth method will need to have access to the TokenReview API so create a CusterRolebinding similar to the one described in Vault documentation: https://www.vaultproject.io/docs/auth/kubernetes.html#configuring-kubernetes
Create Vault role defined in the above Kubernetes config:

cat consul-backup-login.hcl 
path “auth/approle/role/consul-backup/role-id” {
  capabilities = [“read”]
}
path “auth/approle/role/consul-backup/secret-id” {
  capabilities = [“update”]
}
path “auth/approle/role/consul-backup/role-id” {
  capabilities = [“read”]
}
path “auth/approle/role/consul-backup/secret-id” {
  capabilities = [“update”]
}vault policy write consul-backup-login consul-backup-login.hcl

5. Configure Vault AWS Secrets engine:

Enable AWS secrets:

 vault secrets enable -path=aws-s3-consul-backup \
 -description=”Access consul-backup S3 bucket” \
 -default-lease-ttl=”5m” \
 -max-lease-ttl=”5m” \
 -plugin-name=aws aws

Configure the credentials that Vault uses to communicate with AWS to generate the IAM credentials:

vault write $path/config/root \
 access_key=”secret_key” \
 secret_key=”even_more_secret_key”`

Configure a role that maps a name in Vault to a policy or policy file in AWS. When users generate credentials, they are generated against this role:

vault write aws-s3-consul-backup/roles/s3-consul-backup      arn:aws:iam::1111111111111:policy/s3-consul-backup

Now that Vault is configured and all is in place, we can go into the details of the Bash scripts used by consul-backup container. The container repo is published here: https://github.com/WealthWizardsEngineering/consul-backup. It also contains a sample Kubernetes cronjob so it can be run as a job on the cluster. consul-backup has basically 3 scripts as shown below:

environment.sh : as the name indicates, this one sets-up the environment by getting required secrets from Vault. It does things like:

....................................................................
# Login to Vault and so I can get an approle token
VAULT_LOGIN_TOKEN=$(curl -sS --request POST \
  ${VAULT_ADDR}/v1/auth/${KUBERNETES_AUTH_PATH}/login \
  -H "Content-Type: application/json" \
  -d '{"role":"'"${VAULT_LOGIN_ROLE}"'","jwt":"'"${KUBE_SA_TOKEN}"'"}' | \
  jq -r 'if .errors then . else .auth.client_token end')
....................................................................
export GPG_PHRASE=$(curl -sS --header "X-Vault-Token: ${APPROLE_TOKEN}" \
    ${VAULT_ADDR}/v1/secret/infrastructure/${VAULT_LOGIN_ROLE} | \
    jq -r 'if .errors then . else .data.gpg_phrase end')export CONSUL_HTTP_TOKEN=$(curl -sS --header "X-Vault-Token: ${APPROLE_TOKEN}" \
    ${VAULT_ADDR}/v1/consul/creds/management | \
    jq -r 'if .errors then . else .data.token end')....................................................................

2. backup.sh : it’s quite obvious what it does, and thanks Rich Marshall for adding InfluxDB events and a status on our current backup monitoring tool of choice. Here is the script:

#!/bin/bash -eunset_vars() {
  unset GPG_PHRASE
  unset CONSUL_HTTP_TOKEN
  unset ACCESS_KEY
  unset SECRET_KEY
}clean_environment(){
  rm -f consul-backup-*.snap
  rm -f /tmp/tmpfile-*
  unset_vars
}
trap clean_environment EXITBACKUP_FILE="consul-backup-$(date +"%H-%M-%S").snap"
S3_BACKUP_DIR="$(date +"%Y")/$(date +"%m")/$(date +"%d")"# Define where and how to send events to influx
POST2INFLUX="curl -XPOST --data-binary @- ${INFLUXDB_URL}"# Get CONSUL_HTTP_TOKEN, GPG_PHRASE, ACCESS_KEY and SECRET_KEY
source /environment.sh# Backup consul
echo "Using CONSUL_HTTP_ADDR: ${CONSUL_HTTP_ADDR}"
echo "Running consul snapshot save"
echo -n "instance_backup_started,instance=consul value=true" | ${POST2INFLUX}
echo -n "database_backup_started,instance=consul,database=consul_kv value=true" | ${POST2INFLUX}
consul snapshot save ${BACKUP_FILE} || { echo "instance_snapshot_failed,instance=consul value=true" | ${POST2INFLUX} && exit 1; }
echo "database_backup_completed,instance=consul,database=consul_kv value=true" | ${POST2INFLUX}# Inspect the backup
consul snapshot inspect ${BACKUP_FILE} || { echo "instance_inspect_failed,instance=consul value=true" | ${POST2INFLUX} && exit 1; }# Push to S3
echo -n "database_s3-put_started,instance=consul,database=consul_kv value=true" | ${POST2INFLUX}
s3cmd put  ${BACKUP_FILE} s3://${S3_BUCKET}/${S3_BACKUP_DIR}/${BACKUP_FILE} || { echo "instance_s3-put_failed,instance=consul value=true" | ${POST2INFLUX} && exit 1; }
echo -n "database_s3-put_completed,instance=consul,database=consul_kv value=true" | ${POST2INFLUX}
echo -n "instance_backup_completed,instance=consul value=true" | ${POST2INFLUX}

3. restore.sh : it expects a brand new Consul cluster to restore to. It warns the user about the restore impact, asks for confirmation and if answered ‘yes’ it proceeds with the restore. The restore is a full restore, and most likely needed in a full DR scenario, but it can also be used for selective restore by doing a full restore to a new Consul instance followed by consul kv export and consul kv import . The important bits of the restore script are:

....................................................................
Yes )
   s3cmd get s3://$S3_BUCKET/$REMOTE_FILE_PATH $LOCAL_FILE
   echo "" && echo "Inspecting the backup"
   consul snapshot inspect $LOCAL_FILE   echo "Performing the restore"
   consul snapshot restore $LOCAL_FILE   break
   ;;No )
   break
   ;;
....................................................................

And… this is it! Bash to the rescue 👍!! Now this approach is specific for the use case described at the beginning of this article but it shows how you can quickly add some bash wrapper scripts around consul snapshot command and create something fit for your purpose when needed. Happy scripting!!

Ps. below diagram shows the relations between Vault Kubernetes Auth AppRole, Vault role and AWS secrets engine:

Backup a Consul cluster using `consul snapshot` command

When Consul is a Vault storage backend and runs on Kubernetes

Written by Ionut Craciunescu