Continuous Delivery of HashiCorp Vault on Google Kubernetes Engine:Backup & Recovery

This is Part 8 of a series: Index

Overview

To backup GCS data, we enable object versioning which will allow us to restore individual objects to prior versions.

You can view versions using gsutil:

gsutil ls -lra gs://ops-bcurtis-sb-vault-storage/core/mounts
617  2018-10-11T02:05:26Z  gs://ops-bcurtis-sb-vault-storage/core/mounts#1539223526242045  metageneration=1
690  2018-10-11T02:05:26Z  gs://ops-bcurtis-sb-vault-storage/core/mounts#1539223526479479  metageneration=1
771  2018-10-29T18:50:20Z  gs://ops-bcurtis-sb-vault-vault-storage/core/mounts#1540839020037493  metageneration=1
TOTAL: 3 objects, 2078 bytes (2.03 KiB)

You can see the three versions of this file and any one can be recovered using copy:

gsutil cp gs://ops-bcurtis-sb-vault-storage/core/mounts#15392235262
42045 gs://ops-bcurtis-sb-vault-storage/core/mounts
Copying gs://ops-bcurtis-sb-vault-storage/core/mounts#1539223526242045 [Content-Type=application/octet-stream]...
/ [1 files][  617.0 B/  617.0 B]

Next you can see there are now four versions:

gsutil ls -lra gs://ops-bcurtis-sb-vault-storage/core/mounts
617  2018-10-11T02:05:26Z  gs://ops-bcurtis-sb-vault-storage/core/mounts#1539223526242045  metageneration=1
690  2018-10-11T02:05:26Z  gs://ops-bcurtis-sb-vault-storage/core/mounts#1539223526479479  metageneration=1
771  2018-10-29T18:50:20Z  gs://ops-bcurtis-sb-vault-storage/core/mounts#1540839020037493  metageneration=1
617  2018-12-04T14:50:46Z  gs://ops-bcurtis-sb-vault-storage/core/mounts#1543935046514223  metageneration=1
TOTAL: 4 objects, 2695 bytes (2.63 KiB)

Cloud Storage Transfer Service is used to backup the Cloud Storage Bucket from the source project to the destination backup project ops-bcurtis-sb-vault-backup. This runs at 8:00am every morning. If you are doing a upgrade or something else that may have a negative impact on the data store you can disable the transfer job so you do not sync bad data. Once things have been validated enable the sync again

If we accidentally delete a KMS key, there is a small recovery window where we can immediately contact our TAM/support rep).

Recovery

Data Store (Google Cloud Storage)

To recover we can run a rsync between the backup cloud storage bucket and the destination vault storage bucket:

gstuil rsync -r gs://SOURCE_BUCKET_NAME gs://DESTINATION_BUCKET_NAME

Project Destroyed

To recover from an accidental destroy or something that renders the google cloud project to be unusable or lost we would need to reach out to Google Support to recover KMS key. You would then need to build a new environment (switching to the recovered key) and synchronize everything like mentioned above.