HashiCorp Vault Operations Professional exam practice guide — Part 1

I can’t believe 2 years have already passed since I wrote the Vault Ops Pro exam beta back in early 2022 and is now time to re-certify. I thought I would share some of the exercises I go through to prepare for such an exam. I have decided to split this into a multi-part series as there are a lot of content to cover.

• Part 2 (transit secrets engine & auto-unseal)
• Part 3 (auth methods, policies, and tokens)
• Part 4 (Vault agent & auto-auth)
• Part 5 (dynamic secrets)
• Part 6 (Vault HA cluster, backup & recovery)

Why I am calling this is a “practice” guide

In case you are unaware, the Vault Ops Pro exam is a practical exam, so you need to be well versed with the steps required to setup and manage a Vault cluster, add and configure Secrets Engines, Auth Methods, etc. “Studying”, here means “doing”.

Many of the key concepts require you to understand and know how to execute are (unfortunately) also infrequently performed actions such as initializing Vault, setting up auto unseal, or setting up dynamic credentials (although you may consume it on a daily basis, the setup is typically a one-time process). As such, it is important to occasionally refresh oneself and there is no better way than through practice!

Setup and format

One of the things I like about Vault (and HashiCorp tooling in general) is its deployment versatility. It will run on cloud VMs, Vagrant, Kubernetes, Raspberry Pis, Docker, etc. And some tooling such as Vault even has a dev mode which provisions you a Vault instance that runs in memory and provides you with an unseal key and root token in seconds.

TIP: I recommend using dev mode to practice re-keying Vault and rotating your root token

Personally, I like to use VMs because as you will see in the later parts that I have planned for this guide that I will set up a database server to configure dynamic credentials. It is just way simpler to install whatever I need through the Linux distribution’s package manager. That said, it would be a pain to have to do all the setup, get all the binaries, etc. every time you want to stand up a new Vault server or agent, so I have expedited this via Packer images that I create and maintain here.

NOTE: you do not need to do the full GitHub Actions setup, I have instructions on how to build locally and you only need the Vault base image

ANOTHER NOTE: I operate in Google Cloud, but I do have some Packer images for AWS and Azure here. It might be a bit more dated, but you can easily update it to match my GCP image config

I also do not need large VMs. On Google Cloud, I run 3 E2 series, shared-core VMs for everything that I do in this series:

• 2 e2-micro (2 vCPUs/1 GB mem) as my Vault auto-unsealer and Vault server
• 1 e2-medium (2 vCPUs/4 GB mem) as my Vault agent, database server, app server, etc.

In each part of my guide, I will run through an aspect of Vault management that is important to know and also an exam objective. I may also provide additional challenges for readers to try on their own as well as a solution guide at the very bottom. I also will NOT provide links to Vault documentation. You will be given a local copy of the Vault docs and API references during the exam, so knowing your way around the documentation page is also part of being prepared.

**DISCLAIMER**: this series of guides that I have planned are by no means exhaustive. My goal is for the reader to gain familiarity with (IMHO) core principles and concepts of Vault, but it is your responsibility to make sure you go through all the exam objectives and practice on your own.

With all that said, let’s finally start with the content of part 1…

Deploying and initializing Vault

Whether you are using one of my Packer images or something else, once you have Vault installed, a basic server configuration file would look like:

You will then have to initialize it with vault operator init which will produce an output similar to this:

Unseal Key 1: eFWUT4vhTIx6oQqUu/rA83o+aiMp/Pq5QrqMHwjHXeOK
Unseal Key 2: 6HSSoq4Pi/shMmilWs4uzlDYAsiM/pWW+MIdYoWkNeXe
Unseal Key 3: UmMLkF9RgTM3iyfQIvftgNohKnll99wzE4yhLJMCbPLa
Unseal Key 4: yyxU25SN0wFUVTK3nt1WKEWN0PJMjvCogimfbKYCruD8
Unseal Key 5: L7xJZe9UDKScJvHa5DxTaqEISLnO7q4+8L+C0T+B3U5c

Initial Root Token: hvs.Ohgf65vVQe0FNCkztR7aPf8D

By default, initialization will create 5 key shares with a threshold of 3, meaning you need any 3 of the 5 keys to unseal the vault. To start the unsealing process, run vault operator unseal and you will be prompted to provide an Unseal Key and be updated on the progress after each key. Once it is fully unsealed, your vault status should look something like the following:

Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.15.5
Build Date              2024-01-26T14:53:40Z
Storage Type            raft
Cluster Name            vault-cluster-62bfb303
Cluster ID              ba6450b8-9e2a-12c3-4f25-e396c6cffe7e
HA Enabled              true
HA Cluster              https://10.128.0.46:8201
HA Mode                 active
Active Since            2024-02-09T20:02:50.923318799Z
Raft Committed Index    51
Raft Applied Index      51

You will also be provided with an initial root token. This is the key to your kingdom and not something you would (or should) be using for your day-to-day Vault operators. However, for now, we will be using this until we get to policies, authentication and token creation.

CHALLENGE #1: (Re)deploy Vault server with TLS certs and write Vault audit logs to /var/log/vault/my-vault-audit.log

Do not worry — I do not expect you to know how to generate the certs yourselves and neither will the exam. I will not rule out the possibility that they may provide you with some certs to work with, so you will need to know how to update your Vault config appropriately.

• Generate CA cert (valid for 100 years):

openssl genrsa 2048 > ca.key

openssl req -x509 -new -sha256 -nodes -days 36500 \
  -key ca.key \
  -out ca.crt \
  -subj "/O=MyOrg"

• Copy ca.crt to /usr/local/share/ca-certificates and run update-ca-certificates
• Generate server cert:

openssl req -new -sha256 -nodes \
  -newkey rsa:2048 \
  -keyout vault.key \
  -out vault.csr \
  -subj "/O=MyOrg"

openssl x509 -req -sha256 -days 3650 \
  -in vault.csr \
  -out vault.crt \
  -CA ca.crt \
  -CAkey ca.key \
  -CAcreateserial \
  -CAserial vault.serial

NOTE: here is a alternative method for provisioning a CA and generating TLS certs as described by my colleague, Jacob.

CHALLENGE #2: Initialize Vault with 3 key shares and a key threshold of 2 and encrypt the root token with PGP

Similarly here, you will not be expected to know how to generate the PGP key during the exam, but you may be provided with one (or some) in a question that you may have to use. Here is how you can generate your own PGP key:

gpg --ful-generate-key

gpg --output public.pgp --armor --export your-email@example.com

>> SPOILER ALERT!!! SOLUTION GUIDE BELOW!!! <<

Challenge solutions guide

SOLUTION #1

• Copy generated vault.* cert files to Vault TLS directory (and change ownership to vault:vault)
• Update Vault config appropriately (example below):

NOTE: here, you will want to export VAULT_SKIP_VERIFY=true to access the HTTPS endpoint as your cert here cannot be validated

• Initialize and login with your root token

Vault audit logs are not enabled by default, but is recommended that you do. There are different types of devices you can write your logs to, but here we will just specify a local file:

vault audit enable file file_path=/var/log/vault/my-vault-audit.log

SOLUTION #2

• Initialize Vault:

vault operator init \
  -key-shares=3 \
  -key-threshold=2 \
  -root-token-pgp-key="/path/to/public.pgp"

• Sample output:

Unseal Key 1: exxBXOGrQr3Q0Zw9iBSewdtlXXSH5fe/ZiKl9UqyAQBh
Unseal Key 2: XvIcBPmO0VeSCACocnp/9ZS0WDdrn8Ti9Cc975GAU3iT
Unseal Key 3: NZL4fHTlooc/rcwzVOugRN5qxLZTSLEGq7tBOmzZCPfO

Initial Root Token: wcDMA7kbilRiSlRsAQv9HBkDK3Q0E7/XUCYZs85JryOrkXr5s6KPRuq7VnFBrgiPCqn9hK+Gc1aLuIHRU3zQA0ibJ8Es2a8em/9/JujCjWi2FauZ2ZN+N9HOfaEsrYeXwqZPaLVSJmnxDbEbSAGsNiQF7ELOZ3sP3oQOWBeDe8PRb1SjwOij5jKmxPjMPqc99B2kQ7CkqTvKItx3VTAcxDWvfzkMKxeGAtp5CS/716K/0HEnBbkaZwfGWM8xykxjOinfR6XdrBzW3wf7dqqdqRHaSc18MIW7SuK/xo2CCzP3BouGqSQdsLlO4FhXWsuZmpuw+Vlm+TRq/15ihjiRvgtxHJ1AQM01ZccRDOT+e8HoxGYyolhAyySAjiE4ZHqb8rwNhwzQmw/MUdT9+6zKM9mnx6qqJv0NGQOORhgknj/8pc7fr+eUkNNT6r6bHbe4894KuDhRZmCSFZ8Bf9X68NGsmRye9xtq/4d3PyCQ+SZRH5LGMZknVhmsdXtaPm/X4p+cFBBWLVdIIJpCqpr40k0BrzSGgEBvimx8rQtRJHETtqhLqXd5DApii432jfzW49R0YVDpQrtTANCoFNLw3fN85ldJNFceDMSGnc+5yVhHyctDHqQeVvOLUcpMJA==

• Extract root token:

echo '[PGP_ENCRYPTED_ROOT_TOKEN]' | base64 -d > root-token.dat

gpg --decrypt root-token.dat