Linux password rotation automation with Hashicorp Vault

7 min readFeb 1, 2020

This article will demonstrate how to achieve password rotation automation with secret management tool Hashicorp Vault.

First, setting up a new CentOS EC2 instance on AWS as our Vault server. To make sure that HashiCorp Vault cluster that is reachable from your server instances (Inbound TCP port 8200 to Vault), in security group setting, open up the inbound TCP connection for port 8200, as your client need to communicate to Vault API via this port.

Also, launched a few other EC2 instances as client in the network, we will later use the vault to rotate their root password.

Once the instance is launched, SSH into the server and install wget with the following command:

sudo yum install wget

Head to Hashicorp Offical Website copy the appropriate download link(Linux) for your CentOS. Right-click on the download button and click Copy Link Address. Next, go back to the terminal that connected to our EC2 instance, use the wget command and the link address we just copied to download the binary to your server.

sudo wget https://releases.hashicorp.com/vault/1.2.3/vault_1.2.3_linux_amd64.zip

On CentOS 7, the unzip package may not be installed by default. Install the unzip package by the following command:

sudo yum install unzip

then unzip the file in the directory where the binary just downloaded:

sudo unzip vault_1.2.3_linux_amd64.zip

Safely remove the zip file after unzip process is done.Type:

sudo chown root:root vault

Move the vault file to /usr/local/bin/ or or to some other DIR that’s present in your $PATH:

sudo mv vault /usr/local/bin/

Verify Vault is installed correctly with following command:

vault --version

Next we need to set up and deploy our vault server, first we need to configuring Vault before we start our vault server.

Vault is configured using HCL files. There are mainly two components in the configuration file:

Storage and Listener :

Storage is the physical backend that Vault uses for storage. In this example we will use consul as our storage backend, if you are running your vault in dev mode, vault will use in memory storage.

Install and starting a local Consul instance on Vault Server:

Find the appropriate Consul package for your CentOS and right-click on the download button and click Copy Link Address. Next, go back to the terminal, use the wget command and the link address we just copied to download the binary to your server.

sudo wget https://releases.hashicorp.com/consul/1.6.3/consul_1.6.3_linux_amd64.zip

Unzip the file:

sudo unzip consul_1.6.3_linux_amd64.zip

Consul runs as a single binary named consul so we can move the binary to your $PATH and safely remove all other file:

sudo mv consul /usr/local/bin/

You can check the location of your path by entering:

echo $PATH

Starting the Consul dev server by typing the following command:

consul agent -dev

One or more listeners can be specify in the HCL file to determine how Vault listens for API requests. The example below Vault will listens on localhost port 8200 without TLS. If you are have clients that will communicate from a different server then we need set the address to 0.0.0.0:8200, which in the context of servers, 0.0.0.0 means all IPv4 addresses on the local machine. If a host has two IP addresses, 192.168.1.1 and 10.1.2.1, and a server running on the host listens on 0.0.0.0, it will be reachable at both of those IPs. Hence, we can access the APIs via port 8200 from our client servers.

storage "consul" {   
  address = "127.0.0.1:8500"   
  path    = "vault/" 
}  listener "tcp" {  
   address = "127.0.0.1:8200"  
   tls_disable = 1 
}

storage "consul" {   
  address = "0.0.0.1:8500"   
  path    = "vault/" 
}listener "tcp" {  
   address = "0.0.0.1:8200"  
   tls_disable = 1 
}

Save the file as config.hcl . Modify the -config flag to point to the proper path where you saved theconfig.hcl file.

Start the Vault server by the following:

vault server -config=config.hcl

On Linux, to give the Vault executable the ability to use the mlock syscall without running the process as root, run below command and restart the vault server:

sudo setcap cap_ipc_lock=+ep $(readlink -f $(which vault))

Initialising the Vault:

During initialisation, the encryption keys are generated, unseal keys are created, and the initial root token is setup.

To initialise Vault run:

vault operator init

Unseal the Vault:

Every initialised Vault server is sealed. From the configuration, Vault can access the physical storage, but it doesn’t know how to decrypt it. The process of teaching Vault how to decrypt the data is called unsealing.

To unseal the Vault, you must enter threshold number of unseal keys. In this case our “key threshold” is 3. Which means that to unseal the Vault, you need 3 out of the 5 keys that were generated.

Enter the following command three times with 3 different unseal key. The vault will be unsealed:

vault operator unseal

When the vault is unsealed, the value for Sealed changes to false .

Install the Password Generator Plugin:

vault-secrets-gen is a password generator plugin, creates new gen/password and gen/passphrase API endpoints.

Download the corresponding release from this Github link to your Vault server:

sudo wget https://github.com/sethvargo/vault-secrets-gen/releases/download/v0.0.6/vault-secrets-gen_0.0.6_linux_amd64.tgz

In order to unpack the .tgz file on your CentOS server, install tar:

sudo yum tar

and run:

sudo tar zxvf vault-secrets-gen_0.0.6_linux_amd64.tgz

use command mkdir to make sure the directory /etc/vault/plugins/ exist on your server.

Amend the config.hcl file by appending the following to the end of the file and restart the vault server, after you restart your vault server, remember you have to unseal it again:

plugin_directory = "/etc/vault/plugins"

Then move the binary vault-secrets-gen to the vault plugins directory:

mv vault-secrets-gen /etc/vault/plugins/vault-secrets-gen

Enable mlock so the plugin can safely be enabled and disabled:

setcap cap_ipc_lock=+ep /etc/vault/plugins/vault-secrets-gen

Calculate the SHA256 of the plugin and register it in Vault’s plugin catalog:

export SHA256=$(sha256sum "/etc/vault/plugins/vault-secrets-gen" | cut -d' ' -f1)vault write sys/plugins/catalog/secrets-gen \
    sha_256="${SHA256}" \
    command="vault-secrets-gen"

Mount the secrets engine:

vault secrets enable \
    -path="gen" \
    -plugin-name="secrets-gen" \
    plugin

Mount a version 2 K/V secrets backend at systemcreds:

Version 2 of the kv secrets engine supports secret versioning run:

vault secrets enable -version=2 -path=systemcreds/ kv

Configure ACL Policies for CentOS:

Create file rotate-linux.hcl and write the following policies:

# Allows hosts to write new passwordspath "systemcreds/data/linux/*" {capabilities = ["create", "update"]}# Allow hosts to generate new passphrasespath "gen/passphrase" {capabilities = ["update"]}# Allow hosts to generate new passwordspath "gen/password" {capabilities = ["update"]}

and save the following to file linuxadmin.hcl :

# Allows admins to read passwords.path "systemcreds/*" {capabilities = ["list"]}path "systemcreds/data/linux/*" {capabilities = ["list", "read"]}

Run the following commend to configure above policies to your Vault server:

vault policy write rotate-linux rotate-linux.hcl

Generate a token for each client server:

Create a new token for each of your client servers:

vault token create -period 24h -policy rotate-linux

output from above command:

Key                Value---                -----token              6vtW8IOVRTBkvPfUV3RNx7vBtoken_accessor     6CNHqSIG79t7mzROq6YRYPTltoken_duration     24htoken_renewable    truetoken_policies     [default rotate-linux]

Now SSH into each client server you want to rotate the password on the regular basis, Set up your environment variables:

export VAULT_ADDR=http://YOUR.EC2_IP_ADDR.COM:8200

The VAULT_TOKEN below is the output next to the file token above from the token generation for each server:

export VAULT_TOKEN=6vtW8IOVRTBkvPfUV3RNx7vB

Download the script:

Now you can download the script to your client server;

Or you can write the following part to file rotate-linux-password.sh on your client server:

Renew the token:

curl -sS --fail -X POST -H "X-Vault-Token: $VAULT_TOKEN"
${VAULT_ADDR}/v1/auth/token/renew-self | grep -q 'lease_duration'
retval=$?
if [[ $retval -ne 0 ]]; then
echo "Error renewing Vault token lease."
fi

Generate a New Password:

NEWPASS=$(curl -sS --fail -X POST -H "X-Vault-Token: $VAULT_TOKEN" -H "Content-Type: application/json" --data '{"words":"4","separator":"-"}'  ${VAULT_ADDR}/v1/gen/passphrase | jq -r '.data|.value')JSON="{ \"options\": { \"max_versions\": 12 }, \"data\": { \"${USERNAME}\": \"$NEWPASS\" } }"

Save the New Password to Vault:

# First commit the new password to vault, then capture the exit statuscurl -sS --fail -X POST -H "X-Vault-Token: $VAULT_TOKEN" --data "$JSON" ${VAULT_ADDR}/v1/systemcreds/data/linux/$(hostname)/${USERNAME}_creds | grep -q 'request_id'retval=$?if [[ $retval -eq 0 ]]; then# After we save the password to vault, update it on the instanceecho "$USERNAME:$NEWPASS" | sudo chpasswdretval=$?if [[ $retval -eq 0 ]]; thenecho -e "${USERNAME}'s password was stored in Vault and updated locally."elseecho "Error: ${USERNAME}'s password was stored in Vault but *not* updated locally."fielseecho "Error saving new password to Vault. Local password will remain unchanged."exit 1fi

Run the script to rotate the password:

sudo bash rotate-linux-password.sh YOUR_USER_NAME

If your client server is running CentOS this might be:

sudo bash rotate-linux-password.sh centos

Schedule a corn job to auto rotate the password every x days:

To edit the current user’s cron jobs:

crontab -e

add the following corn job to the file, now a password will be generate from the Vault and rotate every x days by each server:

0 0 */x * *  PATH/rotate-linux-password.sh

Accessing Secrets via CLI:

If want to get secret from your vault, runvault get .

Run vault kv list to list all the secret engine and path in the Vault.

In this particular example, our secret password is stored in the following path:

/systemcreds/linux/YOURHOSTNAME/USERNAME_creds

Therefore, our passwords stored in the Vault can be get by:

vault kv get /systemcreds/linux/YOURHOSTNAME/USERNAME_creds/