HashiCorp Vault SSH CA and Sentinel
Overview
In this writeup, we will explore the HashiCorp Vault SSH CA dynamic secret engine in combination with the HashiCorp Sentinel integration.
We will walk through a simple example with multiple Vagrant virtual machines to show how the SSH CA method works.
In addition, we will show how Vault’s Sentinel integration can be leveraged to enforce an end user’s short-lived SSH key’s valid principal (user) matches their Vault identity (LDAP username for example).
Finally, we will show auditing examples for this SSH CA workflow.
Current Issues
Managing SSH keys for hundreds or thousands of users can be a massive and time-consuming job for infrastructure operators. SSH keys must be maintained across virtual machines and kept up to date per user. Managing these keys is made even more difficult because certain users should not have access to certain machines. To make matters worse, access must be revoked as soon as a user leaves. These problems can all lead to error-prone processes that are either done manually, via bash scripts, config management, or some other solution.
Vault’s SSH secret engine takes a different (dynamic/on-demand) approach to SSH keys.
Imagine a contractor begins work at a company. We would like to give them access to a subset of machines (based on their “role” as a contractor), but not worry about managing an associated (long-lived) SSH key for the duration of their contract. Instead of managing a unique private key for each individual contractor, we can setup Vault as an SSH CA that can create signed certificates for the contractors to SSH to target hosts with. This simplifies the management of trusted certificates on hosts down to one SSH CA public key per role (contractors in this case) on a target host instead of an SSH key per user. Diagrams will explain this concept further below.
Vault’s dynamic SSH keys can also be made short-lived. This feature is built into OpenSSH and allows us to keep access windows to resources time-boxed. So instead of managing indefinitely lived SSH keys, we can just give out SSH keys are that are good for say an hour (or less!). If users want to re-login to a destination box after key expiration, they must first re-authenticate to vault and request the generation of a new key for SSH (which is audited in Vault).
Not only can Vault sign client keys, but we can also leverage it for generating Host keys. Host keys work inversely and allow end users to verify the identity of a target machine they are trying to access via SSH. So instead of just approving the authenticity of host message (below), we can setup host keys to verify that destination servers are trusted. This works in the same vein as public website TLS certificates do with your browser. Host key checking will not be covered in this blog (for brevity), however, details are described here.
The authenticity of host ‘192.168.0.100 (192.168.0.100)’ can’t be established.RSA key fingerprint is 3f:1b:f4:bd:c5:aa:c1:1f:bf:4e:2e:cf:53:fa:d8:59.Are you sure you want to continue connecting (yes/no)?
Many other name brand companies have written about similar approaches to leveraging an SSH CA.
How it works
There are several great blogs and videos out there on explaining how an SSH CA works. So I'll leave links to those and briefly explain things here.
Uber SSH Certificate Authority
Erik Rygg (HashiCorp) on the Vault SSH secret engine
SSH leverages asymmetric public key cryptography (public/private keys) to authenticate clients to hosts and vice-versa.
We can set up Vault to act as the Certificate Authority for our SSH certificates (SSH leverages asymmetric public key cryptography (public/private keys) to authenticate clients to hosts and vice-versa). In the client signing case, once configured we can then distribute the CA public key to hosts we want users to be able to SSH into. That public key is added as a trusted user key in the destination box’s sshd configuration. Once users login, sshd will check the key signatures of the users key against the trusted CA key to verify access.
We can take things a step further by creating multiple “roles” in Vault for restricting access to subsets of machines. For example, we may setup a web prod role and a development prod role. Each role has a separate CA public and private key. We then distribute the public keys to the proper destination machines. This greatly simplifies key management as we only need to manage the CA key instead of individual keys for every user.
The below diagram illustrates this configuration. We will discuss Sentinel and LDAP later in this post.
Roles are used to restrict access to subsets of machines.
So how does usage work? When an end user wants to SSH to a destination box they will first authenticate to Vault (with LDAP for example). If given ACL access to an SSH secret engine role, they can then submit their public key to Vault for signing. If authorized, a subsequent new signed public key (certificate) is returned. The user can then use this key to SSH into the remote servers. The returned signed keys can be given a “time to live” (ttl) to restrict SSH access to time-boxed windows. This restriction is enforced by OpenSSH.
Demo
Now that we’ve discussed the benefits of managing SSH dynamically via a CA, let's walk through a demo and explore the resulting audit trail.
Vault SSH CA Documentation https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-certificates.html
https://www.vaultproject.io/api/secret/ssh/index.html
Demo Code (shared service account example)
Start by cloning the code from the Vault-guides repository:
https://github.com/hashicorp/vault-guides
We will be leveraging the example from this sub-directory:
https://github.com/hashicorp/vault-guides/tree/master/identity/ssh-ca/vagrant-local
Follow the instructions here:
After performing the instructions, you should be able to successfully login to the Vault server from the client server as the Vagrant user.
How did this work? (Client verification)
On the Vault configuration side we first setup the SSH-CA (host verification is redacted).
# Vault server
# Mount a backend's instance for signing client keys
vault secrets enable -path ssh-client-signer ssh# Configure the client CA certificate
vault write -f -format=json ssh-client-signer/config/ca | jq -r '.data.public_key' >> /home/vagrant/trusted-user-ca-keys.pem# Add the SSH CA public key to our trusted keys
sudo mv /home/vagrant/trusted-user-ca-keys.pem /etc/ssh/trusted-user-ca-keys.pem
echo "TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem" | sudo tee --append /etc/ssh/sshd_config# Allow host certificate to have longer TTLs
vault secrets tune -max-lease-ttl=87600h ssh-host-signer# Create a role to sign host keys, note the default_user
# is a shared service account
# We will show this example for individual accounts in the next section
vault write ssh-host-signer/roles/hostrole ttl=87600h \
allow_host_certificates=true \
key_type=ca \
allowed_domains="localdomain,example.com" \
allow_subdomains=trueecho '
{
"allow_user_certificates": true,
"allowed_users": "*",
"default_extensions": [
{
"permit-pty": ""
}
],
"key_type": "ca",
"key_id_format": "vault-{{role_name}}-{{token_display_name}}-{{public_key_hash}}",
"default_user": "vagrant",
"ttl": "30m0s"
}' >> /home/vagrant/clientrole.json# Create a role to sign client keys
vault write ssh-client-signer/roles/clientrole @/home/vagrant/clientrole.json# Restart sshd
sudo systemctl restart sshd
The end user then requests a certificate from Vault for the shared service account.
# Client
# Authenticate to Vault
vault login -method=userpass username=johnsmith password=testcat /home/vagrant/.ssh/id_rsa.pub | \
vault write -format=json ssh-client-signer/sign/clientrole public_key=- \
| jq -r '.data.signed_key' > /home/vagrant/.ssh/id_rsa-cert.pubchmod 0400 /home/vagrant/.ssh/id_rsa-cert.pub# Connect to Vault host as vagrant user
ssh -i /home/vagrant/.ssh/id_rsa -i /home/vagrant/.ssh/id_rsa-cert.pub vagrant@vault
In the next section, we will see how to leverage the SSH CA for creating user-specific certificates instead of a shared service account.
Taking it a step further (Sentinel)
What if we want to leverage the functionality for real “user” named accounts and not a shared service account?
We can leverage the Vault Enterprise Sentinel integration to enforce that a user’s LDAP username matches that user’s SSH username. In the below example, I use the “userpass” authentication method instead of LDAP for brevity. However, if you would like to see a quick setup of using Vault and LDAP please refer to this great blog by Vinnie Ramirez.
We will start with a similar demo from above and will layer Sentinel on top. This assumes you have a Vault Enterprise binary enabled as an environment variable.
I’ve created a forked repo to demonstrate this Sentinel example. Note, I’ve also removed the host key verification to simplify the code. Clone this fork for the Sentinel example:
https://github.com/Andrew-Klaas/vault-guides
Then navigate to:
https://github.com/Andrew-Klaas/vault-guides/tree/master/identity/ssh-ca/vagrant-local
We follow the same steps as before but notice the following additions of Sentinel in the Vault server setup script.
#Sentinel
cat <<EOF>> ssh-username-restrict.sentinel
import "strings"
username_match = func() {
# Make sure there is request data
if length(request.data else 0) is 0 {
return false
}
# Make sure request data includes username
if length(request.data.valid_principals else 0) is 0 {
return false
}
# Make sure the supplied username matches the user's name
if request.data.valid_principals != identity.entity.aliases[0].name {
return false
}
return true
}
main = rule {
strings.has_prefix(request.path, "ssh-client-signer/sign/clientrole") and username_match()
}
EOFPOLICY=$(base64 ssh-username-restrict.sentinel); vault write sys/policies/egp/ssh-username-restrict \
policy="${POLICY}" \
paths="ssh-client-signer/sign/clientrole" \
enforcement_level="hard-mandatory"
The above code snippet configures Sentinel to reject any SSH certificate requests to Vault from end users that do not demand certificates that match said user’s userpass (or LDAP) username.
Next, an end user would use Vault in the following manner:
vault login -method=userpass username=johnsmith password=testsudo rm /home/vagrant/.ssh/id_rsa-cert.pub
cat /home/vagrant/.ssh/id_rsa.pub | \
vault write -format=json ssh-client-signer/sign/clientrole valid_principals=johnsmith public_key=- \
| jq -r '.data.signed_key' > /home/vagrant/.ssh/id_rsa-cert.pubsudo chmod 0400 /home/vagrant/.ssh/id_rsa-cert.pubssh -i /home/vagrant/.ssh/id_rsa -i /home/vagrant/.ssh/id_rsa-cert.pub johnsmith@vault
Last login: Tue Sep 24 15:35:07 2019 from 192.168.50.101[johnsmith@vault ~]$ pwd
/home/johnsmith
Note the “valid_principal” must contain the user’s LDAP/userpass/etc name correctly. If they did not match, Sentinel would reject the request for a certificate.
The audit log will now reflect the user SSH’ing as themselves with the added bonus of not needing to manage their SSH key. We can set the SSH key lifetime for 30 minutes to increase security.
Auditing
One of the most important questions at this point is: what does auditing look like?
1. First and foremost: Vault’s audit log will reflect that “johnsmith” requested an SSH key from Vault.
From the Vault audit log:
{
"time": "2019-09-24T15:14:40.67742241Z",
"type": "request",
"auth": {
"client_token": "hmac-sha256:d480cf84fb524bda7bf7017703959034af2bf3338087c7c6cab5a70f78a9403e",
"accessor": "hmac-sha256:45838cf7b568d9085362d216431e7deb9bb521423f972fb852f908d466685ea1",
"display_name": "userpass-johnsmith",
"policies": [
"default",
"user"
],
"token_policies": [
"default",
"user"
],
"metadata": {
"username": "johnsmith"
},
"entity_id": "ab60db90-9ec9-3c27-bb8f-70eb034718cc",
"token_type": "service"
},
"request": {
"id": "f5b9b4ba-6a0e-c549-ca43-cfdd8d8efa14",
"operation": "update",
"client_token": "hmac-sha256:d480cf84fb524bda7bf7017703959034af2bf3338087c7c6cab5a70f78a9403e",
"client_token_accessor": "hmac-sha256:45838cf7b568d9085362d216431e7deb9bb521423f972fb852f908d466685ea1",
"namespace": {
"id": "root"
},
"path": "ssh-client-signer/sign/clientrole",
"data": {
"public_key": "hmac-sha256:ea8df87d5df3de10ed5f13fbe8e9d3b544e5d3127744b986ca5122c3b1f27a8e",
"username": "hmac-sha256:6a35732c1cbd35f5718c2b31d182dd5744ad59b1ffbb77ce64be92aacb793e0f",
"valid_principals": "hmac-sha256:6a35732c1cbd35f5718c2b31d182dd5744ad59b1ffbb77ce64be92aacb793e0f"
},
"remote_address": "192.168.50.101"
}
}
For a shared service account/default user2. Inspecting the signed SSH keys returned from Vault:
$ ssh-keygen -Lf /home/vagrant/.ssh/id_rsa-cert.pub
/home/vagrant/.ssh/id_rsa-cert.pub:
Type: ssh-rsa-cert-v01@openssh.com user certificate
Public key: RSA-CERT SHA256:R00FTWX0aVSbSAXL+2ZN2NIKtjyRpWBk6cc89Yu7qTU
Signing CA: RSA SHA256:7fXfp/Aj6Wr18P1VbtHNslbMOUrjSLQpkgnfcUdxk4Q
Key ID: "johnsmith"
Serial: 14304612407374961173
Valid: from 2019-09-22T17:51:25 to 2019-09-22T18:21:55
Principals:
vagrant
Critical Options: (none)
Extensions:
permit-ptyFor a unique human user account (Sentinel enforcement of LDAP username Valid Principal):
$ ssh-keygen -Lf /home/vagrant/.ssh/id_rsa-cert.pub
/home/vagrant/.ssh/id_rsa-cert.pub:
Type: ssh-rsa-cert-v01@openssh.com user certificate
Public key: RSA-CERT SHA256:AKrecFXwXNjGiT5ttxTf0KrNJtZ+LNOe78hTq38fpq4
Signing CA: RSA SHA256:+LBCed4Xyu2WiPCVvlfirHOdmXCsB+iie8rTmGJ7faI
Key ID: "vault-clientrole-userpass-johnsmith-00aade7055f05cd8c6893e6db714dfd0aacd26d67e2cd39eefc853ab7f1fa6ae"
Serial: 13310695863379851714
Valid: from 2019-09-24T15:14:10 to 2019-09-24T15:44:40
Principals:
johnsmith
Critical Options: (none)
Extensions:
permit-pty3. Auditing on the destination boxes.
For a shared service account/default user:
$ journalctl -xn -u sshd | less
Sep 22 17:47:39 vault sshd[7280]: Accepted publickey for vagrant from 192.168.50.101 port 34704 ssh2: RSA-CERT ID vault-clientrole-userpass-johnsmith-474d054d65f469549b4805cbfb664dd8d20ab63c91a56064e9c73cf58bbba935 (serial 10118457451169283482) CA RSA SHA256:7fXfp/Aj6Wr18P1VbtHNslbMOUrjSLQpkgnfcUdxk4QFor an actual human user account (Sentinel enforcement of LDAP username Valid Principal):
Sep 22 18:58:45 vault sshd[7359]: Accepted publickey for johnsmith from 192.168.50.101 port 55490 ssh2: RSA-CERT ID vault-clientrole-userpass-johnsmith-f4daddc21c6b73792bb354e508bc64881dc9922c69aefe5f3265d3f6ff78cad1 (serial 14588164476024091953) CA RSA SHA256:foz3A23PrgswbQ7R+Yz5verFcOvvICvAzH7WX0twv7USummary
In this post, we reviewed the purpose and benefits of Vault’s SSH CA secret engine. We’ve utilized this engine to simplify SSH key management down to trusted CA keys instead of managing hundreds or thousands of keys for our users. This feature also allows us to enforce the creation of short-lived SSH keys and thus time-box how long users have access to servers.
Additionally, we’ve shown that Sentinel can be used in combination with Vault to enforce usage patterns around the SSH secret engine. An example of which was to only allow users to create SSH keys with their LDAP username as the valid principal.
Other References
Diagrams by Brian Green (Director of Implementation Services at HashiCorp).
