HashiCorp Vault which we use quite often (and we mentioned HashiCorp Vault before) for managing secrets by microservices, is a great tool to manage and audit sensitive information and access credentials.
One of our clients asked us to investigate how HashiCorp Vault can be used to audit the access of admins to servers. They currently have a mixed bag, some admins use a password that was shared with them while others often manually deploy their ssh public keys onto servers. Neither is a great security practice, especially for a very large organization that needs tighter control and visibility on who gets access where. On the other hand, the current modus operandi works for them and people are able to get things done without hassle.
There are two main methods that HashiCorp Vault supports in terms of managing SSH credentials:
- One-time Passwords (OTP)
- Signed SSH Keys.
We configured a new instance of HashiCorp Vault using Ansible and the Active Directory (aka LDAP) authentication method. We also enabled the Vault UI that was recently added to the open-source version of Vault. This way users can log in with their windows credentials and ask Vault for secrets.
So the first way, OTP, requires to install the Vault SSH Helper on each server so that when the admin is connecting this helper on the server can verify the temporary password the user used by asking Vault.
On the downside, admins that already have their Windows SSH Clients (Putty, SecureCRT, MobaXterm, …) configured to login with the same password (or key) each time will need to ask Vault for OTP before each login. This will seriously hinder their flow of work, and such a suggestion might not be taken favorably by people who are responsible for making sure all servers actually work. These admins are also in a position to ignore the suggestion of using such a system if they find it inconvenient.
The second way, Signed SSH Keys, only requires to add a single line to the server’s /etc/ssh/sshd_config file saying TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem. This file is provided by Vault and is used by OpenSSH to verify the keys used by users. When a key is verified, the user is granted access. But the key must come with an additional part, the signed certificate of the key. And Vault is the one doing the signing.
So each time Vault signs some SSH key, this signed certificate is only valid for a restricted period of time, say 15 minutes. During this time this certificate is valid, outside of this time it is not.
Having admins connect with SSH Keys is a great practice, much better than using passwords. And having just a single line on the server, instead of multiple keys in various ~/.ssh/authorized_keys files all over the place, is a great solution.
Vault even provides an easy way for users to ask for these signed keys by using the vault ssh command. Once a user has logged into vault using vault login with his LDAP/AD credentials, vault ssh gives access to all the servers that Vault’s policy gives him access to.
Easy!
But wait, there is a small problem here as well. As I mentioned before, admins have their comforts, and their comforts include a Windows SSH Client. One of the main features of such a client is the list of sessions and the User Interface (UI) where the admin can just double-click a session to connect. Unfortunately, none of these clients (as far as we know) supports Vault yet. And even worse, most of these clients don’t even support Signed SSH Keys, since that is a feature of OpenSSH and windows clients often have their own flavor of SSH agent.
Windows 10 is coming up with a native OpenSSH client as well, which means that Cygwin and/or Windows Subsystem for Linux are not a requirement anymore. But where is the pretty UI with the long list of servers!?
We asked around, and the consensus is apparently “just force them to do it your way, it is security you’re talking about”. While this is true, hurting people’s work is a great way to increase resistance to change — we want to find a way to go the extra mile and find a win-win-win solution.
Please do comment if you can suggest a worthy solution that is win-win. We would love to hear from you!
Discussions around the internets that we asked -
Originally published at www.prodops.io.
