SSH is the defacto standard for system operation and IAC (e.g., Ansible, Chef). It is a big challenge to ensure all the access with proper authorization and authentication. The problem becomes even more complicated when you have multiple VPCs and on-prem colocation. It is a problem with SSH, SSH Key Management, Network Architecture, and Network Security.
The Environment
Let’s define the environment first
- Two VPC or Private Network hosts two infrastructures for two different teams in the same organization (VPC1, VPC2)
- A signal VPN network is set up for both team members to access VPC1 and VPC2
- On top of the VPN network, VPC1 and VPC2 can assume to be isolated networks
- The team need SSH access to authorized hosts after logged in the VPN
For the VPN setup, there are many tutorials for that, or you can reference my previous post here on using Wireguard.
Objective
- Centralized Authorization Management for all the SSH access
- No SSH ports of the working host expose to the public
- Team members workstations in the public network can carry out tasks through Ansible, Chef, and SSH (shell)
Architecture
- SSH Key Management — Bastillion, hosted in VPC1 (bastillion01)
- SSH Jump Host for SSH Proxy in VPC1 (bastillion01)
- Both hosts have SSH port (22) access to all hosts in VPC1 and VPC2
Setup
Key Management — Bastillion
I put the bastillion web inside the VPC network and expose the port on private IP, to restrict access from the public and only reachable from the VPN network.
Software Setup, please refer to the doc https://www.bastillion.io/docs/. And then, we need to set up SSL Cert and Let’s Encrypt SSL Certification.
First, generate your Let’s Encrypt SSL Certification if you own a domain. If not, please use the self-signed certification.
## Locate the fullchain.pem and privkey.pem
## cd to the bastillion directoryopenssl pkcs12 -export -inkey privkey.pem -in fullchain.pem -out bastillion.pkcs12keytool -importkeystore -srckeystore bastillion.pkcs12 -srcstoretype PKCS12 -destkeystore keystorecp keystore jetty/etc/## Get the OBF value
java -cp jetty/lib/jetty-util-9.4.29.v20200521.jar org.eclipse.jetty.util.security.Password <password entered in pkcs12>## Replace OBF value of KeyStorePassword and keyManagerPassword in jetty/etc/jetty-ssl-context.xml## Replace OBF value of KeyStorePassword, KeyManagerPassword, TrustStorePassword in jetty/modules/ssl.mod## Kill Basillion Process if any./startBastillion.sh &
After Bastillion installation, all VMs should have the Bastillion public key added in ~/.ssh/authorized_keys. Then you can manage the SSH keys for the users and hosts in VPC1 and VPC2*.
Bastillion also comes with an excellent Web SSH termination, so now team members can access each host by logging in the web frontend.
SSH Forward Setup
Up till now, team members can access hosts in VPC1 as long as his public key is on the hosts. Depending on the VPN setup, they may not be able to access VPC2. If we allow SSH access to every host, it won’t be easy to control and audit the SSH access, as we need to implement safeguard mechanics (e.g., Fail2Ban) in each host.
SSH Forward and SSH Proxy come to recure. They do not harden the security but centralize all the network traffic in a host to build further mechanic on access control and audit purpose. For example, to log all the SSH access to each host.
Firstly, allow SSH port network access from the Jump Host in VPC1 to VPC2. And in the workstation. Then add to ~/.ssh/config in your workstation.
## Jump Host Config
Host bastillion01
ForwardAgent yes
HostName bastillion01.mydomain.local
User jumper_userid
Port 22
IdentityFile ~/.ssh/user_private.key## All host to use bastillion01
Host *.mydomain.local
User host_userid
ProxyCommand ssh -W %h:%p bastillion01
IdentityFile ~/.ssh/user_private.key
And now ssh worker01.mydomin.local should work. Ansible and Chef should works in the same way as they both respect ssh config.
More advanced setup on SSH port forwarding can be configured to provide not just SSH access but also other TCP access like HTTP, MYSQL, and Mongo.
Discussion and Remark
Placing a jumper host in VPC1 is for ease of understanding. It may be placed in another VPC if the readers want to archive a higher level of network isolation.
Apart from letting workstation to have direct SSH access, there would be another way like remote desktop or develop on jump host. It depends on how the developers and SREs manage their toolset and their access pattern. Something needs further discussion.
In the area access monitoring and audit needs, there is some commercial software for the purpose. I have used ManageEngine PMP, and it has some excellent features like screen recording the SSH session.
Suggestion are welcome
I hope this post is helpful. Please feel free to send me comments. Also, welcome for discussion on other complicated cases in the DevOps area.
