Understanding Bastions Hosts
SSH tunneling is a swiss army knife tool that can be used as an alternative to a VPN, with well known protocols and native support via Linux.
In this article, I will cover some simple use cases, rather than the theory behind SSH tunneling.
The Bastion host is named aptly, it should be a fortification against the wild west of the internet.
A hardened Linux server will typically be deployed at the edge of a network, with SSH exposed in order to provide SSH tunneling into the internal network — this is commonly referred to as a “Bastion” or “Jump” host.
Typically, the hardening process will cover removing root and shell access, disabling password authentication, restricting access to public IPs, defining a regular patching schedule, vulnerability scanning and intrusion protection.
Additionally, it provides a central point where administrative access can can be allowed— managing whitelists across many hosts can spiral out of control.
Local Port Forwarding
Consider the following network in AWS. It’s 4am, you’re on call working for DevOps MSP, and you’ve just received a call from a customer you haven’t worked with before stating that their web server is down. You have the following diagram and PEM keys for each instance in your credential store.
You attempt to SSH into the WEB01 on the public IP 220.127.116.11, however it times out.
ssh email@example.com -i ~/web01.pem
Logging in to the AWS console you find the following Security Group rule, which shows that only port 443 is open to the world.
There is no site to site or access VPN , so at this point you might resign yourself to being unable to access the instance.
However, looking at the security group rules for the BASTION01, you find the following — 18.104.22.168/32 is the DevOps MSP Office IP!
From your laptop, you jump on the office VPN, and issue the following command
ssh firstname.lastname@example.org -L 2222:10.10.0.44:22 -i ~/bastion01.pem -N
At this point the SSH connection will appear to hang, as we have opted not to open a shell.
And from a 2nd Terminal window
ssh email@example.com -p 2222 -i ~/web01.pem
Last login: Sat Sep 22 21:55:50 2018 from 22.214.171.124
What you have just achived here is Local Port forwarding over an SSH tunnel through a Bastion host — let’s break down the structure of the commands.
At a high level, the tunnel is created as a local port, 2222 in this case, which is forwarded through the Bastion to port 22 on the local IP of the WEB01 host
— SSH as ec2-user to the Bastion Host
— Create local port on laptop mapped to remote port on the internal IP
— use encryption key for the Bastion host
— Do not open a shell on the Bastion
The second command connects via SSH to the locally created port on your laptop, to connect to the
— SSH as ec2-user, which is the user on WEB01
— use the local port
— use the encryption key for WEB01
Remote Port Forwarding
Remote port forwarding works in reverse, where the forwarded port is created on the remote system, rather than the system initiating the connection.
Consider the following diagram, which shows our AWS infrastructure as per the previous example, but we have an onpemise datacenter.
You have been tasked with transferring a database backup from the onpremise database server to the new DB01 instance in AWS. Due to tight deadlines, this must be performed today.
However, the security team for the datacenter have advised that creating a NAT rule to open SSH on the server to the internet will take a week, and there is no site to site VPN available.
Outbound internet access is unrestricted in the datacenter, via NAT.
We SSH onto the onpremise DB server, locally to the datacenter.
ssh firstname.lastname@example.org -i ~/db01.pem
On the onpremise DB server, we issue the following command to create a reverse tunnel on the Bastion.
ssh -R 2222:localhost:22 email@example.com -i ~/bastion01.pem
At this point we we would be able to SCP the file from the onpremise DB server to the Bastion host
scp -P 2222 /tmp/dbbackup ec2-user@localhost:/tmp
This method can also be used to expose services, such as MySQL, to the remote site without a site to site VPN.
Remote Multi Hop
Typically you do not have Shell access to a Bastion, so would not be able to copy the file from the Bastion to DB server in the previous example. Therefore you must create a multi-hop SSH tunnel.
On the on premise DB server, build a reverse tunnel to the Bastion
ssh -R 2222:localhost:22 firstname.lastname@example.org -i ~/bastion01.pem
On the Bastion, build a reverse tunnel to the AWS DB server, on port 2223, and map to the reverse tunneled port 2222 from the on premise server.
ssh -R 2223:localhost:2222 email@example.com -i ~/db01.pem
On the DB server in AWS, copy the backup from the onpremise DB server, via the Bastion.
scp -p 2223 firstname.lastname@example.org:/tmp/dbbackup /tmp
Generally, there are easier ways of doing this, for example via S3, and this example would be used in highly restrictive networks.
Tunneling — Not just for SSH
These examples can be used with any port, for example 3389 for RDP to a Windows host.
Additionally, forwarded ports do not need to be accessed via SSH, and can be used by a web browser for exmaple.
One specific use case I came across was where an engineer was working on setting up a switch in a datacenter 3000 miles away with only public IPs. Forgetting to set the default gateway, we lost access to the switch.
However, via an SSH tunnel, we gained access to the switch from a Linux host in the same public subnet, even though it did not have internet access.
Thoughts on security…
I commonly see Bastion hosts setup with port 22 exposed to the internet, on the presumption that SSH is secure due to being setup with encryption keys.
This leaves you wide open to attacks against an unpatched SSH service, or denial of service attacks — a sufficent amount of connections could ramp up CPU or fill the disk with logs, causing the host to be unavailable.
Another assumption is that a tool such as fail2ban is sufficent on internet exposed hosts — this should be your last layer of protection, and not the first.
Some brief points on what to focus on in order to secure your Bastion
- Disable root access, use encryption keys
- Disable shell access
- Restrict the SSH port to only trusted public IPs, don’t assume SSH is secure enough to be exposed
- Install an IDS
- Factor in enough disk space for logs, and implement log rotation
- Forward logs to a collector such as Splunk, or Cloudwatch
- Define a regular patching script, and schedule
- Regularly scan the host for vulnerabilities
- Generate seperate keys for each user, to provide an audit trail
- Implement key rotation before it is deployed in production, not as an afterthought
- Automate! If your Bastion is compromised, you want to be able to redeploy as code quickly. Think Cloudformation, Troposphere, or even a simple ASG with bash script deployment
Some open source tools to help, if your budget is small
- IDS —OSSEC
- Vulnerability Scanner — NMAP or OpenSCAP
- Log Collector — Greylog
Note on generating keys: ssh-keygen defaults to rsa 2048-bit by default, if your key is stolen it is possible to bypass the passphrase as MD5 is used to encrypt the private file
Consider using Ed25519, if your host is running OpenSSH 6.5 or higher
ssh-keygen -t ed25519
If RSA is needed for compatibility, use the newer PEM format
Why not a VPN?
This is actually a very good question. One distinct advantage is familiarity — generally adminstrators will be used to working with Linux, and therefore learning a new VPN solution or installing clients is not required.
Additionally, a Linux host provides more flexibility — for example if AD integration is required for compliance, this can easily be rolled into the OS. A simple packet capture for troubleshooting can be achieved with tcpdump, whereas an off the shelf VPN solution may not provide this.
Generally, I would recommend using a UDP VPN solution where the entry point into the network absolutely must be exposed without IP restrictions. UDP is connectionless and makes scanning for an entry point harder.
Amazon have released SSM Session Manager, which provides console access (VMWare ESXi style) to both AWS Instances and soon on premise systems.
The agent supports Linux, Windows and OSX command line access.
This gives a significant advantage over Bastion hosts, as port 22 can be closed off, and all commands and access attempts are logged via IAM — reducing the need for a log collector.
From an change management perspective, access can be provided via a defined change window, and restricted to certain IAM users.
Controlling access to the AWS console via MFA gives an extra layer of protection to get to your instances.