Understanding Bastions Hosts

SSH tunneling is a swiss army knife tool that can be used as an alternative to a VPN, with well known protocols and native support via Linux.

In this article, I will cover some simple use cases, rather than the theory behind SSH tunneling.

Bastion Hosts

The Bastion host is named aptly, it should be a fortification against the wild west of the internet.

A hardened Linux server will typically be deployed at the edge of a network, with SSH exposed in order to provide SSH tunneling into the internal network — this is commonly referred to as a “Bastion” or “Jump” host.

Typically, the hardening process will cover removing root and shell access, disabling password authentication, restricting access to public IPs, defining a regular patching schedule, vulnerability scanning and intrusion protection.

Additionally, it provides a central point where administrative access can can be allowed— managing whitelists across many hosts can spiral out of control.

Local Port Forwarding

Consider the following network in AWS. It’s 4am, you’re on call working for DevOps MSP, and you’ve just received a call from a customer you haven’t worked with before stating that their web server is down. You have the following diagram and PEM keys for each instance in your credential store.

You attempt to SSH into the WEB01 on the public IP 38.45.55.44, however it times out.

ssh ec2-user@38.45.55.44 -i ~/web01.pem

Logging in to the AWS console you find the following Security Group rule, which shows that only port 443 is open to the world.

There is no site to site or access VPN , so at this point you might resign yourself to being unable to access the instance.

However, looking at the security group rules for the BASTION01, you find the following — 32.33.44.43/32 is the DevOps MSP Office IP!

From your laptop, you jump on the office VPN, and issue the following command

ssh ec2-user@38.45.55.44 -L 2222:10.10.0.44:22 -i ~/bastion01.pem -N

At this point the SSH connection will appear to hang, as we have opted not to open a shell.

And from a 2nd Terminal window

ssh ec2-user@127.0.0.1 -p 2222 -i ~/web01.pem

Connected!

Last login: Sat Sep 22 21:55:50 2018 from 48.44.55.44
[ec2-user@web01 ~]$

What you have just achived here is Local Port forwarding over an SSH tunnel through a Bastion host — let’s break down the structure of the commands.

At a high level, the tunnel is created as a local port, 2222 in this case, which is forwarded through the Bastion to port 22 on the local IP of the WEB01 host

ssh ec2-user@38.45.55.44

— SSH as ec2-user to the Bastion Host

-L 2222:10.10.0.44:22

— Create local port on laptop mapped to remote port on the internal IP

-i ~/bastion01.pem

— use encryption key for the Bastion host

-N

— Do not open a shell on the Bastion

The second command connects via SSH to the locally created port on your laptop, to connect to the

ssh ec2-user@127.0.0.1

— SSH as ec2-user, which is the user on WEB01

-p 2222

— use the local port

-i ~/web01.pem

— use the encryption key for WEB01

Local Multihop

Remote Port Forwarding

Remote port forwarding works in reverse, where the forwarded port is created on the remote system, rather than the system initiating the connection.

Consider the following diagram, which shows our AWS infrastructure as per the previous example, but we have an onpemise datacenter.

You have been tasked with transferring a database backup from the onpremise database server to the new DB01 instance in AWS. Due to tight deadlines, this must be performed today.

However, the security team for the datacenter have advised that creating a NAT rule to open SSH on the server to the internet will take a week, and there is no site to site VPN available.

Outbound internet access is unrestricted in the datacenter, via NAT.

We SSH onto the onpremise DB server, locally to the datacenter.

ssh db-user@10.55.10.44 -i ~/db01.pem

On the onpremise DB server, we issue the following command to create a reverse tunnel on the Bastion.

ssh -R 2222:localhost:22 ec2-user@38.45.55.43 -i ~/bastion01.pem

At this point we we would be able to SCP the file from the onpremise DB server to the Bastion host

scp -P 2222 /tmp/dbbackup ec2-user@localhost:/tmp

This method can also be used to expose services, such as MySQL, to the remote site without a site to site VPN.

Remote Multi Hop

Typically you do not have Shell access to a Bastion, so would not be able to copy the file from the Bastion to DB server in the previous example. Therefore you must create a multi-hop SSH tunnel.

On the on premise DB server, build a reverse tunnel to the Bastion

ssh -R 2222:localhost:22 ec2-user@38.45.55.43 -i ~/bastion01.pem

On the Bastion, build a reverse tunnel to the AWS DB server, on port 2223, and map to the reverse tunneled port 2222 from the on premise server.

ssh -R 2223:localhost:2222 ec2-user@38.45.55.43 -i ~/db01.pem

On the DB server in AWS, copy the backup from the onpremise DB server, via the Bastion.

scp -p 2223 db-user@127.0.0.1:/tmp/dbbackup /tmp

Generally, there are easier ways of doing this, for example via S3, and this example would be used in highly restrictive networks.

Tunneling — Not just for SSH

These examples can be used with any port, for example 3389 for RDP to a Windows host.

Additionally, forwarded ports do not need to be accessed via SSH, and can be used by a web browser for exmaple.

One specific use case I came across was where an engineer was working on setting up a switch in a datacenter 3000 miles away with only public IPs. Forgetting to set the default gateway, we lost access to the switch.

However, via an SSH tunnel, we gained access to the switch from a Linux host in the same public subnet, even though it did not have internet access.

Thoughts on security…

I commonly see Bastion hosts setup with port 22 exposed to the internet, on the presumption that SSH is secure due to being setup with encryption keys.

This leaves you wide open to attacks against an unpatched SSH service, or denial of service attacks — a sufficent amount of connections could ramp up CPU or fill the disk with logs, causing the host to be unavailable.

Another assumption is that a tool such as fail2ban is sufficent on internet exposed hosts — this should be your last layer of protection, and not the first.

Some brief points on what to focus on in order to secure your Bastion

  • Disable root access, use encryption keys
  • Disable shell access
  • Restrict the SSH port to only trusted public IPs, don’t assume SSH is secure enough to be exposed
  • Install an IDS
  • Factor in enough disk space for logs, and implement log rotation
  • Forward logs to a collector such as Splunk, or Cloudwatch
  • Define a regular patching script, and schedule
  • Regularly scan the host for vulnerabilities
  • Generate seperate keys for each user, to provide an audit trail
  • Implement key rotation before it is deployed in production, not as an afterthought
  • Automate! If your Bastion is compromised, you want to be able to redeploy as code quickly. Think Cloudformation, Troposphere, or even a simple ASG with bash script deployment

Some open source tools to help, if your budget is small

  • IDS —OSSEC
  • Vulnerability Scanner — NMAP or OpenSCAP
  • Log Collector — Greylog

Note on generating keys: ssh-keygen defaults to rsa 2048-bit by default, if your key is stolen it is possible to bypass the passphrase as MD5 is used to encrypt the private file

Consider using Ed25519, if your host is running OpenSSH 6.5 or higher

ssh-keygen -t ed25519

If RSA is needed for compatibility, use the newer PEM format

ssh-keygen -o

Why not a VPN?

This is actually a very good question. One distinct advantage is familiarity — generally adminstrators will be used to working with Linux, and therefore learning a new VPN solution or installing clients is not required.

Additionally, a Linux host provides more flexibility — for example if AD integration is required for compliance, this can easily be rolled into the OS. A simple packet capture for troubleshooting can be achieved with tcpdump, whereas an off the shelf VPN solution may not provide this.

Generally, I would recommend using a UDP VPN solution where the entry point into the network absolutely must be exposed without IP restrictions. UDP is connectionless and makes scanning for an entry point harder.

The Future?

Amazon have released SSM Session Manager, which provides console access (VMWare ESXi style) to both AWS Instances and soon on premise systems.

The agent supports Linux, Windows and OSX command line access.

This gives a significant advantage over Bastion hosts, as port 22 can be closed off, and all commands and access attempts are logged via IAM — reducing the need for a log collector.

From an change management perspective, access can be provided via a defined change window, and restricted to certain IAM users.

Controlling access to the AWS console via MFA gives an extra layer of protection to get to your instances.