An AWS Bastion Host unreachable from the internet — Part 1
Part 1: How can SSH remote server work without IP nor SSH key and how to configure the SSH client
In this article, we’ll reinforce the security of our AWS infrastructure by making our Bastion server unreachable from the internet.
What’s a Bastion?
On the Cloud, there are several resources that are not meant to be directly accessed from the internet. That’s the case for your databases, for example. These resources could be accessed by other software components, internally, but not by any human (GDPR restrictions, risk of human factor, …) and especially, not publicly, for obvious security reasons.
However, there are some cases where you still need to connect to these resources, mainly for debugging purposes. This is where the Bastion comes in, by allowing you to remotely connect to what should never be accessed.
The Bastion Host is a virtual machine accessible from the outside world that is very sensitive for external attacks. Usually, the Bastion host requires an SSH key to be accessed (that brings additional challenges : this SSH Key could be leaked at some point, we need to ensure distribution) and, in a best-case scenario, is protected with source IP whitelisting.
Is there a way to do it even better, removing these SSH keys hanging around and enforcing access control on the server? A server without a keyhole would be way more difficult to enter!
We can do much better, by making our Bastion server unavailable from the Internet and by provisioning our EC2 machine without any SSH key. To achieve this, we’ll split the article in two parts :
- In the first part, we’ll explain how to reach an unreachable server, with no public connectivity and how to connect in SSH without any SSH key deployed on the server. Once we’ll have that covered, we’ll see how to set up a client to allow such a secure connection.
- In the second part, we’ll comment on the required AWS infrastructure, with each component and how to deploy the server using Terraform.
SSM : Let’s make the Bastion unreachable!
Our first goal is to remove any public connectivity from the Bastion host. That means no public IP address, no load balancer, no DNS entry, no Security Group allowing any incoming connection.
How to contact a machine that cannot be contacted? The answer is simple, wait for the machine to contact you.
Each EC2 virtual machine has a local loop (127.0.0.1). It also has a special interface, invisible on the Operating System Level, that only AWS can whisper some traffic to using the Instance Metadata Service. You’ve already used this service when you’ve requested 169.254.169.254 to get some meta-data, for example.
The AWS SSM agent is a piece of software, originally for Amazon Linux and Ubuntu, that can interact in many ways with AWS Systems Manager, through the Instance Metadata Service Calls (IMDS v2) — they’re HTTP requests in the end. This AWS SSM agent can listen to these specific requests and can act on an EC2 machine once installed and if it’s allowed to do so.
SSM Tunnelling
The only required network rule for the Bastion is a public access to the internet via an Internet Gateway. You can have no ingress rule on the Security Group assigned to the Bastion host.
The connection type can be pure-SSM (no authentication key would be required) or SSH tunnelled inside an SSM connection. A pure SSM connection is ok as long as you need to connect to a single instance without jumping. You can do a local forward on any port of the instance where you’re connected to.
aws ssm start-session --target i-0d27fb6b7fd9cdeed
SSH over SSM Tunnelling
If you need to make some SSH bounce towards other hosts or if you need to authenticate through an SSH key to these hosts, the sole SSM protocol will not be enough; here comes the SSH-OVER-SSM tunnelling.
The command and the usage are slightly different.
aws ssm start-session --target i-0d27fb6b7fd9cdeed --document-name AWS-StartSSHSession --region=eu-west-1 --parameters 'portNumber=22'
This command cannot be used as it is. It’s meant to be used within an SSH client, as a proxy command to retrieve a SSH socket. Then an SSH client needs to send the authentication SSH key and/or ask for SSH Secure tunnelling or remote TCP forwarding.
You can either enter the full command at each time you want to connect, with the “-o ProxyCommand” command switch, telling how you want to establish the SSH connection
ssh ec2-user@bastion -i ~/.ssh/my_super_secret_key.pem -o "ProxyCommand=aws ssm start-session --target i-0d27fb6b7fd9cdeed --document-name AWS-StartSSHSession --parameters 'portNumber=22'"However, you can make things way simpler and add this into your ~/.ssh/config
# SSH over Session Manager
host i-* mi-*
ProxyCommand sh -c "aws ssm start-session --target %h
--document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
That’s now better. We contact the bastion host via its instance identifier and we don’t require any public connectivity anymore although we still need to provide an SSH key to connect (via -i configuration switch).
Having SSH keys to connect to the server is painful. The private key can be lost, stolen or in need to be stored somewhere and shouldn’t be shared with all your colleagues. In ideal conditions, you also need a different SSH key per server, which makes the situation way more complicated to manage.
EC2-Instance-Connect : Let’s throw away the SSH keys !
Usually, when you provision an EC2 machine, you tell AWS that you want the system to be installed with one SSH public key that you have created previously on the AWS account. AWS only knows the public key. The private key is kept by the user.
Our Bastion can be provisioned without any SSH key and can be still accessed via SSH. The whole trick is to generate a SSH key on the fly, that will be valid for 60 seconds on the server. This is achievable thanks to the ec2-instance-connect feature from AWS, introduced in June 2019.
The first step is to generate an SSH RSA keypair. This key pair will not be passphrase-protected and will be stored on a temporary folder because it is not meant to be persistent.
ssh-keygen -t rsa -f /tmp/ssm_ic -N ''
Once the key is generated, there are two files /tmp/ssm_ic and /tmp/ssm_ic.pub being created on your hard drive. The file with the .PUB extension is the public key we want to send to the remote server.
Let’s send the public key to the remote server now.
aws ec2-instance-connect send-ssh-public-key \
--instance-id i-0dbe593ea67afde9a \
--availability-zone eu-west-1c \
--instance-os-user ec2-user \
--ssh-public-key file:///tmp/ssm_ic.pubOn a given instance, that sits on a particular AZ, we’re bounding an existing user to an SSH public key.
If the operation succeeds, a message is displayed. You have to hurry as you only have 60 seconds to attempt a connection to the server. After 60 seconds, the remote instance will delete the public key you’ve just sent. If you don’t succeed to connect within this period, you need to reiterate and resend your public key to the server.
A couple of things to note here:
- The EC2 instance doesn’t require any internet access for accepting the SSH key.
- The deletion of your key is optional. You could technically reuse the same key pair each time but it’s safer to recreate one.
- The policy required for the server is different than for SSM tunnelling.
EC2-Instance-Connect (IC) works independently from SSM. You can initiate a standard SSH connection (with IP connectivity) or instantiate a SSH-over-SSM connection. That would work the same.
However, you need to know the instance-id and the availability-zone of the EC2 you want to push the key to (you cannot invoke ec2-instance-connect with an IP address).
SSM + Instance-Connect : SSH without SSH keys nor public connectivity
It’s time to pack everything together. If the separated concepts shown before are understood, the rest isn’t very complicated.
First, let’s assume that you don’t know by heart what are the 16-hexadecimal characters instance IDs you would like to use, nor in which Availability Zone the instances are located.
However, you’re likely to know the administrative name of your server. In our case, the server name is Bastion. This will be given as a script argument later on. We can retrieve these instance-id and AZ with a bit of filtering. We select the first running instance attributes returned by the AWS API.
Then we generate the SSH key pair without outputting the art key to stdout. It’s important to ensure that no previous ssh existed because that would cause the ssh-keygen programme to fail.
Once the key pair is generated, we send the public key to the remote instance. Again, we ignore the status message from AWS client as we want to automate operations.
The next step is to initiate the SSH connection to the remote server, on the remote-port 22. Keep in mind that you need to match an existing user on the server (ec2-user is ok for Amazon Linux AMIs).
Our SSH client will notice an opened socket for communicating with the remote server and will start exchanging securely.
Don’t forget the last line of the script to remove the temporary generated key:
This script is meant to be used with the SSH client, so we need to add the following lines on the ~/.config/ssh:
We voluntarily ignore checking the remote public key because another one is issued for each connection — that means you’d have a security warning each time you attempt to reconnect.
Ta-da ! You can now connect to your Bastion host without any IP, nor SSH key!
To keep in mind:
- The IdentityFile is read by the SSH client before invoking the ProxyCommand script. In this case, the identity file changes during the ProxyCommand invocation (which is our case here) and the SSH client will result in an error. This is why it’s critical to ensure that the key pair is removed at the end of the connection.
- The ProxyCommand contains “|| true” at the end because this is the only way we thought abut to ask the MacOS SSH client to execute the last line of the script when it’s completed.
To go further with SSH and TCP Tunnelling
SSH Tunnelling
It’s also possible to do Bastion jump via SSM and instant-connect to other internal machines, which wouldn’t be equipped with SSM or Instant-Connect yet.
In the following example, we attempt a connection to a Privately addressed IP server, non reachable from outside. The connection is first established with the Bastion server via SSM and Instant-Connect, then the remote servers’ SSH key is presented to connect to internal-production-blue-server.
The SSH client is smart enough to link the ProxyCommand via the -q option switch.
TCP (non-secure) tunnelling
We use TCP (non-secure) tunnelling for connecting to specific internal services like RDS or Redis clusters — for troubleshooting purposes only.
This can be achieved on the ~/.ssh/config file slightly differently.
In this first part, we’ve seen:
- What a Bastion host is and what it’s used for;
- How the EC2-Instance-Connect and SSM Tunnelling work;
- How to configure a client to connect to an existing Bastion without IP nor SSH key permanently deployed on the server;
- How to set up a client for secure and non-secure TCP Tunnelling.
If you want to set up the related infrastructure, you can access the second part : How to set up the infrastructure. In this second part, we detail each piece of infrastructure required to achieve the full picture and how to deploy them via Terraform.
