SSH on AWS? There is a service for that.

One (of many) lessons I learnt attending AWS Re:Invent this year, is that when you are trying to do something on AWS, more often than not “there is a service for that”, or there will be one very soon. A notable exception for me has been a managed service for securely accessing shell on EC2 instances. SSH is something (presumably) everyone needs from time to time, yet opening port 22 is like opening a can of worms in terms of security.

For this reason, we usually roll out Bastion hosts to serve as gatekeepers between the internet and our instances. This allows us to create a tunnel from our workstation, via the Bastion, and onto instances in a private subnet. With a gatekeeper in place, we can limit ingress to whitelisted IP’s, and control user access by adding/removing authorised keys. Easy.

Users can only SSH into instances by going through the Bastion host first.

Bastions are hard

While Bastion hosts are what you make of them, things tend to become a lot more difficult when you ask yourself the following questions:

• How do I automate user lifecycle management?
• What happens if someone has their private key compromised?
• How do I know if something is compromised? Where is the audit trail?
• Does my Bastion image have innate vulnerabilities?

And since we are dealing with security, it makes sense to look at existing solutions before rolling our own. In my case I ended up looking at Netflix/Bless (w/kmsauth), which solves most of these issues (it does not help you harden your Bastion image). While Bless is a great solution, it still requires a significant effort to set up:

• All instances need to trust the Certificate Authority (CA). Either it needs to go in machine images you provide, or you need to provide instructions on how to set it up.
• The bastion host needs to keep the local users synced with the IAM users for the account. Since we cross account roles (exclusively) at my work, we would need to parse user names from the trusted entities for all roles in a given account.
• If we also want to know who did what, we would actually need all our instances to keep local users synced with IAM.
• New tools require training. Users would have to be on-boarded to use the Bless client.

More importantly, the Certificate Authority becomes the key to our kingdom. For this reason, Netflix recommends rotating the certificate authority frequently, and limiting exposure by deploying Bless to a standalone account. This naturally ties up resources with the ‘operational’ responsibility for Bless, and is something that might be hard to justify on a small team. It’s tempting to go with a simpler solution, which unfortunately skimps on security.

“There is a service for that”

Recalling my lesson from Re:Invent, it seemed likely that this was something Amazon would be close to solving, if a solution did not already exist. So I looked around, and sure enough there is (and has been for a while) a service for that — SSM and SendCommand. SendCommand gives us the following benefits out of the box:

• Ability to run commands on instances without opening any ports.
• All command are logged to CloudTrail (auditing).
• Users and authorization is managed via IAM.
• Bonus: Commands can target multiple instances.

Since we don’t have to open any ports on our instances to use SendCommand, the ‘backdoor’ is as secure as the account itself. Nice, but all this security comes with some drawbacks:

• Instances need to run the SSM agent (and have privileges for SSM).
• No ability to port forward (e.g. to reach Kibana or a database).
• Cannot use text editors, or commands like tail -f.

I can live with these compromises, but…

Where is my shell?

Unfortunately, AWS does not provide a convenient way to invoke SendCommand on the command line. Here is an example from their CLI documentation:

aws ssm send-command 
    --instance-ids "i-0cb2b964d3e14fd9f" \
    --document-name "AWS-RunShellScript" \
    --comment "IP config" \
    --parameters "commands=ifconfig" \
    --output text

It works, but from a usability standpoint it was not what I would consider a drop-in replacement for SSH. It is something that would look good in a CLI, and since I couldn’t find any existing projects I set out to make one (and learn some Go in the progress).

SSM-SH

The CLI was dubbed SSM-SH since it’s trying to mimic SSH with SSM SendCommand. Here is a screenshot of what it looks like:

Screenshot of SSM-SH

And a code example:

# Filter instances managed by SSM (by name tag):
ssm-sh list --filter "Name=*kristian" --output targets.json
# Run a one-off command against the filtered list:
ssm-sh run --target-file targets.json -- "ps aux | grep agent"
# A shell could be started with:
ssm-sh shell --target-file targets.json

Wrapping SendCommand in a CLI made for a pleasant user experience, and overall it felt like a convenient replacement for SSH. Beside the advantages that come with SendCommand itself, I was happy to find that I:

• No longer needed to know the IP of my Bastion host(s).
• Didn’t have to look up the IP’s of the instances I wanted to connect to.
• Could target multiple instances at once.

If you are intrigued and would like to give it a go, you are welcome to check out the code and further instructions to get started here: https://github.com/itsdalmo/ssm-sh

-Kristian