Secure Port Forwarding in AWS using AWS SSM
This blog post is going to get technical and go through how to setup secure port forwarding with SSM at the end.
If you want to dive right into the technical parts, you can skip the problem overview, and if you don’t want to dive into the nitty-gritty, you can ignore the last sections.
Problem Overview
If you’ve ever worked on a service that resides in a locked-down network, you know how hard it might be to test, debug or diagnose. This could be behind a load balancer, a Content Delivery Network (CDN), or even just a secure API that only serves other applications or users in a locked down network. Locking down your applications so that only the expected traffic can flow through to it is a best practise to help secure environments. However this can hinder your ability to access the app outside the usual traffic flow.
Looking at the diagram above, you can see that the only valid traffic to your application server is via the load balancer, and it’s good practise to lock things like Security Groups and firewalls down this way, so that no other traffic can get in. Once locked down properly, you cannot connect to that instance from the internet, the AWS Account, the VPC, or even from the private subnet itself, only the load balancer can connect to your application servers. This means you have to go through the internet gateway to the load balancer, which can forward on your request to the server. But sometimes you can’t even do that:
In this diagram, there is another application server that is completely inaccessible to users as it powers the main application server. Here even if you were to go via the internet gateway, you cannot hit the private application server endpoints directly as they only serve your other application.
In the past, you may have had ssh access to instances running your applications, in which case you could have ssh’d onto your instance and hit it’s endpoints with a request like curl http://localhost:8080, or even hitting SSL endpoints with curl --insecure https://localhost:8080. But with modern, stateless, replaceable services, ssh access is usually disabled, or at least discouraged, especially in production environments.
Without access to your service endpoint or ssh; troubleshooting, validation, and even rolling out updates can become a lot harder. Normally in modern systems you have access to logs and metrics that have been scrapped off the instance to help with troubleshooting and validation, so I’m not mandating that you always need access, but some valid use-cases for this could be:
- You need to manually troubleshoot/replicate an issue with a request that is failing
- You want to validate instance endpoints are healthy before attaching them to load balancers and you can’t use cloud-native health-checks
- You need access to a private API, for example during deployments for validation or configuration purposes
Fortunately there are ways around these locked-down environments, including:
- Setup a bastion server and allow it to hit your services
This is quite a standard approach and great for when teams need to get into a locked down network to debug issues - Setup a secure reverse proxy to your services
This is probably the option you would want to go for if you are not going to open up your network and you need a reliable connection for substantial traffic - Temporarily open up access to the service port or ssh when needed
This is great for break-glass procedures where you are not expecting to need access, but it might suddenly become crucial if you are debugging a live incident - Use AWS SSM to create a secure tunnel
This is what I’ll be going over in the rest of this post
What is AWS SSM
SSM is AWS’s “Simple Systems Manager”. It’s a toolbox of services and functionality that allow you to manage your environments, servers and “systems”, securely and at scale.
They dropped the “Simple” after a while but kept the acronym, and you can see why: this toolkit has kept growing and is jam packed full of features you were probably not aware of. These include:
- Storing key-value parameters for storing config for use with AWS services
- Managing incidents
- Automating and applying runbooks at scale (including activities such as patching)
- Controlling Changes with “Change Manager”
- Viewing inventory and compliance across servers
- Running ad-hoc commands on instances
(For a full list of current features, see: https://aws.amazon.com/systems-manager/features/).
A lot of these features rely on having the SSM Agent running on the instances that you want to manage. This allows the SSM Service access to those instances and their internals that AWS wouldn’t have access to out of the box, such as running commands remotely on those instances.
What is AWS SSM Port Forwarding
Lots of the functionality that SSM uses is exposed through “Documents”. These are like runbooks that allow you to execute some piece of functionality. A subset of these documents are session documents that open or create some sort of session, and are useful for tasks like running commands on servers. One of the documents that Amazon exposes through the use of the SSM agent is SSM port forwarding. If you’ve used ssh port forwarding before, the premise is the same, except instead of going over ssh, you are using the connection that AWS SSM has with the SSM agent on the server.
If you have the SSM agent configured and running on the instance, and have enough permissions to the AWS account, a simple aws cli command can get this setup for you:
Once you have this tunnel setup, you can hit any APIs or websites by directing traffic to the exposed port on your own server. Targeting localhost from your own computer will tunnel the traffic through this tunnel to the port on the remote server.
This requires no special networking setup, as long as you can run AWS CLI commands locally and the instance has access to AWS SSM, as all traffic, routing and translations are handled by AWS internally.
Security Considerations when using SSM Port Forwarding
AWS make it very easy to manage and use the SSM port forwarding feature, however they don’t make it easy to lock down out of the box.
The SSM agent, once installed and setup, has no authorization dials you can tune. An agent is either connected to the AWS SSM service, allowing commands, tunnels and runbooks to be executed, or it’s not. All of the control happens in IAM. This can make it tricky to lock down. If you give a user or role generic access to SSM so they can open up an Port Forwarding tunnel to an instance, it’s easy to accidentally give them access to run whatever commands they want on that instance, or even worse, on any SSM-connected instance in your account.
Even if you only give them access to Port Forward, those credentials could be used to expose port 22, potentially then gaining command-line access through ssh, or another port running vulnerable software that could be exploited.
Creating a secure, scalable approach to ensure you can make use of port forwarding without exposing too much of an attack surface can be hard.
Solution
Setting up AWS SSM Port forwarding is relatively simple. You can follow this guide from AWS to get setup: https://aws.amazon.com/blogs/mt/use-port-forwarding-in-aws-systems-manager-session-manager-to-connect-to-remote-hosts
However I’m going to go through a solution I’ve used to lock down SSM so that our CI/CD pipelines could access private service endpoints to finalise API-based deployments.
This solution allows you to easily lock down roles to opening only certain ports on certain instances, massively limiting any attack surface. The solution also makes it easy to configure and update, working mainly off tags to control access.
Solution Overview
This solution involves:
- Creating a custom SSM document, restricting ports exposed from the target instance
This ensures that you can grant access only to a certain port, which would not be possible without a custom document - Creating an IAM Policy to give SSM access to that one document to your target role
This enables your IAM role to use the new document, while not opening up access to the full SSM functionality - Locking down the relationship between documents allowed to run and instances allowed to run against using tags
Using tags in the IAM policy provides an easy, scalable way to manage and control this access without updating the policy each time a new instance is created
Architectural overview of solution
The following diagram shows how access is controlled in this solution.
In the rest of this blog post I’ll go through how you can setup this solution, using IAM and tags to lock down access to instances via AWS SSM in a secure, scalable manner. Giving you an easy way to lock down and restrict target ports and instances.
Technical Section
AWS SSM Port Forwarding Pre-requisites
A couple of things to note beforehand:
- This only works with AWS SSM managed nodes (including EC2 servers and ECS tasks, although you could install the agent on on-prem servers too)
- This is not a solution you want to use for a permanent, vital connection
AWS has made this as seamless as possible, however there are a couple of steps needed before you can setup Port Forwarding.
- Ensure that the EC2 or ECS instance is connected to SSM (You can see all managed nodes in the SSM section of the AWS console)
- Ensure you have permissions to create and manage IAM roles and SSM
Setting Up AWS SSM Port Forwarding
There are multiple parts to setting up this solution. I will explain each part of the solution and provide example terraform code you can use as a starting point.
The first step is to have an IAM policy that will allows roles it is attached to to start SSM sessions against instances that have the SSMPort443 tag set to Enabled. This limits them to running only the SSM document called ForwardPort443 that we will define later.
# Get your account ID to use in the policy and role definitions
data "aws_caller_identity" "current" {}
# This policy allows the role to start SSM port forwarding sessions against port 443 on ECS tasks or EC2 instances that have the SSMPort443 tag set to Enabled
resource "aws_iam_policy" "ssm_443_port" {
name = "SSMPortForward443Policy"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
# This describe/list block is just here so that we can find the instance or task we want to use
{
Action = [
"ecs:List*",
"ecs:Describe*",
"ec2:List*",
"ec2:Describe*",
]
Effect = "Allow"
Resource = "*"
},
# This allows us to start the SSM sessions against both tasks and instances
# It limits to only allowing access to instances with the specified tag
# You can remove one of these targets if not needed
{
Action = [
"ssm:StartSession",
]
Effect = "Allow"
Resource = [
"arn:aws:ecs:*:${data.aws_caller_identity.current.account_id}:task/*",
"arn:aws:ecs:*:${data.aws_caller_identity.current.account_id}:instance/*",
]
Condition = {
StringLike = {
"aws:resourceTag/SSMPort443" = [
"Enabled"
]
}
}
},
# This allows us to start sessions against the specific SSM Document
{
Action = [
"ssm:StartSession",
]
Effect = "Allow"
Resource = [
"arn:aws:ssm:*:*:document/ForwardPort443",
]
},
# This is optional, and ensures the role cannot use the default forwarding document
# You can optionally remove all AWS documents (as seen in the second line)
# However a better solution may be to ensure you have not allowed SSM anywhere else, as this allows additive policies
{
Action = [
"ssm:StartSession",
]
Effect = "Deny"
Resource = [
"arn:aws:ssm:*:*:document/AWS-StartPortForwardingSession*",
"arn:aws:ssm:*:*:document/AWS-*",
]
},
# This ensures the role can terminate and resume it's own SSM sessions
{
Action = [
"ssm:TerminateSession",
"ssm:ResumeSession"
]
Effect = "Allow"
Resource = "arn:aws:ssm:*:*:session/$${aws:userid}-*"
},
]
})
}Next you can create the role that will be assumed to run the AWS CLI SSM commands. If you are expecting users to do this directly and not assume roles, you can replace the role with a user.
# Create an IAM role that will be used
resource "aws_iam_role" "ssm_role" {
name = "SSMPipelineRole"
managed_policy_arns = [
aws_iam_policy.ssm_443_port.arn
]
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Sid = ""
Principal = {
AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"
}
},
]
})
}You now need to define the SSM Document that will be used for this connection. We cannot use the default AWS SSM Documents as they do not allow you to restrict ports. In the example below we are limiting the document to only expose 443 on the target instance. You can create multiple documents for each port needed, or can consolidate ports into one document (see the GitHub repo for more details on this).
# Create the new whitelisted forward SSM Document, that is a copy of AWS-StartPortForwardingSession but with restricted ports
resource "aws_ssm_document" "forward_port_443" {
name = "ForwardPort443"
document_type = "Session"
content = jsonencode({
schemaVersion = "1.0",
description = "Document to start port forwarding session over Session Manager to port ${each.key}",
sessionType = "Port",
parameters = {
localPortNumber = {
type = "String",
description = "(Optional) Port number on local machine to forward traffic to. An open port is chosen at run-time if not provided",
allowedPattern = "^([0-9]|[1-9][0-9]{1,3}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])$",
default = "0"
}
},
properties = {
portNumber = "443",
type = "LocalPortForwarding",
localPortNumber = "{{ localPortNumber }}"
}
})
}The only thing left to do would be to ensure your target instances have the SSMPort443 tag set to Enabled. Once they have this and are setup as managed nodes in AWS SSM, your new role should be able to connect to port 443 on the instances.
Here is an example command showing how you can connect using the AWS CLI:
# Forward port 3000 locally to 443 on the target using the specific port document
aws ssm start-session --target "i-1234567" --document-name ForwardPort443 --parameters '{"localPortNumber":["3000"]}'
# Forward port 3000 locally to 443 on the target using the whitelisted ports document
aws ssm start-session --target "i-1234567" --document-name ForwardWhitelistedPorts --parameters '{"portNumber":["443"],"localPortNumber":["3000"]}'All of the above code and more is available here: https://github.com/bauk-blogs/aws-ssm-port-forwarding
Further Improvements
Depending on your environment and your use-case you may want to customise this further. Some ideas or suggestions if you plan on incorporating something similar into your infrastructure are:
- You should still consider locking down IAM in other ways as per your business requirements (e.g. limiting all access to only certain regions)
- If you have a specific set of ports you want to open on the same instances, you can create an SSM document that allows more than one port to prevent having multiple SSM documents
See the example below for creating an SSM Document allowing a set list of whitelisted ports (80, 443, 8080 and 8443)
# Create the new whitelisted forward SSM Document, that is a copy of AWS-StartPortForwardingSession but with restricted ports
resource "aws_ssm_document" "forward_whitelisted_ports" {
name = "ForwardWhitelistedPorts"
document_type = "Session"
content = jsonencode({
schemaVersion = "1.0",
description = "Document to start port forwarding session over Session Manager",
sessionType = "Port",
parameters = {
portNumber = {
type = "String",
description = "(Optional) Port number to expose from the instance",
allowedPattern = "^(80|443|8080|8443)$",
default = "80"
},
localPortNumber = {
type = "String",
description = "(Optional) Port number on local machine to forward traffic to. An open port is chosen at run-time if not provided",
allowedPattern = "^([0-9]|[1-9][0-9]{1,3}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])$",
default = "0"
}
},
properties = {
portNumber = "{{ portNumber }}",
type = "LocalPortForwarding",
localPortNumber = "{{ localPortNumber }}"
}
})
}For more ideas and full code examples, see this GitHub repo.
About the Author:
Kuba Jasko is an AWS Senior DevOps Engineer at Version 1.
