Best practice of network architecture in Cloud (or even in Data Center) is to segregate your infrastructure resources in appropriate subnets. Resources requiring exposure to public in public subnet, and rest of the resources in private subnet, further divide private subnet into more than one private subnets to keep application, database and administrative infrastructure resources segregated into their own private subnets.
For a typical web application or a microservice with three tier architecture, you would expose some part of your application or API to public, using a Load Balancer in public subnet (resolved by a friendly domain name, e.g. {api|{app-name}}.yourcompany.com). Load balancer then will route traffic to your application servers (micro-service instance/container), and application servers will read/write data from a database.
I won’t go into details of the right subnet and IP allocation strategy in this post (may address that in a future post though), the premise of this post is to agree that infrastructure resources will be placed in appropriate subnets, mainly either publicly exposed or kept private (for internal only usage). As infrastructure gets placed into private subnet, following question arises
How can we access resources (like Relational Database, Open Search Server, Caching Server etc) created in a private subnet for development, troubleshooting, maintenance and other administrative activities?
In absence of VPN setup/connectivity, (which would be the case if you are a Startup or a small company leveraging Cloud-native and SaaS solutions for your enterprise operations and development), you would need a separate server hosted in your public subnet, exposed and accessible over internet, with SSH (port 22 open) allowed from list of known IP addresses (of DevOps team). This server is your Bastion Host. In this setup, fact that your bastion host is publicly accessible, with SSH (port 22) open is a security risk.
Stop!! You don’t need to expose your Bastion Host over the Internet.
AWS System Manager — Session Manager
AWS Session Manager, which helps you address above mentioned risk and improve your security posture. Benefits of Session Manager are:
- Centralized Access to Bastion Host (or other EC2 Instances) using IAM Access Policies
- No open ports for Inbound access, no need to maintain SSH keys either
- Easy Access to managed nodes from AWS Console/UI
- Supports Port Forwarding
- Supports Linux, Windows and macOS
- Auditing (AWS CloudTrail) and Logging (CloudWatch and S3, with KMS Encryption of Logs)
- Encryption using AWS KMS
We leverage Serverless as much as possible, except Database, Search and Caching. Only server we have to manage is a Bastion host, which has logic (userdata + cron) in place to sync developer ssh public keys to the host, granting them access as long as host is reachable (public) and port access is allowed (using Security Group Ingress rules). Once developers have access to the bastion host, they use SSH port forwarding/tunneling to communicate with other servers inside the private subnet.
Use of AWS Session Manager helps reduce attack surface by removing Public exposure of the Bastion Host, it no longer requires maintaining ingress rules with allowed IP address to SSH, also with SSMSessionRunAs support enabled you don’t need to maintain SSH Keys either, hence reducing maintenance overhead. Users with access to your account (preferably, SSO enabled and using IAM Role with STS), and withssm:StartSession permission (and appropriate IAM Policy Conditions), this setup adds a layer of security and removes need to customize developer off-boarding process to remove user access when they leave company, as they will no longer be able to obtain STS tokens to interact with AWS Session Manager.
Session Manager Plugin allows you to use SSH into bastion host, even without having any ingress rules on the bastion host running in private subnet, by using ProxyCommand config in your ~/.ssh/config which leverages aws ssm start-session command to initiate SSH connection.
Auditing & Logging
SSM Start/Terminate Session API Calls are tracked as part of standard Management API audit trail, this will be good for record keeping to see which user interacted with an instance using Session Manager, and what SSM Document was used to interact with.
On top of the audit trail of management events of Session Manager, you can configure Session Manager to log Session activity to Cloudwatch Logs and S3, both supporting KMS key based encryption to keep the logs encrypted at rest. Please note that session activity is not recorded for sessions established using SSH ProxyCommand.
Overall, I am happy with Session Manager features and having integrated it with our environment, it is a needed step in the direction to improve our security posture by making bastion hosts private, and having a centralized mechanism to grant users access to establish SSH connections, tunneling to other servers in private subnet. I highly recommend the use of AWS Session Manager to stop exposing bastion host publicly and improve security.
If you liked this story, feel free to follow so that you get notified of my future posts. Below are some of my other stories you may like:
