Adevinta Tech Blog
Published in

Adevinta Tech Blog

An AWS Bastion Host unreachable from the internet — Part 2

Part 2 : How to setup the infrastructure

This article is the second part of “An AWS SSH Bastion Server reachable without IP connectivity nor SSH key”. It’ll focus on how to set up the server on the AWS account via Terraform.

To find out more about the SSM and EC2-Instance-Connect features, you can read the first part of the article here.

Image source

In this post, we’ll build the following infrastructure, piece by piece, using a Terraform module.

How much does this setup cost?

The only cost of our implementation is the EC2 machine used by the Bastion Host, as well as the unavoidable bandwidth output that is common to every AWS service used. Other services are free.

We’re not subject to the time-out while working with SSH-OVER-SSM. For SSM sessions, the timeout can be extended to 60 minutes.

The price of the Bastion ARM instance can be even lowered if a node reservation is made, paid up-front.

Why use a Bastion instead of a VPN endpoint ?

The finality of Bastion versus VPN Endpoint is basically the same : access securely internal resources from your AWS account.

We were tempted to choose the VPN endpoint, because it reduces the burden on the local tunnelling establishment via Bastion : you connect via the VPN Client, that’s it. However, the price for VPN Client is higher.

A VPN Client EndPoint can be associated with multiple subnets as long as they belong to different AZs. In other words, you need a VPN EndPoint association with each kind of network you have, example :

  • Private Subnets, spread across 3 AZ = 3 VPN Endpoint association.
  • Database Subnets, spread across 3 AZ = +3 VPN Endpoint association.
  • Public Subnets, spread across 3 AZ = +3 VPN Endpoint association.

The Bastion is cheaper (in our case, about 100 times cheaper) and there’s no charge per client connected.

If your needs are for short-lived connections, you don’t require connecting to multiple remote hosts at the same time or strong network performance, so I’d recommend to go with the Bastion tunnelling system. Else, you may want to go for the AWS VPN Endpoint.

Setup the EC2 role

Let’s begin — prepare our Bastion Host!

We’ll review the separated blocks used to achieve our goal of deploying a secure EC2 Bastion Host.

The prerequisite to build an EC2 virtual machine is to decide what this machine will be allowed to do. This list of choices is always tied to an IAM Instance Profile. Only one instance profile can be attached to an EC2, so it’s better to be exhaustive while editing it.

A role is a list of authorisations that an entity is allowed to do. Several resources in AWS can take on a role, if they’re allowed to do so. We control who can take on a role, using an Assume Role Policy declared on the same level as the IAM Role itself.

In our case, any object from the EC2 service (so: EC2 machines) are allowed to take on the role. We assign two IAM Role policies (=two lists of operations allowed or denied) of operations. They’re named AmazonAWSManagedInstance

file terraform/resources.tf (exerpt)

resource "aws_iam_instance_profile" "bastion" {
name = "bastion"
role = aws_iam_role.bastion.name
}
resource "aws_iam_role" "bastion" {
name = "bastion"
path = "/"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
#this allows to connect to the EC2 instance
resource "aws_iam_role_policy_attachment" "bastion_ssm" {
role = aws_iam_role.bastion.name
policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}
resource "aws_iam_role_policy_attachment" "ec2_instance_connect" {
role = aws_iam_role.bastion.name
policy_arn = "arn:aws:iam::aws:policy/EC2InstanceConnect"
}

You can retrieve the full example file on github

These Role Policies are defined by AWS and are adapted to our requirements for SessionsManager and Instance-Connect. It’s still possible to define finer-grained tuned policies, but it can be tedious.

We also want to allow our users to perform SSM operations. The SSM authorisations should be managed elsewhere on your role’s IAM profiles and you can refer to the quickstart to know what the required permissions are.

Set up the Security Groups

The concept of security group is ambivalent : it’s a list of rules assignable to a resource, and, at the same time, can be used as a source group to allow access.

For our deployment we have one Security Group SG_Bastion, assigned to the bastion host itself, allowing anyone having the same group assigned to communicate without restriction with any ressource with the same Security Group assigned.

Said differently, this allows potential multiple Bastion hosts to communicate with each other without restriction. This is an optional choice and we could also have decided to prevent the Bastion Hosts to communicate with each other.

Our new SG_Bastion Security Group has been added as an ingress authorization on multiple other existing Security Groups assigned to multiple resources. This can be achieved automatically via a Terraform output :

  • On a Terraform Output, you add the SG_Bastion Security Group ID
  • This data can be retrieved by any other module (as a variable) and is used as a source security group rule allowing the incoming secure bastion connections.

file terraform/resources.tf (exerpt)

##########################################
### Security Group Bastion
resource "aws_security_group" "bastion_in_out" {
name = "bastion"
description = "Bastion Security Group"
vpc_id = var.vpc_id
}
# Everyone from that security group can contact
resource "aws_security_group_rule" "bastion_in_self" {
type = "ingress"
from_port = 1
to_port = 65535
protocol = -1
self = true
security_group_id = aws_security_group.bastion_in_out.id
}
## OUTBOUND rules
# Everyone from that security group can contact
resource "aws_security_group_rule" "bastion_out_everywhere" {
type = "egress"
from_port = 0
to_port = 65365
protocol = -1
cidr_blocks = ["0.0.0.0/0"]
security_group_id = aws_security_group.bastion_in_out.id
}
#Outputting the Security Group ID
output “bastion_sg_id” {
value = aws_security_group.bastion_in_out.id
}

You can retrieve the full example file on github

Set up the EC2 provisioning

EC2 Launch Template

It’s rare to directly create an EC2 instance on one account. Rather, we create an AutoScaller that will ensure the Instance’s health and recreate new instances if required.

To tell the AutoScaler what to deploy, we rest on a Launch Profile or a Launch Configuration. The concept is the same : a list of settings that would define what EC2 to start with. It’s recommended to use Launch Profiles because AWS ceased to deploy new features on Launch Configurations since October 2021.

file terraform/resources.tf (exerpt)

##############################################################
# LAUNCH TEMPLATE
##############################################################
resource "aws_placement_group" "bastion" {
name = "bastion"
strategy = "spread"
}
resource "aws_launch_template" "bastion" {
name = "bastion"
description = "Bastion (Amazon Linux 2 AMI ARM)"
block_device_mappings {
device_name = "/dev/sda1"
ebs {
volume_size = 20
}
}
capacity_reservation_specification {
capacity_reservation_preference = "open"
}
credit_specification {
cpu_credits = "unlimited"
}
disable_api_termination = falseebs_optimized = trueiam_instance_profile {
name = aws_iam_instance_profile.bastion.name
}
image_id = "ami-0c669fe429b4cf93d"instance_initiated_shutdown_behavior = "terminate"instance_type = var.instance_typenetwork_interfaces {
associate_public_ip_address = false
security_groups = [aws_security_group.bastion_in_out.id]
}
monitoring {
enabled = true
}
placement {
group_name = aws_placement_group.bastion.name
}
# These tags are set to the created resource
tag_specifications {
resource_type = "instance"
tags = {
Name = "bastion"
}
}
}

You can retrieve the full example file on github

There are two critical settings here:

  • The image ID (the system disk we want to deploy to): here, an Amazon Linux 2 for ARM64 processors.
  • The network interface: no public IP address, no load balancer attached and one security group SG_Bastion.

Attention! A note about system provisioning
To enable secure connections via SSM and temporarily SSH keys, you need to have installed the AWS SSM plugin as well as AWS Cli V2.

Amazon Linux 2 default AMI already contains the AWS SSM manager as well as the EC2 Instance-Connect plugin. If you’re using another Machine Image (like a vanilla Debian), you need to install those two components.

This is made via the user_data variable on the launch template level

user_data = <<EOF
wget https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_arm64/amazon-ssm-agent.rpm
sudo rpm -i amazon-ssm-agent.rpm
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent
EOF

Alone, this launch template won’t do anything. We now need to deploy it.

Auto Scaling group

The auto scaling group will deploy the required amount of Bastion Hosts (here, one is enough: the instance is not production critical and we never have a lot of concurrent accesses). One of the noticeable differences between Launch Template and Launch Configuration is the Templates versioning. On one hand, you had to recreate a Launch Configuration for any change as they were immutable and on the other hand, with the Launch Template, you can update them by deploying new versions.

That means that your Auto Scaler needs to point to a version of your Launch Template. Here, we decided to stick to the latest version available.

Also, it is required to define an instance refresh to force deploying your changes when a Launch Template is updated.

##########################################
### Auto Scaling Group
##########################################
resource “aws_autoscaling_group” “bastion” {
name = “bastion”
desired_capacity = 1
max_size = 1
min_size = 1
vpc_zone_identifier = var.vpc_private_subnets_identifier
launch_template {
id = aws_launch_template.bastion.id
version = aws_launch_template.bastion.latest_version
}
instance_refresh {
strategy = “Rolling”
preferences {
min_healthy_percentage = 50
}
}
}

On the client side

You’ve completed the infrastructure configuration on the server side. The SSM Manager tunnelling or EC2-instance-connect doesn’t require any additional configuration.

On the client side, you need to have a valid STS Token attached to a user allowed to perform SSM Client Connect Operations and EC2-Instance-Connect.

For the client’s detailed configuration, you can refer to the first part of this post.

In this second part, we’ve seen :

  • How to implement a full working infrastructure for a secure Bastion EC2 Host
  • Detailed information about each infrastructure component

Conclusion

AWS features allow us to operate infrastructure differently. This can be used to secure external connectivity for a Bastion Host, for example.

Securitisation is an on-going work; in these two blog posts we went through how to secure the connectivity but not through the Bastion’s operating system hardening or the extensive logging features offered by SSM.

However, it’s fun to be able to connect to a Bastion Host without direct external connectivity.

Your EC2 Bastion Host is now within your private network and you’re using AWS services as virtual catapults to communicate with it. :)

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store