How to turn an Amazon Linux 2023 EC2 into a NAT instance

Spin up a NAT instance for as low as $3.5/month

Why we need a NAT

Recently I’ve been working on setting up a new VPC on AWS. It contains lots of services both in public and private subnets. Some of these services are made in-house and as a deployment mechanism we chose to have Autoscaling Groups behind Elastic Load Balancers (of both the Network and Application type) that are able to monitor and redeploy EC2 Intsances at the drop of a hat. Relying on Launch Templates with startup scripts that download the latest versions of our applications from AWS ECR and spin up containers that perform their various duties.

The resources that reside in private subnets need a way to download all these fancy things, specifically a NAT, or network address translator.

How does a NAT work

Because of the global scarcity of IPv4 addresses, the accepted normal practice is to only give public addresses to the resources that absolutely need it (also because they’re expensive, and will keep getting more expensive).

This means that all the instances on the private subnets get private ips, that make them reachable from within the VPC (if the security groups and route tables allow it, of course). But having a private ip is insufficient to talk with the public internet, and so when private instances need to initiate a connection with the public internet it need to “borrow” a public ip for the ride. The device that allows for this temporary masking of the private ip with a public one is called a NAT.

Resources within private subnets that need to initiate connections towards the outside internet, need a public IP to mask their request. This is achived by NATting.

Possible solutions

There are three possible solutions:

AWS’s fully managed NAT Gateway

If you absolutely need IPv4 NATting and reliability is your #1 concern there is no better solution than this. The downside is that it costs $0.045/hour which totals at about $32/month, plus an additional $0.045 per Gb of traffic.

IPv6 routing

This is something that I tested and was really impressed by how easy and amazing it was! You basically have to enable the IPv6 feature on your VPC and subnets, create an Egress-Only-Internet-Gateway and then configure your route tables to let IPv6 connections go out unobstructed.

I ended up not adopting this approach because I needed to fetch some info from the AWS Parameter Store and download our private docker image from AWS ECR. Both these services at this time don’t support IPv6. It was a pity because I loved this solution as it’s free and doesn’t need any monitoring as it simply can’t fail.

You can see the current status of IPv6 adoption by various AWS services here.

Turning an EC2 into a NAT instance

In our case we were not particularly concerned with reliability, since the rest of the infrastructure that we built is super fault-tolerant. This NAT’s intended function would be only to provide a way to start up new instances in the unlikely case one should crash.

We have all sorts of HA (High-Availability) and monitoring systems in place, so if for some reason an EC2 crashed, it would just spin itself up and we’d be notified about it. If for some reason the EC2 was unable to download the containers and get back online we’d already be there to investigate the case. In the meanwhile all vital systems would keep humming along.

For a moment we considered following the approach detailed in this post, but ultimately we decided it was overkill for our use case.

Also we are using the same instance as a bastion host, powered by Tailscale (which I really can recommend), which is not really something you can automate, other than by making an AMI which is an approach I don’t like because it’s annoying to maintain.

Steps to making a NAT instance

• Deploy a VPC. In the VPC create a public subnet.
• Deploy an EC2 Instance with a public IP in the public subnet. In our case we went with a t4g.small , for low traffic situations you can easily go smaller. With a t4g.micro you will spend about $3.5/month!
• Disable source / destination check from the networking settings of the EC2 Instance.
• SSH into your instance and run the following commands:

# Turning on IP Forwarding
echo "net.ipv4.ip_forward = 1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Making a catchall rule for routing and masking the private IP
sudo iptables -t nat -A POSTROUTING -o ens5 -s 0.0.0.0/0 -j MASQUERADE

Important Note: Normally the standard network interface would be called eth0 but on Amazon Linux 2023 t4g instances it’s called ens5 . Figuring this out took me the best part of an afternoon so I hope this will be helpful to someone else.

• Configure the route tables of the private subnets in your VPC to point to this NAT instance for all destination IPs that don’t fall within the VPC’s private IP range.
• Configure the security group of the NAT instance to allow only traffic from the VPC’s private subnets

Testing

The most effective way is no doubt SSHing in one of the private instances (it helps if you have a bastion host to do that otherwise it can be quite a headache) and running the command:

tracepath 8.8.8.8

This prints out all the steps that your request makes. If it gets stuck somewhere you’ll know who’s the culprit and will have a good lead on how to advance the investigation.

Additional topics

As usual I’m constrained by the little time available, but i’d be happy to expand this post or to make spin-off posts in the future if special requests were to come in. Here are a few aspects on which I could have expanded more but cut short for brevity:

• Automating the NAT deployment with AWS CDK.
• Automating the NAT deployment with an Elastic IP (static public ip) and/or a static private IP.
• Adding support for rebooting the instance and have it resume NAT functionality on startup.
• Adding an Autoscaling Group to make sure that the NAT instance just spins itself up again without bothering anyone.
• Literally anything else.