Setting up a VPN on AWS VPC
We recently integrated a third party service which required us to setup an IPSec VPN. It required us to have a public IP VPN endpoint, via which all requests should be routed, and another public IP address from which all requests should originate. This becomes challenging because typical AWS setups use the VPC service, where each instance has only its private IP addresses attached to them. The post explains how to do this setup.
The EC2 instances, in an AWS VPC, are running behind a gateway with only private IPs attached to them. The gateway serves two purposes: it is the target for all internet bound traffic in the route tables of the instances and it provides a NAT service to convert from the private address of instances to their public IP address. To understand how the setup works, it is useful to understand how this NAT translation works.

Third Party VPN Endpoint(Public IP): T1.T2.T3.T4
Third Party App Server(Public IP): A1.A2.A3.A4
Our VPN Endpoint(Private IP/Public IP): 192.168.X1.Y1/E1.E2.E3.E4
Source Instance IP(Private IP/Public IP): 192.168.X2.Y2/S1.S2.S3.S4
Understanding NAT
All IP Packets have a source and destination IP address. When a packet originates from an instance(in our example, 192.168.X2.Y2) intended towards a public destination IP address(say, A1.A2.A3.A4), it will be routed towards the gateway. The source IP address of the packet would be the private IP of the instance(192.168.X2.Y2). The gateway, serving as a NAT, maintains a mapping from the private IP(192.168.X2.Y2) of your instance to its public IP(S1.S2.S3.S4). The gateway uses this mapping to rewrite the source IP address of the packet. For the app server, the packet seems to originate from the public IP. The gateway finishes the other leg of translation when it receives a packet with destination as the public IP(say S1.S2.S3.S4) of one of the instances behind it. It changes the destination address of the packet to its private IP address(192.168.X2.Y2) and routes the packet in the private subnet accordingly.
Configuring Routes
In our case, the packets bound to the third party app server must originate with the public IP S1.S2.S3.S4 and should also be routed via the VPN. To achieve this, we must configure the packets going for the app server, to go to the VPN endpoint. This can be done via adding a route on the instance:
route add -host A1.A2.A3.A4 gw 192.168.X1.Y1On receiving these packets, the VPN endpoint must act like a NAT and change the source IP address to S1.S2.S3.S4 before sending it through the VPN tunnel. This can be achieved by manipulating the IP tables on the tunnel endpoint.
iptables -t nat -D POSTROUTING -s 192.168.0.0/16 -d A1.A2.A3.A4/32 -j SNAT — to-source S1.S2.S3.S4With IPSec VPN, the packet itself would be encapsulated inside another packet which is destined towards the other VPN endpoint. As the app server would respond, it would send packets with destination S1.S2.S3.S4. These packets’ destination address also need to be rewritten so that they are routed appropriately. Note that this needs to be done pre routing on the tunnel endpoint, otherwise the packet would be routed towards the gateway.
iptables -t nat -A PREROUTING -s A1.A2.A3.A4/32 -d S1.S2.S3.S4/32 -j DNAT — to-destination 192.168.X2.Y2An interesting thing to note in the setup is that the IP S1.S2.S3.S4 may not be attached to any of your instances at all. It just needs to be owned by you to avoid any conflict over the internet.
An issue with this approach is that the route command needs to be executed on each instance which will send requests to the app server. To avoid this we can choose a machine with a web server running on it, say Nginx, and use it as a proxy server to the app server. Then only this machine requires the route command to run and makes the setup more modular. While all other machines would send requests to the proxy server. We will write more about setting this up in the next post.
We got a lot of help from the following blogs while doing our setup, do check them out too -
About Credy
We are an online lending startup backed by Y-Combinator and some of the biggest investors. We give personal loan to salaried people. We are looking to expand our engineering team to change how lending works in India. If you want to learn more about the kind of problems we are working on, drop us a line at engineering@credy.in.

