Recipe: Connect to an RDS Database in a Private Subnet from your Workstation — Over HTTPs
Cooking time: 35 mins
You and your development team are coding away on their respective local machines. Getting the code compiled and running is easy enough, but what about resources such as databases?
Many AWS services can be interacted with over HTTPs (Secrets Manager, Parameter Store, DynamoDB, S3, Lambda Invoke — to name a few) and it’s easy enough to invoke them provided you assume an appropriate IAM role.
HTTP is a great protocol from a point of view firewalls and corporate transparent proxies. Trying to make an outbound TCP connection from your office over port 3306? Don’t be surprised if that outbound port is blocked. Connecting to a DynamoDB on the other hand? No problem.
One way to address the DB connectivity issue is for each developer to have their own instances of the DBs running on their local machines. If you have many microservices, however, the task for each developer to maintain a fleet of DBs on their local machines may prove arduous (and costly as far as people-time goes). Also, if using Aurora you need to ensure that your local MySQL instance approximates the config of your Aurora instances (collation, time zone, etc) — this too takes time.
One time tested solution is to use a bastion host and do an SSH port forwarding session. Doable. You can put the bastion host in a private subnet, and then front it with an NLB. Problem here is that your corporate firewall may detect and reject the SSH traffic originating from your workstation.
Another time tested solution is to setup AWS Client VPN. This is a bit more laborious and once again — your corporate firewall may detect and reject the VPN traffic.
There is Another Way!
You can map a local port (say 3306) on your workstation and proxy the traffic to the desired RDS instance in AWS, over HTTPs!
Ingredient List:
- 1 EC2 bastion host in a private subnet that can access the desired RDS DB. AWS Linux 2 is recommended.
- 1 SSM Agent installed on the EC2 bastion host (should be pre-installed if using AWS Linux 2)
- 3 VPC Endpoints allowing your EC2 bastion host to talk to SSM service
- 1 NAT gateway to allow EC2 to download packages from the internet
- 1 Session Manager Plugin for the AWS CLI installed on your workstation
- Eye of newt, toe of frog, pinch of salt— to taste
Cooking Instructions:
Detailed instructions would result in a Tolstoy of an article, so I’ll outline the general steps and the links to AWS documentation where you can get the nitty-gritty.
EC2 in a private subnet with a NAT gateway is a standard snack. Plenty of tutorials exist on how to do this. SSM Agent should be pre-installed on your EC2 instance. If not, follow Working With SSM Agent Guide.
Next, you need to Create VPC Endpoints for your EC2 to be able to talk to the SSM service.
Once completed, you should be able to Start session from AWS Console and connect to your bastion host. Do not proceed until you’ve tested this. Once the session is established, ensure you can connect to the RDS instance from the bastion host.
Next, install the Session Manager Plugin for AWS CLI. Test the setup by opening a session (aws ssm start-session) from your workstation to the bastion host via AWS CLI. If all good — proceed to next step.
Next, you need to configure your SSH client to proxy any commands to SSM. Read the Enable SSH Connections Through Session Manager guide. One thing to note is that you need to specify the AWS profile when proxying the SSH command. I’ve also appended the --profile argument in ~/.ssh/config
You can enable OpenSSH client on Windows 10 (if not already enabled) by reading this guide.
Cross your fingers, and run ssh -N -L 3306:mydb.rds.amazonaws.com:3306 ec2-user@bastion_instance_id -i bastion_instance.pem
The console window will “hang” — that’s a good sign. That’s means the magic is working. If something is wrong you will be presented with an error.
Try connecting to localhost:3306 and if connected, drop a couple of tables (I’m afraid that’s the only way). Then listen intently for screams from your fellow team members.
