How to connect your Lambda function securely to your private RDS instances in your VPC using security groups, and still have internet access!

The title says it all.

Daniel Schwartz
Oct 6, 2018 · 8 min read

I work a lot with AWS Lambda and all its intricacies on a daily basis and I have learnt a lot of things that I see a lot of people struggle with. So I decided to write down a specific guide on how to do something I often see giving people trouble. This is not specifically a tutorial, more of a guide, and assumes you have basic knowledge of AWS Lambda and other AWS services.

Let me give you the use case: I have an Amazon Relational Database Services (RDS) instance that I am using to store sensitive data. It is privately accessible (meaning the RDS endpoint resolves to a private IP address only). This database must be securely locked down on the network for obvious reasons. Therefore our RDS instance is sitting in a private subnet in my Virtual Private Cloud (VPC).

A private subnet is a subnet with no direct route to an Internet Gateway (IGW). In other words, it has no public IP addresses, no publicly accessible resources, no direct internet access to/or from that subnet.

Next, we have a suite of Lambda functions that are performing data processing tasks, database queries for API calls from API Gateway, storing metadata from SQS/SNS/S3 triggers or what ever else you use Lambda for. We want to give it access to our RDS instance without opening up our RDS instances security groups to anyone unnecessarily.

One last detail, our Lambda functions will still need internet access for accessing 3rd party APIs or reaching public resources or for what ever else you are using the internet with Lambda for. When we place our Lambda functions inside a private subnet in our VPC we lose internet access from our function by default. Timing out any outbound requests.

So how do we do it?


Step one: Preparations

We need to make sure we have a few things:

  1. An IAM user with required permissions to do our work.

Step two: Placement

We need to ensure we place everything in the correct place otherwise we will run into problems.

  1. Place your RDS instance in your private subnet(s) as per best practice.

Step three: Configuring our routes

We need two main Route tables for all of this to work. One for the public subnet(s) and one for the private subnet(s).

Public Route table need two rules:

Destination            Target
10.0.0.0/16 local
0.0.0.0/0 igw-<your_igw_id>

Private Route table needs two rules:

Destination            Target
10.0.0.0/16 local
0.0.0.0/0 nat-<your_nat_id>

The reason for these rules are that in the public subnet when a device attempts to connect to a public IP address (for example 8.8.8.8) it will fall under the 0.0.0.0/0 rule which has a target of the IGW. This means that it will use the IGW to break out to the internet. The private subnet does the same thing except instead of the IGW we use the NAT Gateway we setup in the public subnet. The other rules are the default local rules for routing within our VPC.

Step four: Configuring our security groups

We need 2 main security groups for this scenario. One for our RDS instance so we can lock it down, and one for our Lambda functions so we can provide access.

Lambda Security group rules:

sg-<lambda_sg>
Direction Protocol Port Source
Outbound ALL ALL ALL

Note: no inbound rules needed here as they have no affect on Lambda.

RDS Security group rules:

sg-<rds_sg>
Direction Protocol Port Source
Inbound TCP 3306 sg-<lambda_sg>
Outbound ALL ALL ALL

Note: we have outbound ALL incase our RDS needs to perform outbound operations like updates etc.

We are using the security group of the Lambda function as a source in the RDS security groups rule. This rule says that any source attempting to connect to RDS on port 3306 must be associated with that security group otherwise it will refuse the connection. This allows just the Lambda functions we associate with the Lambda security group access to the instance. This is the most secure way to allow your Lambda RDS access.

We do it this way for another reason as well. Lambda when it is placed in your VPC does not have a predetermined IP address that you can assign. It is given a random private IP address from the available pool of IPs in the subnets it is placed in. It can also change IP address without warning, or change subnets without warning.

Note: No Network Access Control Lists (NACLs) are required. In-fact NACLs would not work for this purpose. You cannot reference a security group in a NACL, meaning you need to specify IP addresses or CIDR blocks, which we do not have for Lambda. You do NOT need NACLs to make your setup more secure. In fact for Lambda it is best practice to use just security groups. Security groups are sufficient if you use them correctly. So, using NACLs would be very complicated to allow in only the Lambda IP addresses (even though they are changing often) without opening up the NACL to anything else. This is where I see a lot of people get confused. If you decide to use NACLs on-top of this for whatever reason be aware that Lambdas IP addresses can, and will, change.

Step five: Configuring our Lambda functions

Next, we move onto our Lambda functions. First thing we need to do is use the correct permissions in our execution role. If you haven’t made an execution role yet, then go to IAM and make a new Role. This service linked role will be used to allow Lambda to create the Elastic Network Interfaces (ENIs) it needs to connect to the VPC. Lambda also needs CloudWatch logging permissions which is standard.

Note: you do NOT need to give Lambda ANY RDS API access in order to connect to the database. Meaning you do not need to give Lambda access to turn on your RDS instances, or delete them, or make new ones — unless that is what you intend your Lambda function to do, in which case… go for it.

When creating the execution role I use 2 AWS Managed policies:

  1. AWSLambdaBasicExecutionRole — provides CloudWatch logging.

I assuming at this point you have created your function and uploaded your code, configured your triggers, and edited the timeout and memory settings.

Next, we need to define the Network settings for our function. Things we need to specify are: the VPC, the subnets and the security groups. When selecting the subnets be sure to choose the 2 (minimum recommended) private subnets. Then choose the Lambda security group we made earlier.

Note: If you place your Lambda function in a public subnet (i.e. one with a route to an IGW) your function will timeout connecting to the internet. This is because the ENIs Lambda creates in your VPC are not given a public IP address and will need a NAT Gateway to translate the private IP address to a public one. We specify the public IP address for the NAT Gateway as an Elastic IP Address (EIP). The NAT Gateway will then route traffic to the IGW and out to the internet. Therefore securely connecting out private resources to internet access.

Step six: testing our configurations

After all this, how can we tell we did this correctly?

  • Can my function connect to my RDS database?

Below is some sample Node.js 8.10 code I used to test my MySQL connection within my function. (Disclaimer: I am not a Node.js expert, I am still learning. Please do not use my code verbatim without sufficient testing and using best practices!)

const mysql = require('mysql')const connection = mysql.createConnection({
host: process.env.RDS_HOST,
user: process.env.RDS_USER,
password: process.env.RDS_PASSWORD,
database: process.env.RDS_NAME
})
connection.connect(err => {
if (err) {
console.error('error connecting: ' + err.stack)
callback(null)
}
console.log('connected as id: ' + connection.threadId)
})
connection.end(err => {
if (err) console.error('error closing connection: ' + err.stack)
console.log('connection closed')
callback('done')
})

Note: Use Lambda’s Environment Variables for storing sensitive data such as the RDS endpoint, username, password, database name, etc. They can be encrypted at rest, or transit.

  • Can my function connect to the internet?

I will combine this with:

  • Can I perform AWS SDK calls from my function?

Assuming I have given my execution role permissions to GetObject from S3, I will do this as a test. Below is some sample code (same disclaimer!).

try {    const params = {
Bucket: 'bucketname',
Key: 'test.json'
}
const response = await S3.getObject(params).promise()
console.log(response)

} catch (err) {
console.error(err)
}

Note: You don’t have to make SDK calls like this, just bear in mind that the Node.js AWS SDK calls are all Asynchronous.

If all of the above tests worked then we are done! If you are having things like timeout issues, I will go through some common issues below that may help.


Common issues, with common fixes

1. My function is timing-out, but only in the VPC!

This is by far the most common issue. Usually it is because the function is attempting to connect to the internet but the configuration is incorrect, resulting in a timeout. Be sure to check:

  • Lambda must be in private subnets.

2. My function can’t connect to my RDS instance!

Assuming your network configuration is fine and you have internet access but can’t connect to your RDS:

  • Check if NACLs are interfering.

3. My function sometimes times-out but not all the time!

Intermittent timeouts are more difficult to troubleshoot.

  • Check that one (or more) of the subnets Lambda is in are not public.

4. I can’t save my VPC settings in Lambda!

If you created your function and then want to add VPC settings afterwards, then you need to give your Lambda execution role the ENI describe, create and delete permissions we did earlier, before attempting to save the VPC settings.

5. My question is not here!

I will say this, most of the issues I see people have with Lambda are code related issues. This is specifically about the services and not the code. But if you have some questions I will do my best to answer them in the comments.


Thanks for reading! If you have feedback for me on the article, or comments, or questions about the article, or Lambda in general, do not hesitate to reply below! If you have more ideas for things I can write about then let me know. I may add to the article over time with more info.

All the best!

Daniel Schwartz

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade