How to Use Postgres with AWS Lambda and Python

Introduction

AWS Lambda is a powerful serverless computing service that allows you to run code without the need to manage servers. It’s an ideal choice for building scalable and cost-effective applications. If your application needs to interact with a PostgreSQL database, you can seamlessly integrate AWS Lambda with PostgreSQL to create serverless, data-driven solutions. In this guide, we’ll walk you through the steps to use Postgres with AWS Lambda.

Before we get started, verify that your database is able to accept and send data via TCP on the database’s port (the default for Postgres is 5432).

Step 1. Create Lambda Function

Our first step after navigating to Lambda in the AWS Management Console is to create a new Lambda function.

For this tutorial, we’ll create a function called myFunction. For the runtime, select Python (I used 3.11 — to see older versions of python, scroll down in the dropdown menu). The x86_64 architecture should be selected by default.

To create the Lambda function, click the “Create function” button.

Step 2. Write your Lambda code locally

Now that you have created your Lambda function, it’s time to write and test your code locally before deploying it to AWS Lambda. We’ll start with the following template:

Edit this file as needed to fit your query and return requirements. For now, save it as lambda_function.py. We’ll add the code to finish connecting to the database in step 4.

Step 3. Install aws-psycopg2 binaries

AWS’s Lambda environment does not include psycopg2 by default, and it requires a slightly different distribution than some local OS. Store the binaries for aws-psycopg2 in the same directory where you created the lambda function. The easiest way to do this is by using the following command in the terminal:

pip install aws-psycopg2 -t <directory-where-lambda_function.py-is-stored>

Your directory should look like this afterwards:

Step 4. Establish DB Connection

Next, we need to get information about your Postgres instance to set up the connection.

Option A: Create Environment Variables for Postgres Connection

To connect to your PostgreSQL database from AWS Lambda, you can use environment variables to store sensitive information like database credentials. You’ll need the following pieces of information from your Postgres instance: username, password, host, port, and db name. If your Postgres instance is running on AWS RDS, you can find that information (except for password) on the Connectivity & security and Configuration tabs of your database:

Next, we need to configure your Lambda function. Navigate to the Environment Variables section under the Configuration tab.

Next, select “Edit” and add the following five environment variables:

Finally, we need to add some code in our Lambda function to be able to access these environment variables:

For local testing, set up environment variables with the same names.

Option B: Store Credentials using SecretsManager

To avoid storing credentials in plain text in your Lambda function, we can create a new secret in AWS Secrets Manager. In a new tab, navigate to AWS’s Secrets Manager and select “Store a new secret.”

Next, select “Credentials for Amazon RDS database” and enter your db credentials. (Note that this step assumes that your Postgres instance is configured with Amazon RDS. If not, use the Credentials for other database or Other type of secret options to store your credentials and database information).

Next, give your secret a name. We named this secret “mysecretname”.

Follow the rest of the default settings in the wizard and store the secret. Finally, add the following code to your Lambda function to access the credentials:

You should now be able to test your code locally before uploading to AWS.

Step 5. Upload Code

Once you have tested your Lambda function locally and it works as expected, you can proceed to upload the code to AWS Lambda.

First, compress your lambda function file with the aws-psycopg2 files you installed in the previous step. Your directory before compression should look like this:

Next, we need to upload this .zip file to the Lambda function we created in step 1. From your function’s code tab, select the “Upload from” dropdown menu and choose .zip.

Follow the wizard to upload the compressed folder. Note that with psycopg2, the uploaded file size will likely be too large to be able to see the code source in the web editor.

Step 6. Test Function

Before putting your Lambda function into production, it’s essential to thoroughly test it to ensure it works correctly with your PostgreSQL database. You can do this within the Lambda Console:

1. Create a Test Function: In the Lambda Console, click the “Test” button to create a test event for your Lambda function.
2. Configure Test Event: Provide the necessary input data or event for your function and give the test event a name.
3. Run Test: Click “Create” to save the test event, and then click “Test” to execute your Lambda function with the provided test event.
4. Check Logs: Review the execution results and logs to verify that your Lambda function successfully interacts with your PostgreSQL database.

With these steps completed, you should now have a functioning AWS Lambda function that can connect to a PostgreSQL database, making it a valuable component of your serverless application.

Conclusion

In conclusion, integrating Postgres with AWS Lambda opens up numerous possibilities for building serverless, data-driven applications. By following the steps outlined in this guide, you can create, test, and deploy Lambda functions that interact seamlessly with your PostgreSQL database, enabling you to build scalable and efficient solutions on the AWS platform.