Database Proxies and Why You Should Use Them

When building a service with microservices, each microservice performs a specific task and can be independently deployed and scaled. This means that a service can be composed of many microservices, each with a short runtime, rather than a monolithic application that performs all tasks in a single process.

Lambda is well-suited for microservices with short runtimes because it automatically manages the infrastructure and scales based on the incoming traffic. Each Lambda function is designed to perform a specific task, just like a microservice, and can be deployed and scaled independently. This allows developers to focus on writing code for specific tasks without worrying about the underlying infrastructure.

In serverless architectures, code is executed in response to events, such as HTTP requests or database triggers. When using serverless architectures, it’s common for applications to need to connect to a database to store and retrieve data. And since serverless applications are event-driven, they can generate a large number of database connections in a short amount of time. This can put a strain on the database and cause performance issues or even crashes.

Opening a lot of connections to a database can reduce performance because each connection consumes resources, such as memory and CPU time, on both the client side and the database server side. When a large number of connections are opened, the database server can become overloaded and unable to handle additional requests, leading to slow response times or even crashes.

Additionally, each connection has overhead associated with it, such as authentication and authorization checks. When a large number of connections are opened and closed frequently, this overhead can become significant and slow down the overall performance of the application.

To address this issue, developers can use a database proxy, which acts as a middleman between the application and the database. The database proxy can help reduce the number of open connections to the database, optimize query performance, and provide an additional layer of security by filtering and validating incoming requests.

Databases Proxies and How They Work

A database proxy is an intermediate layer between client applications and a database server that manages the connections and requests between the two. Its primary purpose is to improve the performance, scalability, and security of database communication.

When a client application sends a request to the database server through the proxy, the proxy intercepts the request and acts as a middleware to manage and optimize the communication. The proxy can perform various functions, such as load balancing, connection pooling, caching, and query optimization, to improve the performance of database communication.

One of the primary benefits of using a database proxy is that it can reduce the number of open connections to the database server, which can help improve the server’s performance and reduce the likelihood of overload. The proxy can manage a pool of database connections and reuse them for multiple requests, which reduces the overhead associated with opening and closing connections.

Another benefit of a database proxy is that it can provide an additional layer of security by filtering and validating incoming requests. The proxy can enforce security policies and perform authentication and authorization checks to ensure that only authorized clients can access the database.

On AWS, there are multiple options for setting up a database proxy, depending on the database engine and the deployment requirements.

One option is to use the Amazon RDS Proxy, which is a fully managed database proxy service that simplifies database access for applications by managing and optimizing database connections. The RDS Proxy supports various database engines, including MySQL, PostgreSQL, and Aurora. It uses connection pooling and multiplexing to reduce the number of open connections to the database, which improves performance and scalability. It also provides automatic failover and connection retry logic to ensure high availability of the database connections.

Another option is to use an EC2 instance as a proxy for the database. This approach involves deploying a proxy server on an EC2 instance and configuring it to manage the database connections. The proxy server can be configured to use various open-source proxy solutions, such as HAProxy, Nginx, or PgBouncer, depending on the database engine and the deployment requirements. This approach offers more flexibility and customization options, but it also requires more management and maintenance.

In either case, the database proxy is typically deployed in a VPC (Virtual Private Cloud) and configured to communicate with the database server over a secure network connection. The proxy can also be configured to enforce security policies, such as SSL/TLS encryption, authentication, and authorization, to ensure secure communication between the client applications and the database server.

The Benefits of Using a Proxy with Lambda

Connecting to a database from a Lambda function can pose several challenges, including:

1. Connection management: Lambda functions are stateless and ephemeral, which means they are created and destroyed on demand. Each function invocation requires establishing a new database connection, which can be inefficient and result in a high number of database connections.
2. Scalability: As the number of function invocations increases, the number of database connections required also increases. This can quickly overwhelm the database server and result in degraded performance or even downtime.
3. Security: Lambda functions typically execute within a shared execution environment, which means they share resources with other functions running on the same server. This can pose a security risk, especially when accessing sensitive data in a database.

Database proxies can help overcome these challenges by providing a secure and scalable way to access the database. A database proxy can act as a middle layer between the Lambda function and the database, managing database connections and providing a secure endpoint for the function to access the database.

For example, a database proxy can manage a pool of database connections, enabling Lambda functions to reuse existing connections instead of creating new connections for each invocation. This can reduce connection latency and improve the performance of the Lambda function.

A database proxy can also provide load balancing and query routing to distribute requests across multiple database instances. This can help ensure that the database can handle high traffic and varying demand.

In terms of security, a database proxy can enforce SSL/TLS encryption, authentication, and authorization to ensure secure communication between the Lambda function and the database. The proxy can also monitor and filter incoming requests to prevent unauthorized access or SQL injection attacks.

Proxy in Action

Terraform Code

Let’s take a look at how we can build a lambda function that utilizes a database proxy. We are going to use terraform to build the infrastructure and see how lambda can utilize an Amazon RDS Proxy to improve performance and security.

First, let’s take a look at our terraform resources.

resource "aws_lambda_function" "example_lambda" {
  function_name = "example_lambda"
  role         = aws_iam_role.example_lambda.arn
  handler      = "lambda_function.handler"
  runtime      = "python3.9"
  filename     = "lambda_function.zip"
}

resource "aws_db_instance" "example_db" {
  allocated_storage    = 10
  engine               = "mysql"
  engine_version       = "5.7"
  instance_class       = "db.t2.micro"
  name                 = "example_db"
  username             = "admin"
  password             = "password"
  parameter_group_name = "default.mysql5.7"
  skip_final_snapshot  = true
}

resource "aws_db_proxy" "example_db_proxy" {
  name        = "example_db_proxy"
  engine_family = "MYSQL"
  debug_logging = true
  require_tls = true
  idle_client_timeout = 120
  role_arn = aws_iam_role.example_db_proxy.arn
  vpc_subnet_ids = [aws_subnet.example_subnet.id]
  vpc_security_group_ids = [aws_security_group.db.id]
}

As you can see, setting up a database proxy using RDS doesn’t need to be a complicated process. When you create an RDS Proxy in AWS, the service automatically provisions an Amazon Elastic Compute Cloud (Amazon EC2) instance to serve as the proxy endpoint. The EC2 instance running the RDS Proxy is managed by AWS, which means you don’t need to worry about configuring or maintaining the server yourself. — Note that the code above does not include the IAM roles or security groups required to set up a working environment.

Lambda Code

Next we need to write our Lambda code to use the RDS proxy to access the database.

import os
import pymysql

# RDS Proxy endpoint and database credentials
DB_PROXY_ENDPOINT = os.environ['DB_PROXY_ENDPOINT']
DB_HOST = os.environ['DB_HOST']
DB_USER = os.environ['DB_USER']
DB_PASSWORD = os.environ['DB_PASSWORD']
DB_NAME = os.environ['DB_NAME']

# Lambda handler function
def lambda_handler(event, context):
    # Connect to the RDS Proxy
    conn = pymysql.connect(host=DB_PROXY_ENDPOINT, user=DB_USER, password=DB_PASSWORD, database=DB_NAME)
    
    # Execute a simple SELECT statement
    with conn.cursor() as cursor:
        cursor.execute('SELECT COUNT(*) FROM users')
        result = cursor.fetchone()
    
    # Close the database connection
    conn.close()
    
    # Return the result as a string
    return f'Total users: {result[0]}'

In this example, the Lambda function retrieves the RDS Proxy endpoint and database credentials from environment variables. It then uses the pymysql library to connect to the database through the RDS Proxy and execute a simple SELECT statement. Finally, the function returns the result as a string.

Note that this is a simple example and you would need to modify it to suit your specific use case. Additionally, you would need to configure your Lambda function to use the appropriate execution role and permissions to access the RDS Proxy and database.

Conclusion

When using a microservice architecture, it’s important to understand the benefits of using database proxies.

Database proxies can be particularly useful in high-traffic applications, where multiple Lambda functions may need to access the database simultaneously, and applications with varying demand, where the number of database connections required may fluctuate. Database proxies can also be useful in applications with strict security requirements, such as those handling sensitive customer data or complying with regulatory requirements.

Hopefully this post was able to give a concise summary of why you might need a proxy and the relevant information for getting started.