How to use Java in your DB-connected AWS Lambdas
Intersecting the Java world with the serverless world
Over the last few years, we’ve been hearing new buzz words and trends in IT: cloud… serverless …scaling.. countless new languages.. all somewhat out of reach for us. Some developers were able to get started quickly in those realms with interpreted languages (e.g. Python, JS) and NoSQL databases.
They’re like the popular kids.
But what about us Java geeks, knee deep in full-fledged monolith (or SOA if you’re lucky!) Java apps who want to do serverless?
What about those of us used to frameworks well integrated with SQL databases which hold TONS of data and who can’t just switch to NoSql DBs?
What about if you’ve been doing this for 10+ years and don’t even know where to get started?
As discussed in our architecture post, we needed to create a brand new backend app to manage bookings, so this was our chance. So what are our (realistic) options to sit at the popular cloud kids table?
Disclaimer
I Love My Local Farmer is a fictional company inspired by customer interactions with AWS Solutions Architects. Any stories told in this blog are not related to a specific customer. Similarities with any real companies, people, or situations are purely coincidental. Stories in this blog represent the views of the authors and are not endorsed by AWS.
We examined our choices and decided on a serverless architecture using Java Lambdas and an RDS MySQL database. We decided to keep Java and MySQL since this is what we know best, especially considering the small timeframe we have to stand up our app. We’re already adding enough unknowns as it is.. so keeping our programming language and database choice provides us with at least partial footing on this newly-discovered territory.
Are you trying to do the same? We’ll cover some tricks and caveats and hopefully save some people from committing the same mistakes.
So let’s talk Serverless and Database Connections
During our research we discovered that yes, serverless architectures are awesome, can scale and what not. However, a regular SQL database can run into trouble though with this paradigm, since it cannot scale up its DB connections at the rate that AWS Lambda functions can be spun up.
The database has a finite number of max connections, and when you get there, or are close to getting there, the DB processes start to resemble a relay race by sloths. As the database instance’s resources get drained, the transactions start piling up and pretty soon you’ll find yourself with a crawling and sometimes even unresponsive database.
So how do we solve this problem? Well, you can make your own database connection pooling service (very complicated)… or you can use an RDS Proxy, like we did. The proxy will basically hold on to established connections and farm those out (or create new ones if there’s not enough) to the Lambdas as they spin up.
Besides handling the connection pooling for us, it turns out that the proxy also gives us another extra.. a quicker failover handling when the primary DB instance becomes unavailable.
We have stood up a MySQL Multi-AZ database, i.e. a primary and a failover standby. If for some reason, the primary DB instance goes down (maybe for maintenance, or because it got moody), the (hot-standby) failover will take its place, and the RDS Proxy will start sending requests to the new primary (i.e. former standby). While all this is happening, the TCP connection between the client and the RDS Proxy doesn’t break and since there wasn’t a DNS change for the client, the communication doesn’t fall prey to a possible badly configured or non-respected TTL on the connection.
Authentication
There are 2 choices for connecting to the database, and we wound up using both! These are User/Pwd and IAM auth.
User/Password authentication
This is just the regular user + pwd auth that you normally do with any SQL database. Obviously, you don’t want to keep these credentials in plaintext so people usually store them in Secrets Manager and then configure the ARN of the secret as a Lambda environment variable (for example).
When the code executes, it’ll grab the ARN of the secret, invoke the Secrets Manager SDK to retrieve and decode it, and then use the credentials to authenticate against the RDS Proxy (or database).
We’re using this kind of authentication on our PopulatorLambda, since it needs to login with regular admin credentials to the DB instance to create and seed the schema, as well as create a “lambda_iam” user for the IAM authentication, which we will cover next.
For the Lambda to connect to the RDS Database (not thru the Proxy) via user/pwd, it’ll need:
- an IAM policy granting access to retrieving the credentials from SecretsManager:
"Statement": [
{
"Action": [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret"
],
"Resource": [ "SECRET-ARN-OF-ADMIN-USER" ],
"Effect": "Allow"
}
]Please note that we also added the Secret ARN of the lambda_iam user in our own implementation, because we also needed to retrieve the lambda_iam user’s password, in order to create the lambda_iam DB user in the database.
- a Trust relationship to lambda.amazonaws.com
IAM Authentication
This is the Ferrari of the authentication mechanisms and here’s why: you authenticate by providing a username, but instead of providing the same user/password for all eternity, you generate an authentication token lasting 15 min. using an RDS cert. This is obviously more secure than the other auth method.
Another big improvement that we noticed was that this auth mechanism saved us from having to use SecretsManager in the Java Lambda, which saved us on avg. 1.2 seconds in the Lambda’s cold start.
One thing to bear in mind is though, that there’s a limit around how many such requests can authenticate per second (~30/s). Keep this in mind if your traffic spikes might surpass this limit.
So how does RDS IAM Authentication work?
We need to create a DB user (“lambda_iam”) and store its credentials in SecretsManager, create some IAM policies, flip a few switches, and perhaps do some praying 😅
Here’s what happens:
- The Lambda client connects to the RDS Proxy using the DB user lambda_iam and the authentication token it generates with the RDS cert.
- The RDS Proxy then looks up the lambda_iam user in Secrets Manager, grabs its password, and
- uses it to connect via user/pwd authentication to the RDS Database.
In our codebase, we use IAM Authentication for our REST API Lambdas.
For more info on this type of auth, visit this page explaining how RDS Proxy connects via IAM.
RDS IAM Authentication Checklist
For IAM Auth to work, you’ll need these settings:
Lambda:
- To connect to the RDS Proxy via IAM Auth, your Lambda will need IAM permissions to access the RDS Proxy in its VPC, and connect to it via the lambda_iam user:
- AWSLambdaVPCAccessExecutionRole
- Trust relationship to lambda.amazonaws.com
- An inline policy for the connection via lambda_iam user:
"Statement": [
{
"Action": "rds-db:connect",
"Resource": "arn:aws:rds-db:eu-west-2:123456789012:dbuser:*/lambda_iam",
"Effect": "Allow"
}
]Note: the Resource refers to “dbuser” and “lambda_iam”! Make sure to double-check this IAM statement if you’re getting errors!
For details on how we implemented these policies via CDK, check out the code in ApiStack.java!
RDS Proxy:
- has IAM Auth Only ON
- has TLS Required ON
- has IAM policies granting GetSecretValue and DescribeSecret for both the database admin user and the lambda_iam user secrets:
"Statement": [
{
"Action": [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret"
],
"Resource": ["SECRETARN-LAMBDAIAM-USER", "SECRETARN-ADMIN-USER"],
"Effect": "Allow"
},
]- Trust relationship with rds.amazonaws.com
Check out this CDK implementation in DbStack.java.
RDS database:
- the DB has IAM Auth turned OFF (since if the RDS Proxy is accepting IAM Auth, your RDS DB is only allowed User/Pwd auth)
- need a User created on the database for the RDS Proxy to authenticate with:
CREATE USER IF NOT EXISTS 'lambda_iam' identified by '{{password}}';
GRANT DELETE, UPDATE, INSERT, SELECT ON deliverydb.* TO 'lambda_iam';
FLUSH PRIVILEGES;Secrets Manager:
There should be 2 secrets:
- One created automatically by the RDS Proxy created in DbStack.java, which stores DB credentials (e.g. host, username, pwd, port..)
- One created in ApiStack.java with just the lambda_iam user and its password
To Hibernate… or not
Hibernate is a heavyweight, common ORM tool in the Java server world, and we are quite comfortable with it since we’ve been using it for over a decade. We leverage its lazy-loading and caching capabilities quite extensively, and mapping capabilities. In short, we like it.
In this case, we are trying to get away from monolithic applications and want to deepen our footing in the microservice world, and do so in the cloud by using AWS Lambdas.
Microservice implies API calls with only a small set of functionality. Microservice implies only running a DB query or two in each call (or a few, if you’re generous). So because it’s a microservice and AWS-Lambda-based one at that (i.e. not container-based), importing a heavyweight ORM library such as Hibernate in order to do a couple of queries seems analogous to stomping an ant with an elephant. For our case, it’s overkill.
So… if not Hibernate, then what?
Well, as strange as it seems, we are stepping into that time machine and going back to the early 2000’s… in plain ol’ JDBC times. And yes, It feels like a bad dream, quite frankly.. but considering all we need to do are simple Selects, Inserts, and Updates (and even an occasional “complex” Select with a join), it does seem excessive to set up an entire framework instead of just writing a few lines of code.
You can catch how we used JDBC CRUD statements in our Lambdas in SlotService.java.
