AWS Lambda Event Validation in Python — Now with PowerTools
This blog has an updated version at my new website, Ran The Builder.
In my previous blog post “AWS Lambda Event Validation — from Zero to Hero”, I showed how to parse AWS Lambda event schemas correctly and how to handle event validation exceptions.
The “weapon” of choice was an excellent Python library called Pydantic.
In this blog post we’ll take it up a notch and improve our validation solution even further by using a new & awesome library called AWS Lambda Powertools.
If you’re unfamiliar with event validation with Pydantic, check out Part 1 in this series.
So, why should you continue reading?
Well, for 3 simple reasons:
- You will learn how to reduce your validation code in your lambdas.
- You will offload the maintenance of Pydantic based AWS events schemas from your code to AWS.
- You will be introduced to one of the best new AWS Lambda related libraries which might solve you other issues you didn't even know you had.
How can you improve on the already excellent Pydantic validation?
Recently, I had the pleasure of contributing a new parser utility code to an amazing and relatively new project on Github: AWS Lambda Powertools.
This repo, which started as Python oriented (but now supports other languages such as Java and more to follow) provides an easy to use solution for lambda logging, tracing (with cloud watch metrics), SSM utilities and now validation and advanced parsing for incoming AWS Lambda events.
The new parser utility, will help you to achieve next level validation. It’s based on Pydantic and it’s marked as an optional utility. In order to install it, you will need to specify it like this:
pip install aws-lambda-powertools[pydantic]
For other non parsing usages for the libraries such as logger, metrics (and more) see this excellent blog post.
Validation with Pydantic
Well, currently, if you followed my guidelines in my previous blog post, you had to write (and now maintain) a handful of AWS Lambda events such as Eventbridge, DynamoDB streams, Step functions and more. Basically, a schema for each AWS event that a lambda receives.
You found out how to write these Pydantic schemas by either looking at the AWS documentation or by printing the event JSON.
It’s an important process it can get tedious quickly.
Let’s take a look at an Eventbridge event:
The detail field is a dictionary which describes the user schema, the actual message that we would like to extract, validate and parse.
In order to achieve that, the following Pydantic schema can be used:
The lambda handler which parses the event will look like this:
…and it works. You have to write a little bit of code but it works.
The problem in this solution is that you need to maintain an AWS event schema which might get changed by AWS at some point. When it does, this validation schema fails and raises a validation exception in runtime. Not ideal to say the least.
However, it can get better, much better!
Validation with AWS Lambda Powertools
Let’s take a look at the following lambda handler implementation:
Let’s make several observations here.
- We don’t use EventBridgeModel anymore and are no longer required to maintain it.
- The validation code has been shortened into one line! a simple decorator.
- The received event in the lambda code is of type UserModel which is more accurate than Dict[str, Any] as it was before.
How does it work?
The decorator receives a model and an optional param as envelope. Envelope is a class that knows how to parse an AWS event schema, validate it and extract the UserModel from the AWS event.
In this example, the EventBridge envelope will extract the model UserModel from the EventBridge schema, and trigger the handler with the parsed UserModel as the event parameter. Envelope is an optional parameter and in case you don’t supply it, the parser will parse the event while assuming it is a UserModel input.
At the time of writing, the currently supported envelopes are EventBridge, Sqs and DynamoDb streams with more to follow.
In case you don’t want to use a decorator you can use instead the parse function which works just the same:
You can find more details and more use cases here.
Handling parsing exceptions
Be advised that validation errors can arise in case the input is malformed. The validation exception will contain detailed information regarding the malformed fields.
Both the event_parser decorator and parse function can raise ‘ValidationExceptions’. There are two ways to handle this:
- Use the parse function instead of the decorator and add an except clause to handle the exceptions in the manner you see fit.
- Don’t add except clauses (use either parse function or decorator) and set your Lambda function retries to 0. In addition, send failed invocations (such as any failed validation inputs) to a dead letter queue.
The reasoning behind the second approach is that any failed input validation events will also fail in retry, so there’s no point in retrying.
Extending the AWS events schemas
Let’s assume that you wish to use the parameters from the EventBridge model (such as region or source etc.) and also get the extracted UserModel schema You can achieve that by extending the model from AWS Lambda PowerTools:
There are 3 options for writing the lambda handler here.
The first solution uses pure Pydantic, the second one uses the event_parser decorator and the third one uses the parse function. You can choose any solution that you like.
Please note that:
1. An envelope parameter is not needed in this case.
2. The parsed event is of type ‘MyEventBridgeModel’, which means it has all the fields from both schemas, EventBridge and UserModel!
Let’s sum it up
By using AWS Lambda Powertools parser utility, you can improve your AWS event validations in 2 ways:
- You reduce the parsing & validation code making it simpler and more readable.
2. You don’t maintain AWS lambda events schemas.
Head over to the repo’s github page, and start parsing ;)
About Me
Hi, I’m Ran Isenberg, an AWS Community Builder (Serverless focus), a Cloud System Architect and a public speaker based in Israel.
I see myself as a passionate Serverless advocate with a love for innovation, AWS and smart home applications.
Connect with me at https://www.linkedin.com/in/ranisenberg/
