Lambda event filtering using AWS CDK
Introduction
This write-up is for anyone looking to implement the new Lambda event filtering into their serverless flows using AWS CDK.
Within this post we’re building a sample application that listens to filtered SQS messages, stores data in DynamoDB and captures table activity with filtered DynamoDB streams in order to perform specific actions (e.g. notifications).
Background
Yesterday AWS announced that AWS Lambda supports filtering options for SQS, DynamoDB and Kinesis Event Sources. Until now one had to implement the filtering logic into their own source code leading to potentially unnecessary lambda invocations thus higher traffic, increased code complexity and higher costs. I didn’t even know how bad i wanted this feature.
Sadly at this point there is only CloudFormation support, CDK needs to use escape hatches.
Setup and requirements
We’ll be using yarn (with it’s workspaces feature) and cdk.
We can already open the folder in our IDE.
Edit the package.json and add workspaces configuration as well as some scripts. ☺️
Next we’re going to add the directories for our infrastructure and lambda functions. Afterwards we run a few commands to setup our CDK app and install the CDK modules we need.
There’s only one thing to do before defining our infrastructure:
If this is your first CDK application in your AWS account, you will need to execute a one-time command to bootstrap the CDK:
cdk bootstrapBuilding the stack
Open the code in packages/cdk/lib/cdk-stack.ts and paste the code.
At first we defined the SQS Queue, followed by the lambda function that is supposed to receive the messages. In order for that to work we create an EventSourceMapping.
As mentioned earlier as of today (27th of November, 2021) there is only CloudFormation Support for Event Filtering, in CDK we have to use an escape hatch in order to monkey patch the underlying CloudFormation Template.
The Filter being used filters out all every message that is not valid JSON and has no age property with value 27.
There are quite some filters that we can use:
Creating the lambda functions
As we have defined our infrastructure the next step is to get our hands dirty and produce some ‘production ready’ business logic.
Therefore we need to create some folders and files. We’re also going to use webpack to “bundle” our files, just as an example.
Open the package.json placed in packages/lambdas/ and paste the code:
Now do the same for tsconfig.json and webpack.config.js
After executing a yarn install we’re ready to place our code in packages/lambdas/lib/sqs/index.ts which iterates through all the sqs events and upserts the person record in dynamodb.
We’re almost finished as we create the DynamoDB Worker Lambda in packages/lambdas/lib/dynamo/index.ts. For sake of simplicity we’ll just log the changes.
Declare the module ‘is-base64’ in declarations.d.ts:
declare module 'is-base64';Now we’re able to build our application, deploy it and approve the security changes:
yarn lambda build:prod
yarn infrastructure cdk deploy
If everything went through we can test our application and see event filtering in action.
We head over to AWS console, send a test message to our queue and see that the person record in DynamoDB appears correctly.
Editing the message will only result in other records in our DynamoDB table if the age is 27:
Now matter how often we send messages to our sqs that don’t comply with our filter, our worker lambda will never get invoked.
Changing the mail for the record with primary key “Philip” triggers the dynamo worker lambda.
Cleanup
Don’t forget to delete everything once you’re finished!
yarn infrastructure cdk destroyConclusion
I hope you were able to follow along with this post and understood how to apply event filtering to your serverless stack.
Building applications for specific use cases that use only a subset of all messages that flow through event-driven architectures will help optimize the compute efficiency of lambda functions.
