Spot-on error management with AWS Lambda, API Gateway, CloudWatch and Serverless
Strange as it is, I believe having bugs is healthy. Because too often, an app that has no bug is an app that has no monitoring system set up to detect them in the first place.
I’ve recently discovered the serverless ecosystem. To be more precise, I worked on a project that used the Serverless framework to deploy NodeJS Lambda functions, connected to, among others, API Gateway and CloudWatch.
After two months of development, we realized that we needed an efficient way to throw and detect Lambda execution errors. We also realized that it was not that trivial: It took us a few weeks to get a reliable system in place.
This article shares some of the learnings we made on the way. In particular, how to:
- 🏓 Throw and harness errors throughout your Serverless app
- 🚦 Return relevant HTTP responses, with customizable status codes and payloads
- 🎯 Monitor your execution errors in CloudWatch
- 🔧 All that with minimum boilerplate and maximum reusability
Sounds interesting? Let’s dive in!
#1 Throw and harness errors
Let’s begin by creating a CustomError class that extends the original Error object. It should stay agnostic enough to be thrown anywhere (even in a shared util), yet carry enough data to be correctly used when needed.
Here, we will add two properties to it:
- statusCode: A related HTTP status code that can be parsed by API Gateway if needed.
- key: An string identifier that adds context to the status code and can be used for front-end translation purposes.
Now, we could harness those errors by wrapping our Lambda functions with a try/catch, but let’s use middy.js instead: If you’re not familiar with it (and you definitely should!), Middy uses the middleware design pattern to allow a better segregation of business logic from configuration such as JSON validation, logging, or in this case, errors handling.
Neat! So what does our handleError middleware looks like? In the case of a synchronous lambda, it depends on how it is integrated to API Gateway.
#2 Proxy integration
When connecting a Lambda function to API Gateway, one can decide whether to activate proxy integration or not. Serverless officially recommends the use of proxy integration, and made it its default Lambda function integration, as it requires less configuration.
In a proxy integration, API Gateway passes the raw request as-is (headers, query string parameters, URL path variables, payload, authorization context etc…) to the integrated Lambda function, and expects a similarly integrated response from its return results. Meaning that any status code will be correctly interpreted by API Gateway to build the request response.
With that in mind, we can make full use of our custom errors with a proxy HTTP response building middleware:
However, in this configuration, CloudWatch considers that the Lambda function was executed successfully, and the thrown error is not captured.
Alternatively, if we throw the error in our middleware (or, equivalently, pass the error either as the first argument to the callback), the error is captured, but API Gateway does not interpret it and always returns a 502.
The only way to have custom errors both captured by CloudWatch and interpreted correctly by API Gateway is to move from proxy to lambda integration.
We can do that by adding integration: lambda to our Lambda functions configuration. This had small implications on the shape of our Lambda functions events and responses, but nothing unmanageable*: You will find a detailed implementation on the GitHub repository that I made along with this article.
*Largely thanks to the default integration mapping templates set up by Serverless
#3 Lambda integration
Moving from proxy to non-proxy integration enables the configuration of integration responses, which specify how a response status code, body and headers are to be mapped from the Lambda function returned value.
The good news is that Serverless sets up default method and integration responses that do most of the job, so no additional boilerplate is needed! Check the Serverless documentation for more details. Long story short: Throwing any string containing [400] will actually trigger a 400.
We can use this to our advantage by reworking our formatting middleware:
And, sure enough, there it is! The best of both worlds:
Note that this only worked for the status codes set up by default by Serverless. So we should add a check in our custom error constructor:
#4 Handling non-custom errors
For the moment, we’ve considered all non-custom errors as 500s, but that is not something we always want.
We can improve this by detecting and parsing incoming errors, and using 500s only as a last resort. For instance, Axios errors have a handy isAxiosError property:
We can create such parsers for each type of error we can encounter, and give them to a parseError function:
Finally, we can use this function in our formatErrorForApiGateway middleware:
And voilà 👍 Thanks to Serverless and middy.js, we set up a neat error management across our synchronous lambdas and API Gateway with minimum boilerplate!
#5 Using mapping templates
However, the purists among you may be frustrated by the shape of our errors:
The error key is already troublesome to parse, any customization would be hard to implement (what if you want to pass a payload ?), and forget about using any other status code than the eight ones provided by Serverless!
But fear not, there is a way ! Called mapping templates.
Mapping templates are “recipes” that API Gateway use to reshape responses after standard mappings have been applied, giving control on the responses bodies, headers and status codes. Different templates can be triggered depending on the content of the Lambda function return values, through matching patterns.
Let’s use them to replace the body of our error response body by the value in the error message:
Now, let’s shape the error body the way we want:
And voilà! Nice and clean:
Last but not least, we can extract the response configuration in a separate file to avoid having to apply the same mapping templates to each Lambda function:
Conclusion
To summarize:
- Activating proxy integration for your Lambda functions requires less configuration, but hides execution errors from CloudWatch
- To solve this issue, Lambda integration with default Serverless integration responses is a good way to go
- Mapping templates will give you maximum freedom on the shape of your HTTP responses, at the cost of some additional boilerplate
Feel free to check out the GitHub repository that I made along with this article: You will find detailed implementations of each solution, as well as correct Typescript types and a bonus sendMessage middleware which sends an EventBridge event when an error occurs 👍 It’s ready to deploy if you want to try it for yourself!
Thanks for reading 🙏 I hope that you learned something from this article and that it will save you some time on your next Serverless project!
