Enterprise Serverless 🚀 AWS Limits & Limitations
This section in the series on Enterprise Serverless specifically covers some of the limits and limitations that you are bound to hit when working on large scale serverless projects, and how to solve or work around them. It’s a list that I will continue to update as things come to mind. You can read the ideas behind the series by clicking on the ‘Series Introduction’ link below.
The series is split over the following sections:
- Series Introduction 🚀
- Tooling 🚀
- Architecture 🚀
- Databases 🚀
- Hints & Tips 🚀
- AWS Limits & Limitations 🚀
- Security 🚀
- Useful Resources 🚀
AWS Limits and Limitations
The following are some of the typical issues you may face in large serverless projects and how to work around them or factor them into your overall architecture and designs:
Max foreach loop iterations in AppSync
When using AWS AppSync with VTL there is a limit with loops that you should be aware of otherwise it can result in rework. As detailed on the AppSync limits page, the max number of iterations in a foreach loop is 1000, which is hugely annoying when projecting values in a DTO for example.
The easy workaround in this scenario is to use a lambda resolver, as although possible in VTL, batching is a lot of work and you are unable to share code easily i.e. no functions.
Lambda Resource Names Max Length
An issue you can hit quite often in serverless is hitting the maximum 64 character resource name for a lambda. The easy way to get around this is to obviously just shorten the name, however when auto generating code for example this is one to think about up front!
CloudFront deploy time
As discussed earlier in the series AWS CloudFront deploy times can take some time to deploy and slow down pipelines — so it may be worth looking at CloudFlare workers.
Max resources per stack
Discussed in the tooling section, serverless-plugin-split-stacks is a fantastic plugin to adopt into your serverless projects from inception as you will no doubt hit the maximum resource limit of 200 when working on large serverless projects, and this plugin splits the stacks into further stacks of 200 resources.
CloudFormation secure strings in Parameter Store
An annoying limitation with AWS Parameter Store with CloudFormation is being unable to create secure strings, which I assume would be required in production environments in a lot of projects. The way around this in past projects is to use non secure strings in QA or ephemeral environments, and PP and Production to have the parameters added via the pipeline.
CloudFormation Parameter Store value changes
If you expect the parameter store values to be updated when changing them in your CloudFormation then you would be wrong! This is an issue that I hit earlier on in a recent project, and one workaround is to simply use the AWS CLI instead; although I have seen some interesting ways suggested to get around this issue such as adding a tag with a UUID generated dynamically to force the change.
Max AWS IAM Roles
When working with ephemeral environments with least privilege in mind when it comes to AWS IAM roles and security, and with a couple of scrum teams, you can quite quickly hit the maximum amount of IAM roles for the account on large serverless projects.
One suggested way around this is to have an AWS account per developer, which is something I have never tried personally on a project. For recent projects this is something that has been worked around, but good to keep in mind.
AppSync shared regexes, messages & resources
When working with AWS AppSync and VTL one of the more annoying issues when working on large scale serverless projects is the lack of reusability when it comes to shared regexes, info/error messages and custom resources such as i18n translations, as for the most part your VTLs are static code i.e. no imports of modules.
One way to mitigate this on projects using serverless is to use the substitutions property in the serverless YAML/JSON which then allows you to use injected values (key value pairs) and then replace in VTL code to reuse resources such as regexes or config values. An example of using a substitution value for a person name regex in VTL is below:
#set($valid = $util.matches(“${personName}”, $ctx.args.person.name))AppSync max response size (1MB)
A previously undocumented issue which I hit on one serverless project was hitting the maximum response size in AppSync which was around 1MB. The particular scenario was returning base64 data from a legacy system which in a lot of instances was over the limit. The workaround in this scenario was to store the images in S3 and return the S3 path rather than the base64 data, but this is something to think about up front when designing your APIs.
75GB hard limit on Lambda
For a given AWS account the maximum overall lambda function and layer storage is 75GB which is sizeable in all fairness, but one to keep in mind when working with large numbers of deployed lambdas per ephemeral environments.
This is where building the lambdas using plugins such as serverless webpack come in to their own, where tree shaking and optimising builds means your lambda size is greatly reduced, as well as ensuring that versioning your lambdas is turned off in serverless by setting versionFunctions to false.
Max payloads between step function tasks
AWS Step Functions have a limitation of 32KB of data being the maximum size of payloads between tasks (inputs and outputs), which should be thought about when designing your workflows. The best practice to mitigate this issue is using AWS S3 to store data between steps which in experience works very well on most projects.
DynamoDB TTL may live longer than expected!
When working with DynamoDB and using TTL you would be forgiven to assume that your records will be removed instantly when hitting the TTL time — however in my experience you would be wrong. On average I have seen this take up to 15 minutes so it is worth keeping this in mind when working on a feature.
When using AppSync with DynamoDB for example, I have gotten around this issue in projects by including the TTL in the query expression to ensure the record does not come back if it has not been cleaned up by AWS yet.
Next section: Security 🚀
Previous section: Hints & Tips
Wrapping up
Lets connect on any of the following:
https://www.linkedin.com/in/lee-james-gilmore/
https://twitter.com/LeeJamesGilmore
If you found the articles inspiring or useful please feel free to support me with a virtual coffee https://www.buymeacoffee.com/leegilmore and either way lets connect and chat! ☕️
If you enjoyed the posts please follow my profile Lee James Gilmore for further posts/series, and don’t forget to connect and say Hi 👋
About me
“Hi, I’m Lee, an AWS certified technical architect and polyglot software engineer based in the UK, working as a Technical Cloud Architect and Serverless Lead, having worked primarily in full-stack JavaScript on AWS for the past 5 years.
I consider myself a serverless evangelist with a love of all things AWS, innovation, software architecture and technology.”
** The information provided are my own personal views and I accept no responsibility on the use of the information.
