You Have to Learn How to Make the Best Serverless Architecture

One does not simply put everything into AWS Lambda.

Preface

Quite some time ago I’ve started putting every little thing I develop out there for everyone to use for free (a.k.a. open source). Frankly, I never expected anything from doing so, but all of the sudden, I’ve begun getting questions/requests/suggestions, which is exciting.

One of the questions led me to an hour-long conversation on architecture for a small project. I was trying to help a person I bearly knew (let’s call him Bob). This kind of things makes our lives interesting!

Anyway, the small chat made me think that it can be quite tricky to come up with a serverless architecture even for a simple case.

Here is the problem that we’ve discussed.

Problem

The problem and the given solutions are very AWS specific. However, these approaches can be used in many other Cloud Providers

A company has multiple clients that send some incremental updates in XML format. There is a need to create a service that would ingest the updates with the lowest possible maintenance effort.

Attempt 1. API Gateway + Lambda.

Bob wanted to go the most obvious path — create an AWS Lambda and expose it via AWS API Gateway. He discovered bleshik/aws-lambda-servlet and wanted to use it for the solution. Since, I’m the maintainer of this library, he reasonably decided to ask me what I think about the idea.

After a small talk, it became obvious that AWS Lambda is not even able to ingest the updates because they are often larger than the largest allowed AWS Lambda payload, which is 6 MB.

Attempt 2. S3 + Lambda.

AWS Lambda alone cannot solve the given problem. However, you can still go with the serverless approach with 1 additional step — use S3.

Easy. Make the clients export the data to a dedicated S3 bucket, then automatically trigger AWS Lambda when a new file is added there.

Seems to be perfect. However, Bob could not go with this approach, because there is an additional requirement on how exactly the file is uploaded — it has to be an HTTP POST request. The uploading piece of logic is supported by another team, and it’s troublesome to make them change the logic too much. I’ve worked in big companies, so it’s understandable.

Attempt 3. API Gateway + S3 + Lambda.

Nowadays, the most popular serverless combination is AWS API Gateway that exposes AWS Lambda functions.

However, AWS API Gateway may expose many other AWS services, including AWS S3. Thus, you can create an HTTP endpoint that uploads data to S3. Then, AWS Lambda can be triggered on that event automatically.

Moreover, the API can be made private using the API Gateway’s Resource Policy. This is important because it was one of Bob’s requirements to have it available for a given VPC only.

Summary

While the idea behind the Serverless approach is very simple, it’s not always easy or justified to go that way. It’s not a silver bullet. Do not get blind and fall to what’s popular, instead use common sense, and do what’s reasonable for the final goal.

Also, keep in mind that a “Serverless” service is just a fully managed one, provided by someone else, there are many different kinds of such services. Therefore, there many ways of solving same problem, just keep searching for the best one for you.

Comments, likes, and shares are highly appreciated. Cheers! ❤️