Is Serverless with Lambda, the “silver bullet”?

Photo “Passion Led Us Here” written on the ground
Photo Ian Schneider on Unsplash

What should we start with?

We took our first step in Serverless with a non-core business feature: the management of our surfers’ GDPR consent.

  • It was a new feature
  • more or less isolated
  • with a short time to market target, due to European regulations
  • Coding Language
  • Size of your code/package
  • Memory allocation to run your Lambda

Is an AWS Lambda fast and robust enough to support a Search API?

Immoweb is a digital real estate classified player; our core business relies heavily on a robust and fast search feature. With around 500.000 visits and millions of search queries per day, there is little to no room for failing the search API rewrite.

The Search API

The current architecture is quite common. The clients, our website and mobile applications, are calling a CloudFront cache. If the cache already exists for the same search criteria, the CloudFront sends directly the answer to the client (response in +- 10ms). If the cache doesn’t exist or if the cache has expired, the request is forwarded to the API Gateway which triggers the Lambda function. This lambda is responsible for translating the query into the Elasticsearch query format, getting the response and executing some business transformation into a nested readable JSON format sent back to the client.

Api schema using Cloudfront, Api Gateway , Lambda and ElasticSearch
Classic API design in Lambda Serverless

Event & Lambda, the “silver bullet” that might kill you?

Mastering Lambda takes time and sometimes you can face a really bad situation like we faced recently. In order to insert several documents in one DynamoDB, we decided to implement a simple ETL in Serverless with the following architecture. Files are uploaded on a S3 bucket, a Lambda function is triggered on PUT and POST events, when a file is uploaded. This Lambda reads and transforms the data to be loaded into the DynamoDB. It’s a pattern that we had already used on another project and it was working well .

File are uploaded on S3 bucket, this event trigger a Lambda function to store data in DynamoDB
Lambda invoked by S3 Events

Serverless = Limitless : Lesson learned

As discussed with some colleague, back in time, when we were managing our own datacentre and servers, there was a physical limit like hard drive space, numbers of machines, that were limiting the outbound…

Usage and Budget Alarms

It seems obvious, but always set several budget Alarms, a good practice is to do it on different percentages based on the previous month usage. For example, set several alarms at 60%, 70%, 80%, 90%.
If you receive several alarms over a short period of time, there is something fishy in your account.
You can also set alarms for the consumption of several services.

Lambda concurrency

In some environments (like dev, or test), limit the amount of executions a Lambda can run in parallel, per lambda. In our case, if we limited it up to 10 concurrencies, we would be able to divide the bill by 100.
Link : https://aws.amazon.com/about-aws/whats-new/2017/11/set-concurrency-limits-on-individual-aws-lambda-functions/

Logs and CloudWatch

CloudWatch is powerfull but it can be expensive. It’s recommended to limit the amount of data you are ingesting into CloudWatch.
You can set an appropriate log level according to the environment and to the need (don’t log your debug trace in production)
Force developers to output the bare minimum into the log trace.

Manage S3 events carefully

Learn how events are generated and carefully measure the event types and the paths where you are listening to these events. It’s better to link an S3 event on a dedicated folder for a specific file extension instead of placing the listener on the root of the bucket triggered on every object.

Evaluate other Amazon services

Amazon offers hundreds of services, sometimes some existing features or managed services offer better results in terms of efficiency/cost than developing your own services based on serverless functions. Creating a simple ETL with Lambda might be a good solution, but always compare it with services like Glue, Data Migration, that might improve your efficiency and keep your cost under control.

Tips to start the right way

The serverless world is evolving at a fast pace, with features and improvements added constantly. It’s truly a game-changer, helping developers (backend, API, DevOps, Big Data,…) to answer faster to business needs while decreasing the infrastructure complexity. To avoid some pitfalls and start serverless the right way, I recommend following the AWS Heroes Serverless Community and especially Yan Cui and James Beswick one of the AWS Serverless “Evangelists”.

Ressources:

Tunning Lambda :

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store