Benchmarking AWS Lambda runtimes in 2019 (Part II)

We continue benchmarking AWS Lambda…

  • AWS Lambda execution durations are stable
  • API Gateway and AWS Lambda are reliable, no errors
  • API Gateway latency accounts for more than just code execution in AWS Lambda
  • Package size isn’t always an indicator for faster cold-start, demonstrated by Golang and Ruby
  • Python and Node.js have great performance

Benchmarking process

In this case, we reduced the number of runtimes we benchmarked down to 5, Python 3.6, Node.js 8.10, Java 8, C# (.NET 2.1), and Haskell.

Lambda Pre-requisites

AWS Lambda functions interact with other AWS services or APIs through libraries. In this case, we needed a library to serialize and deserialize JSON inputs and outputs, a library for date/time, and another library to interact with AWS DynamoDB. Official AWS SDKs are available for C#, Python, Node.js, and Java and non-official AWS SDKs for Haskell such as Amazonka or AWS SDK.

CRUD — Benchmark Result

For this benchmark, similar to the procedure we used in Part I, we performed 20 requests per second (rps) for a period of 1200 seconds. Making a total of 120,000 requests per runtime.

Cold-start

We were expecting an increase in the cold-start due to the addition of dependencies. However, we were surprised by the dramatic increase for Python and Node.js, by around 30x and 35x respectively in the worst case scenario.

Execution durations

Once we get over the cold-start period, we start seeing very stable execution times for C#, Python, and Java. However, we got some spikes in Haskell and Node.js for all the CRUD operations that were performed.

Create product — maximum execution duration
List product — maximum execution duration
Get product — maximum execution duration
Update product — maximum execution duration
Delete product — maximum execution duration

Memory usage

Memory usage increases when performing AWS DynamoDB reads or writes. However, it is still below 128 MB for all the runtimes except Java, which demanded a maximum of 147 MB in some of the requests.

Lambda read/write to DynamoDB, maximum memory usage

Errors

We performed a total of 720,000 requests against 25 different functions, 5 functions per runtime, and we only registered 4 errors in Haskell functions, which were caused by timeouts.

Package vs max cold-start

  • Python 5.1 KB 118 ms
  • Node.js 6.0 KB 201 ms
  • C# 633.5 KB 784 ms
  • Haskell 5.8 MB 1300 ms
  • Java 10.1 MB 4050 ms

API Gateway latency

We continue to be interested in the full duration of the request from the point in which API Gateway receives the request until a response is returned to the client.

CRUD — API Gateway latencies without cold-start

Conclusion

We’re happy with the results and we feel that there’s good potential for improving not only AWS Lambda but also the synergy with AWS API Gateway, for example, when provisioning VMs to execute the code. We will continue to monitor how the different runtimes evolve in this rapidly changing environment.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Tai Nguyen Bui

Software Engineer @theagilemonkeys and passionate about Tech and Motorbikes