To decide which ads to serve, our GumGum ad server needs to make decisions not just based on different targeting requirements, but also based on other factors such as the ad goals, RPM (revenue per million), viewability, click through rate etc.
Recently we had a goal to increase the viewability of slot ads. To do this, we decided to add scoring based on the slots’ performance. This required us to store the performance data for slot ads, which means we wanted to record how many times the slot ad had an impression and how many times it was viewable.
- When we receive an ad event for an impression or a viewable event we could write to our performance database ScyllaDB.
- Decouple writes from ad server using AWS Lambda.
What is Lambda?
AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. AWS Lambda automatically scales your application by running code in response to each event, so you only pay for the compute time you consume, and you’re never paying for over-provisioned infrastructure.
Moving to AWS Lambda
We chose to start using Lambda for writing slot performance data, but we were already receiving ad events to our ad server instances which were doing a lot of processing, validations and logging. To decouple writes of slot performance data, we created a SQS queue and started writing messages to the queue from these ad events. The messages from the queue are then ingested by Lambda, and the Lambda simply writes to ScyllaDB.
We started sending around 65,000 messages per minute to our SQS queue which needs to be ingested by the Lambda. Each Lambda when started needs to initialize the DB client which takes a few seconds, so we realized cold start of Lambda won’t help us in achieving our throughput needs. So, our first step was to use provisioned capacity to keep a certain number of execution environments initialized.
At this point, the problem of cold start is gone, but each message is being processed separately, so our next step was to read the messages in bigger batches. We started with reading 10 messages at a time which improved the performance a little, but it could still be further increased. So we then increased it to 50, then 100, and finally 200 which was good for our use case.
Now we are able to ingest messages at a higher rate, but the execution time is still high. We realized that’s because we are performing updates to the database for each message. So to improve the execution time, we started doing batch updates which reduced the execution time.
Finally, we are able to process around 100,000,000 messages a day with execution time of Lambda around 60ms (p95), costing us only about $3.25 a day.
At GumGum we took on a project that required us to store performance data, and we decided to go with moving to AWS Lambda and decoupling writes using a SQS queue. After several optimizations, we were left with an optimal system that is very cost effective.