Serverless implementation for real time priority engine
We are building a new data-in system these days to replace our legacy system which was written few years back, got complicated as years went by and our business grew.
The data-in system receives offline files from our Data Providers and DMP Customers. It can get the files from many endpoints, in many formats and many structures. We typically get a thousand of files per day, containing ~2 billions of events. These events are being processed by our serving system after a transformation is made to a unified format and structure.
One of the 3 key parts of the system is the priorities engine (a.k.a ‘Prioritizer’), which chooses the next files to handle based on priority levels and fairness.
It gets requests from several sources.
Initially, we implemented the Prioritizer as a dedicated Java server with REST API. As battle-scarred java developers, that’s what came to our mind first (I say Java but we do write everything in Scala for the last 2 years).
Soon after we realized it’s better to have it as serverless AWS Lambda.
Why serverless?
The prioritizer is stateless (states are stored in its db) and its computations are pretty short, which makes it natural candidate for AWS Lambda.
The ‘cool’ argument is pretty strong but in addition, it gives us a better response to the following issues:
Availability
The priority engine must be highly available, otherwise we might miss a request with new files or a status update.
It means we need a replication for the server, which means double monitoring, double maintainance, etc.
Pricing
Although we handle sometime thousands of files a day, the Prioritizer handles no more than 20k requests. We definitely don’t use the servers all day long (and two replicated servers double the idle time).
So ‘pay only for what you use’ fits us.
Doing the switch
AWS Lambda’s principle ‘bring your own code’ allowed us to do the switch easily with leaving all the business logic unchanged and do only minor changes in code:
Implementing Handler Method
The method in the code where AWS Lambda begins execution. The methods is getting two arguments: input and context.
The input can be either:
- Pre-defined object format for AWS integrations & events — for example: when Lambda is triggered by s3 event like ‘ObjectCreated’, the input type is S3Event.
- Java’s simple data types, POJOs, and Stream input/output
- Scala’s case class is not supported out-of-the-box, but there is a way to use it.
Adding a Handler method per Lambda Function was the switch’s main change.
Configuration
can no longer be read from an external file. All configuration params are passed through Environment Variables which are part of the Lambda setup.
Jar limitation
Today, the limit for the jar size is 50MB, and 250MB is for the uncompressed code. That’s AWS ‘s official requirement, although it was proven to be much more than that.
We use Maven’s shade plugin which creates a fat jar, so we had some clean-up work to do — excluding as much dependencies as possible.
API
All the Prioritizer’s clients reside in AWS ec2/emr. Therefore, we use the Lambda’s java API to invoke the functions. All the REST layer code was removed.
If REST api to the external world was really needed here — AWS’s API Gateway gives the missing layer (use Signature Version 4 to add authentication information to the requests).
Performance
Performance is not a huge issue, as it’s a backend offline flow, but you do pay in Lambda for the duration so it is a concern.
Therefore, we wondered if database connection is created per request.
Based on AWS documentation, not necessarily:
“After a Lambda function is executed, AWS Lambda maintains the container for some time in anticipation of another Lambda function invocation. In effect, the service freezes the container after a Lambda function completes, and thaws the container for reuse, if AWS Lambda chooses to reuse the container when the Lambda function is invoked again.”
But they also say:
We did some tests and measure the requests duration. We did that in order to understand whether the db connections are being re-used in our case.
The results were satisfying: only first requests after a long idle time took longer and most of the requests were much shorter. For example, one of the Lambda functions took us 4 seconds when called hourly, but took 900ms when called in few seconds frequency.
Deployment
Uploading a jar to multiple Lambdas can be quite a hassle. Every time you build your jar, you need to update manually all Lambdas.
We created a Lambda function which solves it and auto-deploy jar to all relevant Lambdas once it is put in a dedicated s3 folder (as explained here).
Monitoring and Alerts
There are few basic built-in metrics in Lambda (Invocations, Errors, Duration and Throttles), and alerts can be set for them through CloudWatch. For applicative metrics we use Graphite.
Conclusion
Although the priority engine servers weren’t pricy, by using the Lambdas we spend few cents per month. The switch was pretty smooth.
It was a conceptual experience and now that we know and understand it, we plan to extend the use of this approach in future projects.
