Serverless, Promises and Predictive Reordering

Lately, we hear about how everyone and everything is serverless, a concept that can be described as multi-layered functions - think JS promises running your entire app logic anywhere. Amazon has lambda, Google has functions and Azure followed with the same. Should every engineering team be going serverless?

A few words about serverless

Martin Fowler to the rescue! Martin describes serverless at length, but the TLDR; version is as follows:

• the supported use case is event based
• it functions in a stateless way
• lastly, someone else manages that complexity to run your functions for you, you don’t need to worry about where and how (sort of) it runs.

Think 10 years ago about all the ops-to-devops migration brainwashing: “devs and ops need to work together vs throwing your apps over the fence!” Well think again since we are taking it back, you should throw your code over the fence and someone else will figure out how to run it. You trade not dealing with servers for freedom: you can’t choose your most desired frameworks, can’t choose your own language, can’t choose which HTTP protocol to use. Someone else dictates that for you but you gain a lot, mostly around server management. The word “serverless” is a little confusing since ultimately there are servers (mostly container based) that execute these functions and, as we are going to see later you do need to be aware of that.

But I’m an engineer, freedom is my middle name!

Can we get more freedom and develop something that gives you the ability to choose which HTTP protocol you want to use such as chunking or sockets, or maybe streaming so I can return a 100GB video response from my Lambda function?

The beauty about our industry is the predictability of new trends based on past trend behavior. In order to identify trends, one should always look at the history and identify a trend based on current market behavior. So lets give it a try and see what trade offs were made between freedom vs ease of use:

• php in the 90’s started with the same idea of dropping html+php function on a remote server and that worked anywhere, not a lot of freedom but you get ease of use.
• J2EE with their stateless session beans was the first time someone tried (emphasis on tried) to build a fully managed environment that runs our code on a remote server, again very similar concept to what Lambda is trying to do today but at the expense of freedom.
• In 2006 we started to see managed cloud environments such as AWS (freedom: yes! ease of use: meh…)
• In 2008 Google AppEngine which is a Java/python environment trying to give you sort of a serverless environment (with way too many xmls and configs).
• Then we switched to platforms that give you even more freedom like docker.
• Parse in 2011 is another example of a platform that gave you the ability to run remote functions without the need to manage how they run, again trading freedom in favor of ease of use. Parse turned into an API platform for mobile apps (and in 2017 was shutdown by Facebook).
• Lastly, Lambda in 2014 came and removed freedom for better ease of use/functionality.

In conclusion, if your team favors freedom over everything else, these serverless platforms aren’t trending there and in fact are trending in the opposite direction. Sure, you will have more flexibility than today but don’t expect they will achieve freedom at any point.

What about consolidation? Will I be able to run my GCP function on top of Azure and vice versa?

Looking at the consolidation frameworks at the PaaS layer, the answer is most likely yes. Maybe we will get one API that can run serverless functions across all vendors but it will probably be OK at best (ie ManageIQ) since each vendor will have “added features” not supported by the consolidation layer.

What about the cold start problem? Can we solve that? Can we bring up and run an instance in less than 100ms?

Breaking down the cold start problem for a given function:

1) we need a scheduler when and where to run

2) we need to run it somewhere, a VM (1–2min) or container (1–2sec)

3) serving the function

All 3 take time, hence most serverless vendors don’t offer SLA for cold starts. The few tests that I ran showed me that you spend 600ms to 2sec to boot up a new function. To avoid the need to boot up a function every time all vendors use some sort of “keep this function live”, essentially caching the container somewhere. This overhead is what makes serverless great for offline events (rest API response for email opt-outs), but it’s not going to play nice with serving HTML. Think about the user on the other side waiting for the response. Think about google indexing: it’s expecting the first byte to respond within 100ms and finish serving the content in less than 1–2 sec, otherwise Google will penalize you for being slow. Hence functions are less than ideal for this use case. Most serverless functions execute (assuming they use caching for cold starts) between 75–200ms: not bad compared to where we started but also not great for HTML content.

Based on the above, Serverless are here to stay, they keep evolving towards better performance, a little bit more freedom, and as long as you don’t mind the vendor lock-in they will greatly fit your event based use cases.

At OrderGroove we use Twilio for text based messaging to communicate our predictive AI recommendations. The predictive text reminders we send are a stateless offline event that fits perfectly with serverless functions and as such, last year, Twillio announced their support for serverless functions. Given where serverless platforms are headed this is a logical evolution of our predictive reorder solution.

Stay tuned for more…