7 checkboxes to compare serverless orchestration services

Bas van Essen
Mar 13 · 12 min read
Example of a serverless architecture. Source: martinfowler.com

Since 5 years ago, major cloud providers and new vendors have slowly but surely introduced serverless architectures that include custom code run in managed and temporary containers. They help us as developers to reduce our operational costs, complexity, engineering lead time and thus make us capable of completing software projects sooner, including new requirements meanwhile coming to the surface. But how to compare all those vendors? An interview with scientists that made a test framework.

A team of six computer scientists from the University of Rovira i Virgili Tarragona, existing of Pedro García López, Marc Sánchez-Artigas, Gerard París, Daniel Barcelona Pons, Álvaro Ruiz Ollobarren and David Arroyo Pinto, recently introduced a new paper called ‘Comparison of Production Serverless Function Orchestration Systems’. In this paper they created a framework of 7 checkboxes to assess the maturity and quality of serverless architectures that include run-time support for the orchestration of serverless functions. They applied the framework by assessing the offerings of IBM, Microsoft and AWS. In DevOps circles, their serverless function orchestration services are known as “Functions as a Service” (FaaS) platforms.

Recommended for engineers: a descriptive long-read about serverless.

A conversation and explanation about the test framework

I very recently invited the authors of the test framework for an interview, in one of the dozens of developer Slack communities out there. So they could thoroughly pick their words and tell why they included certain checkboxes into their test framework. I chose to let this interview take place in the Cloud Foundry Slack Community, consisting of 10.000+ members, because it’s part of one of the few non-profit organizations in the cloud architecture market.


Bas van Essen:

Welcome Gerard Paris 😀. You’re the first interview respondent in this fresh #live-interviews channel of the Cloud Foundry Slack Community.

To give an introduction, I’ll interview you today about your research paper ‘Comparison of Production Serverless Function Orchestration Systems’.

Gerard París:

“Hi Bas! 👋 I’m here with Álvaro Ruiz and Daniel Barcelona, who conducted the research together with me. One of our other, co-researchers, Pedro García, may join us soon…”

Bas van Essen:

Great. Let’s start! First as an introduction. What lead you guys of investigating/comparing the serverless orchestration offerings from AWS, IBM and Azure?

Gerard París:

“We form a research group on Cloud and Distributed Systems. Among cloud technologies we see serverless as a very hot and promising topic. However, there’s still plenty of room for improvement and research. Specifically, we were interested in the elasticity benefits serverless could provide to data analytics workflows. In this paper we evaluated the major FaaS providers, focusing on orchestration services and function parallelism.”



Bas van Essen:

Some people consider Google as the real introducer of the concept of serverless, instead of those believing serverless is an invention of AWS. But Google’s offerings were not suited for this research, which you carried out in the summer of 2018?

Gerard París:

“Yes, Google was arguably the first company to introduce the notion of simply run your code without the need to provision and manage servers, with App Engine. However, this paper is focused on function orchestration services. For example, AWS offers orchestration of Lambdas with Step Functions, Azure has Durable Functions and IBM has Composer to orchestrate Cloud Functions. We didn’t include Google because it does not offer any kind of orchestration service for their Google Cloud Functions.”

Bas van Essen:

Ok, clear. To already give a glimpse of the results, let me put forward a figure that is included in your research paper.

Source: ‘Comparison of Production Serverless Function Orchestration Systems’ (July 2018)

As you can see here, you compared Amazon’s, IBM’s and Azure’s offerings by assessing them on:

ST-safe, Programming Model, Reflective API, Parallel Execution Support, Software Packaging and Repos, Billing Model; Architecture.

To begin with the first on top, ST-safe, can you elaborate here on what it stands for and why it is important to compare this?

Gerard París:

“ST stands for Serverless Trilemma. It’s a set of desirable rules for serverless function compositions:

  1. a function should be a black-box: the inner code is independent of the orchestration, therefore the functions of a composition can even be written in different languages,
  2. a composition of functions should also behave as a single function, and
  3. double-billing of functions should be avoided (this is what happens when a function synchronously invokes other functions and you end up paying for both).”

Bas van Essen:

Why didn’t Amazon Step Functions meet these needs (at the time of the research)?

Gerard París:

Because an AWS Step Functions state machine cannot be treated as a normal function (therefore it breaks the 2nd rule, known as the substitution principle).


Bas van Essen:

You can read that overall AWS Step Functions came out on top of your test. But how important do you consider the substitution principle?

Gerard París:

In practice, it’s not that important. The major limitation is that AWS Step Functions cannot include another composition inside an orchestration, so it limits code reusability. Another limitation is the cumbersome programming model, compared to their competitors. However, in terms of performance, billing model, maturity of the project, AWS was clearly the most advanced orchestration service.


Bas van Essen:

Before continuing with talking about the assessment of the programming model, I have to ask if you’ve seen a lot has been changed in the offerings of these vendors that would change this test outcome?

Gerard París:

Yes, that’s right. Azure Durable Functions and IBM Composer were experimental projects back in summer’18. IBM Composer is now an Apache Incubator project [<https://github.com/apache/incubator-openwhisk-composer>] and they added support for parallel tasks, which are actually more performant than the other candidates evaluated in the paper. As far as we know, AWS Step Functions has not changed that much.



Bas van Essen:

Would that bring IBM in second or even close to a first position within the test conditions you set up?

Gerard París:

Yes, in terms of overhead of parallel compositions, IBM Composer is now in first position. Most of the other metrics of the evaluation framework would remain the same.


Bas van Essen:

Ok, continuing with assessing the programming model, what makes an function orchestration platform stand out from the others?

Gerard París:

Flexibility and simplicity is what developers look for. Azure Durable Functions provides the most advanced coding abstractions (async/await, eternal orchestrator, singleton addressing, …) but it also requires advanced coding skills. IBM Composer is the easiest to use but is limited to a set of composition constructs. AWS Step Functions programming model is not very user-friendly (complex JSONs) and is also limited in the number of constructs it offers.


Bas van Essen:

Thanks for elaborating directly on the way the three vendors meet these needs. Makes it easy to talk about the checkboxes and assess them. What about your next assessment point, reflective API’s, which desire does it describe?

Gerard París:

It refers to the ability to observe the current state of a function composition: running/failed/completed, which stage it’s on, stop it or pause it, etc. For this checkbox, Azure Durable Functions has the most rich API.


Bas van Essen:

To what kind of serious problems/flaws can it lead when this requirement is poorly met?

Gerard París:

It can lead to not-responding long-running tasks. Reflective APIs are very important for debugging and monitoring. If you launch a composition of several functions, you need means to know the state of the execution and act upon it in real-time through API calls.


Bas van Essen:

And then talking about availability of the so-called Parallel Execution Support, which at the time of the test was not yet met by IBM Composer. What does it exactly embody and why is it important?

Gerard París:

It’s the ability of allowing concurrent execution of functions in the orchestration. It’s an important feature because it enables users to carry out larger tasks more efficiently.

Also, we are interested in processing data analytics workflows which require massive parallelism. In fact, our research group is coordinating an H2020 research project whose aim is to create a Serverless Data Analytics Platform [<http://cloudbutton.eu/>], with industry partners such as IBM Research and Red Hat. We believe that being able to run efficient parallel tasks in workflows is a key feature to enable easier Data Analytics at scale.


Bas van Essen:

Let me introduce your next checkbox by quoting the description in your research paper text:

Software packaging and repositories. This checkbox assesses IBM’s, AWS’s and Microsoft’s function orchestration platforms on the base of ’Modularization and software reuse of serverless applications’.

According to your research, IBM appears to meet this need by offering its OpenWhisk packages to bundle together functions and their triggers. OpenWhisk also permits to publish and search packages in a public namespace in the IBM Cloud.

On the other hand, Amazon’s SAM standard and metadata appears to be more detailed and powerful than those of IBM’s OpenWhisk. In the sense that AWS’ SAM metadata model supports more parameters for serverless applications, such as memory allocations, timeout, resource dependencies, and events and triggers.

Lastly, you conclude that Azure Durable Functions doesn’t contain repos. So which of the vendors comes on top, if you can say so, and why?

Gerard París:

Amazon’s SAM model is the most complete in its offering. However, it does not support Step Functions, thereby disabling composite applications orchestrated with Step Functions. We actually don’t have a clear winner here.



Bas van Essen:

The next one is the billing model. How do you assess that, if you take into consideration that vendors use quite difference pricing models? And maybe you can also elaborate on IBM and Azure, in comparison with the absence of an assessment last July.

Gerard París:

AWS seems more mature and its pricing model is very clear for the user: you pay for state transitions in Step Functions and for function execution time. So, a user can precisely estimate the cost beforehand. At the time of the assessment, Azure Durable Functions was an experimental project and had unpredictable storage costs originated by event sourcing. Finally, IBM just bills you for function execution times, including that of the orchestrating functions.


Bas van Essen:

None of those bill with what you guys call ‘double-billing’, or do they? Can you explain simply what it is and give examples of vendors involved in this?

Gerard París:

Yes, fortunately none of these vendors do ‘double-billing’ in their orchestration services. In fact, the ‘double-billing’ we refer to is not much of a vendor billing detail, but rather a problem of certain serverless function orchestration architectures.

To put it simply, a naive implementation would be to use a main function to synchronously call the orchestrated functions. Then, the vendor would bill you also for the time the main function is blocked waiting for responses from the other functions. An orchestration service should always avoid this undesirable behaviour.



Bas van Essen:

To proceed, the last checkbox is architecture. What were your obtained insights regarding this?

Gerard París:

The architecture point is closely related to ST-safeness that we mentioned earlier. We expect an orchestration service to be event-driven to meet the architecture of the FaaS model. If the orchestration service is build upon the ‘reactive core’ of the FaaS platform, it can take advantage of its intrinsic scalability, whereas an external client scheduler introduces a limitation in elasticity as well as it is not ST-safe. Therefore, a reactive scheduler is preferable than an external synchronous client.


Bas van Essen:

So this wasn’t a checkbox on which AWS the ‘winner’ of this comparison (now IBM would be, as you said), scored the best. But isn’t all regarding scalability a really essential characteristic of serverless function orchestration platforms, in this ever more high-paced and high-demand market?

Gerard París:

We don’t see a clear overall ‘winner’ of this comparison. In the paper, we state that AWS Step Functions was the most mature service and exhibited an overall reliable performance, and this still applies. Now that IBM Composer provides support for parallel tasks, it’s a better option for those who need efficient task concurrency but the other checkboxes remain more or less the same.

The choice of one or another depends on the requirements of each application. Second, I would say that scalability is an essential characteristic of a FaaS platform, in fact AWS Lambda can scale to thousands of concurrent functions. But when we just analyse function orchestration services we find that Step Functions synchronously interacts with Lambda functions from an external scheduler that is not a function in the FaaS platform.


Bas van Essen:

That in itself is not hindering AWS from providing a supreme function orchestration service in terms of scalability?

Gerard París:

The results of the experiments showed that AWS Step Functions appeared to be the most efficient service for both short and long-running orchestrations, but its main drawback I would say is its limited programmability.


Bas van Essen:

Ok, so only from a simplicity/ease of use point of view there is a drawback. Coming back to what you said about the dependence on the user perspective of the function orchestration service, which of course makes completely sense to involve in this, what kind of distinctions would you make? What kind/type of users would you distinguish to determine which user would be better supported by which service?

Gerard París:

Each of the evaluated services has certain limitations. For example, IBM Composer cannot handle long-running workflows (currently, there is a limit of 50 actions in any composition). Amazon Step Functions limits the size of function parameters and results to only 32KB while Azure transparently allows parameters and results of any size. Another difference is that Step Functions allows to interact with other AWS services like ECS, while IBM Composer or Azure Durable Functions only handle Functions.


Bas van Essen:

Allright, just an example: if I’d make a simple distinction, with on the one hand enterprises that may want to use FaaS to only to power a relatively small part of their online products, versus startups that may want to make completely use of FaaS — do you then see test differences? You may like to take vendors out of the scope of IBM, AWS and Microsoft into consideration, if you want.

Gerard París:

That’s a difficult one to answer, and we may deviate from function orchestration services to general characteristics of FaaS platforms, but I also don’t see clear differences in vendors to make such a distinction.



Bas van Essen:

Last question. You were talking about simplicity from a programming model perspective. But recently I also saw this illustration passing by, accompanied with the question ’10 years of AWS architecture: increasing simplicity or increasing complexity’?

Has the complete assortment of the analysed vendors, with all their bundles and prices, also been a potential assessment point in an earlier stage of the research?

Gerard París:

It’s an interesting illustration: architectures are getting complex but offer increased scalability and elasticity at different levels (computing, database, …). In the paper, when we talk about the simplicity of the programming model we are focusing on the orchestration code. Of course it’s also important how these function orchestrations are bundled together with other cloud services, and this could be an idea for further research.

Bas van Essen:

Thank you very much, Gerard! Thank you for your time. I’ll get in touch with you again later.

Gerard París:

Thanks for this interesting interview, Bas! By the way, I forgot to mention we have a repository with the code of the experiments of the paper: https://github.com/gerardparis/faas_orchestration_comparison. Thanks again!


Editor’s note: if you need more insights to be able to grasp what serverless is, I’d recommend the article introduced below:

I hope to you see live during one of the future interviews I’ll conduct in the #live-interview channel of the Cloud Foundry Slack community. Feel free to ask questions yourself too during or after one of those interviews.

Cheers,

Bas van Essen

Bas van Essen

Written by

Programming Insights Provider. Blogger. DevOps, Golang and JavaScript enthusiast. Father. Dutchman.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade