Chaining Serverless Functions for Stateful Workflows: AWS Step Functions using Adapter Pattern

Zeynel Özdemir
Apr 5 · 7 min read
Image source

I have been experimenting with serverless architecture recently, specifically with AWS Lambda. I like having reusable, independent modules in my projects and AWS Lambda is a perfect place to deploy them. Before long, I found myself looking for a good way to coordinate workflows of these independent Lambda functions. Luckily, AWS already had a service for that, which is really easy to integrate with Lambda: Step Functions.

I want to talk about one design pattern that I have been using for various projects and successfully implemented with Step Functions for connecting incompatible functions together. Although I will focus on AWS platform, you can use this design on any other service as well.

For me, the best way to understand a concept is understanding the underlying reason why we needed it in the first place, so I want to start from the beginning with a simple example.

The problem

Imagine that we have built a website where users can register, take private notes, save them and list them. We already deployed our REST API to AWS Lambda, using API Gateway to expose them. We have everything nice and integrated.

One day, we decide to add this fancy EXPORT button.

What happens when a user clicks on it?

We export all of their private notes, translate them into a chosen language and then send the translations as email to user.

So we start by writing a new Lambda function for new endpoint :

exportNotes handler

A sample POST request body to would be:

{ "translateTo": "french" }

Do not think about the details of getNotes, translate, sendMail functions in the code snippet above. Assume that they just magically works. Let’s focus on the general structure.What are the main steps we are taking in function?

  1. Fetch user notes from database.
  2. Translate notes to the target language.
  3. Send translated notes to email of user.
  4. Return a response.

It feels like a lot of work to do in a single lambda function, doesn’t it? Imagine that translation takes 10 seconds, sending email takes 5 seconds. User won’t see any response for 15 seconds after clicking the button, which is not good for user experience. Instead, we can return the response immediately and execute these tasks in background or somewhere else. But the response time is not the only problem here, as you soon will discover.

Okay, one step at a time. We are already working on a serverless architecture, so let’s take advantage of it. We can start by dividing the workload of to new lambda functions.

Breaking a fat lambda function down to smaller lambda functions

As i mentioned at the beginning of this article: I like smaller, independent modules. Translating text and sending email tasks are perfect candidates for such modules. They have well-defined jobs and can be shared by any part of the code (imagine that we have another endpoint that sends email or translates a text).

It is time to create new lambda functions: and .

lambda function:

translate handler

lambda function:

sendMail handler

“But why are we creating new lambda functions for sending email and translating? We can share their code between lambdas and call them asynchronously any time we want.” you might say.

Well, you can certainly do that but then you have to write and maintain error handling logic for each function in the code. Embedding this code in every lambda function will create hard dependencies between them eventually and it can become difficult to maintain all the connections as you start to change things. What if you change something in function? You have to redeploy all of your lambda functions that are embedding it. Besides, we want to use existing AWS services/integrations on discrete function level (monitoring execution time/resources used, counting number of emails sent, logging, defining policies for email/translation separately for security reasons etc.), which is really tricky if we don’t make them separate Lambda functions.

What makes these two Lambda functions beautiful is that, they are not aware of each other, nor any other Lambda function. In fact, they do not care about the business logic of the function that is calling them. They expect an input, do their job in their isolated environment and return an output. Simple as that. You can even use them across multiple projects, since they have an explicit interface to communicate.

Connecting the pieces

By now, we have three Lambda functions:

  • function translates given text and returns the translated text in response.
  • sends an email to given address with given message and returns a success response.
  • is exposed as an API endpoint. Client makes a request and translated notes are sent to user email soon.

Okay, we moved email sending and translating services to out of function and now we can invoke them from anywhere we want. But did we solve the original problem here? still takes 15 seconds to respond. What we need is to return the response immediately and continue executing other tasks sequentially.

So what is the workflow here:

— — → — — →

There are more than one way to coordinate this workflow. We can invoke lambdas asynchronously, use SNS, SQS, implement our own error handling, retry logic in the code…

Or, we can simply use AWS Step Functions.

AWS Step Functions provides serverless orchestration for modern applications. Orchestration centrally manages a workflow by breaking it into multiple steps, adding flow logic, and tracking the inputs and outputs between the steps.

In a nutshell, Step Functions allows us to define our workflows as state machines. We will just start this state machine and Step Functions will take care of the rest.

I want to leave lambda outside of the Step Functions definition, since it is the entry point to the API request and knows about the business logic. I want independent pieces to be in this state machine, so that we can reuse it for other workflows as well.

translate and sendMail state machine

We can define this state machine in Amazon States Language:

You can create Step Functions using the JSON definition above. I created one and named it .

Now we can invoke in , and pass all user notes as input. What happens then?

  • Step Functions starts by invoking Lambda function with given input and waits for it to finish.
  • Then, it invokes Lambda function with the output of previous task (output of ) and waits for it to finish.

Can you see the problem here?

We are giving the output of function as input to . They have their own interface and do not understand each other.

This is the output of function:

This is the input format sendMail function accepts:

Missing piece: Adapters

We have discrete Lambda functions that performs one work. That is good but we still need a way to connect them together in our workflow. We don’t want to change their input/output structure and make them compatible with each other. This is not scalable and creates dependencies between them. We want to make them work together, without modifying their source code.

This is where adapters come into play. They are regular Lambda functions which prepares the input for next task, using the data in the flow. We can add an adapter between these two incompatible states.

translate and sendMail state machine with adapter

Here is the new Step Functions definition for the state machine above:

Notice the Lambda between and . It is basically a bridge between them. It prepares input for by using the output of .We also introduced InputPath and ResultPath to the definition, to give the data a nice structure.

We have 4 Lambda functions: and one Step Function definition .

Let’s write the final versions of the Lambda functions, and won’t change.

  • : Fetch all the notes of user, prepare input data and invoke , return the response without waiting for Step Functions response.
exportNotes handler

Lambda function:

translate handler

Lambda function:

sendMail handler

Lambda function:

That’s it!

We can trace the data in to understand how independent pieces works together:

Now we can easily add more Lambda functions to this workflow using adapters. But remember, every adapter you use means one more lambda function invocation, which can be unnecessary overhead if you overuse them. You can have less adapters in a workflow by taking the unrelated lambda services to first steps and then using cumulative adapters.

KI labs Engineering

KI labs Technical Blog

Zeynel Özdemir

Written by

Computer Science Engineer

KI labs Engineering

KI labs Technical Blog

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade