Passing data between Lambdas with AWS Step Functions
This is a small tutorial to demonstrate the passing of parameters between Lambda functions, using the new AWS Step Functions console to define a State Machine.
First, I would recommend following the AWS tutorial “A Lambda state machine”. This will do a great job to get started with a hello-world State Machine. I found that while it’s a great tutorial, I couldn’t find much coverage of some of the essentials for those of us new to Lambda, especially passing of data.
Here I would like to focus on the passing of parameters between two hello-world Lambda functions. Until now it was common to pass data either by SNS, or from within a function, with “invoke”. The state machine introduces a new method of passing data, with lots of bonus goodies like wait states, retries, parallel and choice branches.
In order to do this, we will start with two basic Lambda functions, pass some text to the first, have it altered, then pass it to the second, have it altered again, and then output the final text from the whole state machine.
State Machines are not available in all regions, at the time of writing: US East (Northern Virginia), US East (Ohio), US West (Oregon), EU (Ireland), and Asia Pacific (Tokyo) Regions so choose one of these for the tutorial.
It goes without saying this should be carried out in a sandbox account, as there are security considerations with IAM and VPC selections.
Create two Lambda functions
In the AWS console, find the Lamba screen and select “Create a Lambda function”.
Select “Blank Function” as a blueprint, and select Next on the trigger screen.
Give a Name of “Step1Test”, keep Node.js 4.3.
Overwrite all of the inline text with the following:
exports.handler = (event, context, callback) => {
// Take a var and pass it to step 2
var comment = event.Comment ;
comment = {
"Comment": comment + " .. and digested by step 1." ,
}
callback(null, comment);
};Select a role as you did for the AWS tutorial to allow basic execution, and take the other defaults and choose Next. Choose Create Function.
Create the second function
Go back to the Lambda functions again and select “Create a Lambda Function” again. Follow the same procedure as before until you get to the Configure screen. This time, give a name of “Step2Test”, keeping node.js 4.3.
This time paste in the following inline code, overwriting what is there.
exports.handler = (event, context, callback) => {
// Take the data from step 1 and modify, send to standard output
var comment = event.Comment ;
comment = {
"Comment": comment + " .. and digested by step 2." ,
}
callback(null, comment);
};Follow through as with the first until you have your two Lambda step functions.
How Lambda parameters work in node.js
It’s worth throwing in a brief summary of how parameters work. Lambda accepts — in default mode — parameters come in the form of a JSON object passed in with a handler. In our default state machine, the Execution config will pass in a “Comment” variable, like this:
{
"Comment": "Insert your JSON here"
}It could be any variable; but as it’s already there in the State Machine default execution we’ll use it, that is why our Lambdas we just created are expecting the Comment variable, with an associated string value. In this part of our Step1Test Lambda script :
exports.handler = (event, context, callback) => {It tells the handler to take the input and load it into “event”, and expect “callback” to get some sort of output.
Then we query “event” to get the input Comment value, and put it into a local “comment” variable for manipulation:
var comment = event.Comment ;After manipulation we pass the “comment” string to the “callback” the handler is expecting:
callback(null, comment);Creating the State Machine
In the console, go to the “Step Functions” section (under Application Services). Choose “Create a State Machine”.
On the next screen, give it a name of “StepTest”, then scroll down to the Code box. Paste in the following:
{
"Comment": "My test to prove passing parameters between Lambdas",
"StartAt": "Step1Test",
"States": {
"Step1Test": {
"Type": "Task",
"Resource": "",
"Next": "Step2Test"
},
"Step2Test": {
"Type": "Task",
"Resource": "",
"End": true
}
}
}Once you have pasted in the code, you will need to add the ARN (resource locators) for your two lambda functions. If you put your cursor between the two double-quotes after the Resource heading, you should be offered a list of functions. Select Step1Test for the the first resource, and Step2Test for the second.
Optionally, you could copy the ARN from the top of your Lambda function screen and just paste it in.
When done it should look something like this:
{
"Comment": "My test to prove passing parameters between Lambdas",
"StartAt": "Step1Test",
"States": {
"Step1Test": {
"Type": "Task",
"Resource": "arn:aws:lambda:your-region:your-account:function:Step1Test",
"Next": "Step2Test"
},
"Step2Test": {
"Type": "Task",
"Resource": "arn:aws:lambda:your-region:your-account:function:Step2Test",
"End": true
}
}
}
If your syntax is good, you should be able to scroll up to the Preview button and click the refresh icon, and have it draw a basic picture of how it thinks your step machine will work.
Note: the console does not allow editing after creation, so if the code is not right once created, the only option is to create a new State Machine.
Scroll to the bottom and select “Create State Machine”.
You will get a pop-up box asking about execution Role. In the “Select an IAM role for your tasks” drop-down list, select the role “StatesExecutionRole-region” that has been automatically created for you. Then OK to continue.
All going well you will have a success message. Next is the execution, we can define various executions with differing parameters. Select the offered “New execution” button. We get a new execution screen for out StepTest machine.
Executing the State Machine
For Name add “StepTestExecution”. As you might recall we wrote our Lambdas to expect the Comment variable which is there as a default. All we need to do is change the “Insert your JSON here” text between the quotes to anything we like, such as “Starter text” so we have something like:
{
"Comment": "Starter text"
}Scroll down and select “Start execution”. If all goes well we will see an execution page for our machine, with a couple of green boxes, something like this.
Our step machine has worked, so lets take a look at the data that has been passed through. In the “Execution Details”, we will see the Input and Output for the whole State Machine. Choose the Input tab. We should see the input text:
Choose the Output tab. We can see the output of the whole machine.
A roaring success, as we see out starter text has the string append from our first Lambda function, and our second.
As an extra check we can look at each step in turn. In the “Graph” box, select the green “Step2Test” function with a single click.
Note the box on the right, the focus shifts to “Step Details”. In the Step details, choose the input tab. You will see the input to Step 2.
Of course, the output tab will show the same string with our extra text appended.
That’s it! It is version 1.0 of the Step Functions, and with luck we will see some sort on inline editing become available, it seems at the moment the only options are to create or delete.
Note that Step Functions are not free, but they are very cheap including (at the time of writing) up to 4000 transitions free followed by 2.5c per thousand.
I also hope to see some improvement from AWS in the area of code deployment for both State Machines and Lambda functions themselves, even with now four code management tools in the AWS toolbox, none of them seem quite right for elegantly and simply pushing out dev-state versions.
AWS Step Functions — a welcome addition to AWS.
Recommended viewing: Tim Bray talks about Step Functions at the AWS Re:invent confernce 2016.
