AWS Lambda caching issues with Global Variables

This Blog has moved from Medium to blogs.tensult.com. All the latest content will be available there. Subscribe to our newsletter to stay updated.

We all love the Serverless revolution. The famous Lambda functions helps us to run applications, websites, stream processors and Whatnot without managing any servers. It is so easy to work with Lambda functions and we at Tensult run quite complex websites on Lambda, built with ~200 functions!!! One fine day we started seeing issues with one of the Lambda functions and we didn’t understand what was going for a couple of hours.

We generate reports for all the categories of the website: one report for each category. If no report is existing for a category then we will initialise the report and then start the generation process. We noticed some category reports started getting items from other categories. Upon deeper investigation we figured out that it was an issue with how we have used global variables inside the Lambda function code.

The very same day one of my friend’s friend also had similar issues and I helped him very quickly as I already know what was the issue so I thought I should write a a blog on this issue.

To take the full advantage of the Lambda functions and get things done correctly, we need to understand how Lambda functions work behind the scenes else we will get into lot of issues and these will be very very hard to debug. The entire function code and all the variables outside of the Lambda Handler should be treated as constant and it shouldn’t have any shared variables in the global scope otherwise they will be modified across requests and cause caching issues. I will explain further with sample code so that it’s easy for you to understand the issue. Here I have chosen NodeJS 8.10 as Runtime but the concept is applicable to any Lambda supported Runtimes.

let userGreeting = "Hello <name>! Have a Nice day!!"
exports.handler = async (event) => {
    console.log("event received", event);
    userGreeting = userGreeting.replace("<name>", event.name);
    console.log(userGreeting);
    return userGreeting;
};

The above code is overly simplified but it suffices to give you context. Here we have defined “userGreeting” as a global variable and replacing it in the handler to generate greeting as per the input parameters.

Run 1: input = { name: ‘Dilip’ }

START RequestId: 57ecb648-949c-11e8-9d95-c14eee1f73e8 Version: $LATEST
2018-07-31T08:33:04.902Z	57ecb648-949c-11e8-9d95-c14eee1f73e8	event received { name: 'Dilip' }
2018-07-31T08:33:04.904Z	57ecb648-949c-11e8-9d95-c14eee1f73e8	Hello Dilip! Have a Nice day!!
END RequestId: 57ecb648-949c-11e8-9d95-c14eee1f73e8
REPORT RequestId: 57ecb648-949c-11e8-9d95-c14eee1f73e8	Duration: 8.92 ms	Billed Duration: 100 ms 	Memory Size: 128 MB	Max Memory Used: 20 MB

Run 2: input = { name: ‘Kola’ }

START RequestId: 689b30e3-949c-11e8-82e8-97480c18d4cf Version: $LATEST
2018-07-31T08:33:32.616Z	689b30e3-949c-11e8-82e8-97480c18d4cf	event received { name: 'Kola' }
2018-07-31T08:33:32.617Z	689b30e3-949c-11e8-82e8-97480c18d4cf	Hello Dilip! Have a Nice day!!
END RequestId: 689b30e3-949c-11e8-82e8-97480c18d4cf
REPORT RequestId: 689b30e3-949c-11e8-82e8-97480c18d4cf	Duration: 0.47 ms	Billed Duration: 100 ms 	Memory Size: 128 MB	Max Memory Used: 20 MB

Please make a note of the output, it is same even though the input has changed, this is because “userGreeting” is modified after the Run 1 and it will be remain cached with modifications till the function container is reinitialised and that happens only when function is updated.

How to fix this?

We shouldn’t use any modifiable variables in global scope so we need to convert them to a function scope by wrapping a function around them. Now there is no global variable which getting modified in the request scope hence there won’t be variable caching issues.

function getUserGreeting() {
    return "Hello <name>! Have a Nice day!!";
}
exports.handler = async (event) => {
    console.log("event received", event);
    let userGreeting = getUserGreeting();
    userGreeting = userGreeting.replace("<name>", event.name);
    console.log(userGreeting);
    return userGreeting;
};

Run 1: input = { name: ‘Dilip’ }

START RequestId: 6ca5f8b0-949e-11e8-bf7b-dff3b8b6145b Version: $LATEST
2018-07-31T08:47:58.391Z	6ca5f8b0-949e-11e8-bf7b-dff3b8b6145b	event received { name: 'Dilip' }
2018-07-31T08:47:58.391Z	6ca5f8b0-949e-11e8-bf7b-dff3b8b6145b	Hello Dilip! Have a Nice day!!
END RequestId: 6ca5f8b0-949e-11e8-bf7b-dff3b8b6145b
REPORT RequestId: 6ca5f8b0-949e-11e8-bf7b-dff3b8b6145b	Duration: 9.69 ms	Billed Duration: 100 ms 	Memory Size: 128 MB	Max Memory Used: 20 MB

Run 2: input = { name: ‘Kola’ }

START RequestId: 8e4f1188-949e-11e8-abc1-058894d830c6 Version: $LATEST
2018-07-31T08:48:54.867Z	8e4f1188-949e-11e8-abc1-058894d830c6	event received { name: 'Kola' }
2018-07-31T08:48:54.867Z	8e4f1188-949e-11e8-abc1-058894d830c6	Hello Kola! Have a Nice day!!
END RequestId: 8e4f1188-949e-11e8-abc1-058894d830c6
REPORT RequestId: 8e4f1188-949e-11e8-abc1-058894d830c6	Duration: 3.49 ms	Billed Duration: 100 ms 	Memory Size: 128 MB	Max Memory Used: 20 MB

Please see now output is correct as per the input given.

Conclusion

Now you have understood why you shouldn’t use global variables which are modifiable in request scope while writing the Lambda functions. We also have seen how can we fix this by wrapping a function around a modifiable global variable. I hope you will never face such issues but just in case you face similar issues then now you know what to do.