Executing long-running tasks/processes in AWS Lambda

AWS Lambda is an event-driven, serverless computing platform that executes your code in response to events. Serverless support more or less almost all use cases, But one thing that still requires some thinking and finding a way to do is using lambda for executing the long-running tasks.

We have many other options for this purpose but they are not cost-effective plus others require a lot more effort which is sort of expensive in terms of the use case. For example, we want to migrate old data from s3 to some other data source like Elasticsearch or DynamoDB.

For this purpose, if we launch an EC2 instance and run a system or windows service for data migrations that's a lot of resource wastage and not a good engineering practice when we can use serverless which is way more cost-effective in terms of development, effort, and money.

Right now what stops us is AWS Lambda execution timeout which is limited
for this purpose, we have many ways do that one way to achieve this is by executing lambda in a state machine and keeping track of your current state.

Here is the code example in C# where we basically restarted our state machine by keeping track of the remaining time of lambda execution and the most important thing is remembering our current state for the next execution because we don’t want to execute the same logic again and again and stuck in the infinite loop.

Hope you got that and enjoyed the read. let me know in the comment section if you have any concerns.