Implementing Idempotency in Serverless Architectures

Dr. Werner Vogels, Amazon’s CTO, reflected on the lessons learned from 10 years on AWS, saying

Expect the unexpected.

He adds that it’s important to build systems that embrace failure as a natural occurrence. Needless to say, microservices must also be designed with this “breakability” in mind.

Failures can occur intermittently as API requests are exchanged between the server and the client. Let’s say a client sends a request to a server and something happens to the server, such as a network issue, that prevents it from returning a response to the client. The client doesn’t know what happened to the server until it sends another request.

However, reattempting the request multiple times is not a viable solution either. If a client ends up making double payments due to a missing payment approval message, it will not only lead to customer complaints but also result in significant business repercussions. So, how do you fix it?

Idempotency

The client can make multiple requests, and the server will validate them as usual. On the server side, you can implement logic to handle duplicate requests, ensuring that the same event is executed effectively once. The concept of idempotency revolves around guaranteeing a ‘safe retry’ mechanism within a distributed service.

The most common approach is to utilize an idempotency key, which includes a unique value for each client request and verifies whether the same request has been submitted previously.

The idempotency record indicates whether the previous request was successfully completed and stores the status as either ‘Completed’ or ‘In Progress’ in the database. Additionally, it is not feasible to validate duplicate requests indefinitely, so it is necessary to set an expiration time to ensure they are processed within a certain timeframe. The process of validating duplicate requests is as follows, ensuring that the client receives the same response without the need for duplicate payment processing.

1. Compare the idempotency key and payload for the same hash value.
2. Set the idempotency record for the initial request to ‘In-Process’.
3. If the operation on the initial request is successful, change the record to ‘Complete’.
   3–1. Return a response to the client when the operation on the initial request is complete.
   3–2. Retry the request if the record has expired, and change the record to ‘Complete’ after retrying the operation.
   3–3. If the record has not expired and a ‘Complete’ record is found, return the same response.
4. If the operation fails on the initial request (due to timeout or other reasons), maintain the record as ‘In Progress’.
   4–1. On a retry, compare the idempotency key and the hash value of the payload.
   4–2. Retry the operation because it is a request with the same payload and is in ‘In-Progress’ status.
   4–3. When the retry is complete, change the record to ‘Complete’.
   4–4. Return the response to the client.

Idempotency with AWS Lambda Powertools

Serverless architectures, such as event-driven services, consist of many API requests between services. According to the official documentation for AWS Lambda, Lambda is fault-tolerant, meaning that if you call a function and it throws an error, the Lambda service retries your function. While this retry is not critical for all workloads using Lambda, implementing idempotency for safe retry is essential for workloads that really only need to work effectively once, such as order payment, inventory deduction, and so on.

Fortunately, AWS provides idempotency utilities through AWS Lambda Powertools, an open-source developer library that abstracts the implementation of idempotency and makes it easy to convert it into an idempotent job. It is available for Python, Java, TypeScript, and .NET, but this post will focus on Python. Each element in the idempotency implementation layer corresponds to the following.

• Client -> Client
• Server -> AWS Lambda
• Database -> Persistent Layer (DynamoDB)

The idempotency record indicates whether the previous request was successfully processed and is stored in the Persistent Layer, DynamoDB, as a Key-Value pair, as shown below.

• id: Idempotency key to validate if the same request has been entered before
• data: Stores the response value returned by the previous request in JSON (to return the same response)
• expiration: Expiration date for records in COMPLETE status
• in_progress_expiration: Timeout period in INPROGRESS status
• status: Display the record status of INPROGRESS, COMPLETE, EXPIRED
• validation: Hash value of the payload

Using the @idempotentdecorator provided by AWS Lambda Powertools, you can easily convert your existing code to an idempotency job as shown below. The full code is available on GitHub.

import uuid
from aws_lambda_powertools.utilities.idempotency import DynamoDBPersistenceLayer, idempotent, IdempotencyConfig

persistent_layer = DynamoDBPersistenceLayer(table_name="persistent-storage-lambda-idempotent")
config = IdempotencyConfig(
    event_key_jmespath="powertools_json(body)",
    expires_after_seconds=5*60
)

@idempotent(persistence_store=persistent_layer, config=config)
def lambda_handler(event: dict, context: LambdaContext) -> dict:
    amount = int(json.loads(event['body'])['amount'])
    ddb_client.put_item(
        TableName='test-duplication-table',
        Item={
            'user_id': {
                'S': str(uuid.uuid4())
            },
            'amount': {
                'S': str(amount)
            }
        }
    )
    return {}

To indicate the storage location of the idempotency record, specify the name of a DynamoDB table in the persistent_layer. Currently, we only support DynamoDB for storing the idempotency record. If you want to specify Redis or another database, you’ll need to customize the idempotency logic.

Since POST operations inherently modify the state of the database, they necessitate the implementation of idempotency. You can specify the idempotency key using event_key_jmespath. Taking the payment scenario as an example, the body value of the event is specified as the idempotency key to verify if the same user has made the same payment request for the same item. Through this configuration, you can ensure secure re-execution without any adverse effects, such as double payment, caused by errors during function execution.

Idempotency Test

The following shell script calls the Amazon API Gateway URL associated with the Lambda function. A total of 7 requests are made, and the last 3 requests are duplicate calls with the same body value.

curl -X POST https://apigw-url -d '{"amount": "20000", "user_id":"1"}' -H "x-api-key: ${apikey}" -H 'Content-Type: application/json'
curl -X POST https://apigw-url -d '{"amount": "10000", "user_id":"2"}' -H "x-api-key: ${apikey}" -H 'Content-Type: application/json'
curl -X POST https://apigw-url -d '{"amount": "30000", "user_id":"3"}' -H "x-api-key: ${apikey}" -H 'Content-Type: application/json'
curl -X POST https://apigw-url -d '{"amount": "40000", "user_id":"4"}' -H "x-api-key: ${apikey}" -H 'Content-Type: application/json'
curl -X POST https://apigw-url -d '{"amount": "50000", "user_id":"5"}' -H "x-api-key: ${apikey}" -H 'Content-Type: application/json'
curl -X POST https://apigw-url -d '{"amount": "50000", "user_id":"5"}' -H "x-api-key: ${apikey}" -H 'Content-Type: application/json'
curl -X POST https://apigw-url -d '{"amount": "50000", "user_id":"5"}' -H "x-api-key: ${apikey}" -H 'Content-Type: application/json'

First, let’s comment out the lines containing @idempotent to see the result of the code without idempotent enforcement.

Item of DynamoDB is duplicated

All seven requests were stored in DynamoDB, even though they included duplicate requests. If this had been an idempotent operation, only 5 requests would have been stored in DynamoDB without the duplicate records.

On the contrary, if you uncomment @idempotent again to to enable idempotency in the code and test the same request using a shell script, you will observe that the duplicate request is not executed due to the implemented idempotency validation logic, as demonstrated below.

Item of DynamoDB table is not duplicated

Conclusion

Every revenue-generating service faces the risk of disrupting business continuity, making it crucial for every production-ready API to implement idempotency, particularly for critical workloads.

AWS Lambda Powertools can greatly reduce the development and operational effort needed to implement idempotency. By leveraging the features offered by Lambda Powertools, you can streamline idempotency implementation, ensuring the reliability of your service, even in the presence of failures in both client-server and serverless environments.