Asynchronous API with DynamoDB Streams
Best of Speed and Resilience
--
Responsiveness is one of the most important parameters for the success of any web application. And asynchronous processing is The Solution for attaining this responsiveness. A server request from the browser should return immediately — without wait for completion. The data flow should be designed in a way that it does not depend upon an immediate response from the server.
There are several architecture patterns to do this. But a major problem with asynchronous processing is error handling. How would the client know if the request failed? We should not lose data in the process. In fact, a fire and forget service, cannot afford to fail. Even if the processing failed for some reason, the data has to reach the DB.
DynamoDB streams provides a cool solution to this problem. Let’s check it out
What is DynamoDB Streams
Most of the traditional databases have a concept of Triggers. These are events generated when some data change in the DB. DynamoDB Streams are quite similar. With one difference — instead of generating a distinct trigger per data change, it generates a stream of events that flows into a target — Lambda or Kinesis.
We can have a Lambda function triggered by such events — which can process this data. The incoming API call can directly dump the data into the DynamoDB — using API Gateway service integration. This ensures very low response time in the API. The DynamoDB streams can be configured to invoke a Lambda function that can
We have an advantage here — the data is already available in our DB before we start processing it. Even if the downstream processing fails, the data is already available in the DB. And in the unlikely event of DB insert/update failing, the API itself will return an error and the client will be able to handle it.
Thus, we have best of both the worlds — low response time as well as error resilience. Let us try to implement this in our AWS account.
Lambda Function
Start with creating a Lambda Function. This is simple. Just go to the Lambda Console, create a new Lambda Function. Call it StreamProcessor. (or any other name you like)…
