Job Scheduling in AWS Using DDB Streams

Keep calm and let AWS do the scheduling for you

Shilpi Gupta
Mar 25 · 3 min read
Photo by Curtis MacNewton on Unsplash.

This article is about using AWS services to periodically schedule your jobs. Some of the use cases where such a type of scheduling is required include sending timely notifications to users, checking the status of a task, etc.

Different AWS services, like DynamoDB Streams, cloud watch events, and SQS, can be used to implement job scheduling in AWS. This article focuses on using DynamoDB TTL and Streams for scheduling jobs.


High-Level Approach

Photo made with draw.io.
  1. The scheduler will put the items to be scheduled in a DynamoDB. The item will be stored with the TTL value equal to the time after which it needs to be executed.

Error Handling With DynamoDB Streams and Executer Lambda

DynamoDB Streams will resend records until they are successfully processed by the lambda (i.e. no exception is thrown by the executer lambda) or they exceed the maximum age configured on the event source mapping. Until then, lambda will not execute the next events in the stream.

If there is a situation where you need to process the message separately due to an error, you can use the dead-letter queue (with Amazon SQS) to push the message and continue with the remaining items in the stream.

Pros of this solution

  1. Using Dynamo TTL allows for setting dynamic values for the scheduling interval. The scheduling interval can be configured per item.

Limitations of this solution

  1. According to the official documentation of DynamoDB TTL, an item can take up to 48 hours to actually delete after the TTL expires. The actual time depends on the nature of the workload and the size of the table. The DynamoDB Stream DELETE event will be triggered only after the item is actually deleted. Thus this solution is not recommended when the scheduling interval is too slow (like one minute, five minutes, etc.) or when the event scheduling interval needs to be precise.

Conclusion

Thanks for reading. I hope you liked the article!

Better Programming

Advice for programmers.

Thanks to Zack Shapiro

Shilpi Gupta

Written by

A software developer who’s just curious to learn and share.

Better Programming

Advice for programmers.

More From Medium

More from Better Programming

More from Better Programming

More from Better Programming

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade