Understanding the scaling behaviour of DynamoDB OnDemand tables

Yan Cui
Yan Cui
Mar 9 · 7 min read
  1. change the table to OnDemand
  2. apply 40k writes/s traffic to the table right away

From 0 to 4000, no problem!

Let’s assume that a newly created table can handle X amount of read/write throughput before it needs to scale. We can find out the value of X by:

  1. generate a traffic that ramps up to Y writes/s throughput in one minute (using UUIDs as keys to avoid hotspots)
  2. observe the throughput at which writes are throttled

Table handles previously CONSUMED peak & more

From the announcement post by Danilo, one paragraph piqued my curiosity in particular.

The road to 40k writes/s

I ran another experiment to see how quickly a new table can scale to 40,000 writes/s and the steps it took.

The road to 40k writes/s, slowly…

What if the traffic ramps up to 40,000 writes/s steadily instead? I ran a series of experiments to find out.

0 to 40k in 60 mins (1 hour)

In this case, the traffic increases steady over the course of an hour until it reached 40,000 writes/s. As you can see from the results below, the rate of progression is too fast for the OnDemand table to adapt to on the fly. As a result we incurred a lot of throttled write events along the way.

0 to 40k in 90 mins (1:30 hours)

Rerunning the experiment and slowing up the rate of progression shows improved results. But we still experienced a lot of throttling.

0 to 40k in 120 mins (2 hours)

0 to 40k in 150 mins (2:30 hours)

What about costs?

A lot have been made about the fact that OnDemand tables are more expensive per request. A rough estimate suggests it’s around 5–6 times more expensive per request compared to a provisioned table.

Experiments are good!

As a side, all these experiments cost us a total of $271.61 for over 217 million write requests.

  • no need for capacity planning (in most cases)

Summary

In summary:

  • OnDemand tables can handle up to 4000 Consumed Capacity out of the box, after which your operations will be throttled.
  • OnDemand tables would auto-scale based on the Consumed Capacity.
  • Once scaled, you can reach the same throughput again in the future without throttling.
  • You should consider pre-warming OnDemand tables that need to handle spikes of more than 4000 Consumed Capacity to avoid initial throttling.

DAZN Engineering

Revolutionising the sport industry

Yan Cui

Written by

Yan Cui

AWS Serverless Hero. Independent Consultant. Author of https://productionreadyserverless.com. Speaker. Trainer. Blogger.

DAZN Engineering

Revolutionising the sport industry