DynamoDB is a fully Managed NoSQL Database Service that provides single-digit millisecond latency for any scale. DynamoDB automatically spreads the data and traffic for your tables over a sufficient number of servers to handle the throughput. All of your data is stored on SSDs and automatically replicated across Multi-AZs (three) within an AWS region.
In order to understand DynamoDB we need to start with data model. In DynamoDB a table is a collection of items and each item is a collection of attributes (name-value pair). DynamoDB table is schema-less. The only limitation is the item size (attribute names + values) needs to be smaller than 400KB in binary and UTF-8. The value can be a scalar, a document (JSON), or a set.
The minimal requirement for a table is the primary key. There are two kinds of primary keys: 1) Partition Key, 2) Partition Key + Sort Key. The partition key is also known as hash attribute because DynamoDB uses an internal hash f(x) to evenly distribute data across partitions based on this hash attribute. So, in order to uniformly distribute data and achieve the full amount of request throughput you provisioned it is highly desirable to choose the partition-key 1) with high cardinality, 2) uniformly requested, 3) and as randomly as possible.
DynamoDB manages partitioning (item distribution) for you automatically, adding new partitions as your table grows in size or RCU/WCU, as partition-total = MAX (partition-tableSize | partition-throughput). The general rule of thumb is the single partition can hold approximately 10GB of data. Be careful though, the throughput is based on partition. In other words, if you have 2 partitions your actual throughput is only half of your provisioned RCU/WCU (RCU/#Partitions = RCU per Partition). The side effect when your data size grows beyond the partition size (1oGB) the less throughput will be provisioned per partition. DynamoDB doesn’t deallocate/Coalesce partitions. In other words, if you have a large table, and suddenly you delete most of the data, the partitions will still be allocated to the table like before the deletion. Understand this behavior will help us design appropriate throughput to begin with. For example, if you create a table with much larger throughput (more partition potentially), then later decrease the throughput. Since the partitions cannot be deallocated your throughput is actually smaller than if you had initially created table with less throughput. Another typical work load is bulk upload, to keep above mentioned principle in mind if you bulk load data too fast (by increase write throughput too much) DynamoDB might create more partitions that will cause lower write capacity per partition later on when you try to decrease the the write throughput.
What is DynamoDB’s consistency model? The are 2 most common consistency models knowns by ACID and BASE. They both has pros and cons. In general, ACID transactions are far more pessimistic. In the NoSQL world, ACID is less fashionable as databases loosen the requirements for immediate consistency in order to gain scalability and resilience. DynamoDB supports both eventually consistency and strongly consistency reads. The latter has double RCU requirement so as to be as twice expensive as eventual consistency reads. This is all driven by your business requirements that some application will tolerate the stale reads. DynamoDB ensures durability. So when you write an item into DynamoDB table the moment you receive an “operation successful” response is the moment DynamoDB tells you that it has persist (durable write) your request. However, it takes time to propagate to all servers. Practically this operation will finish within 1 second. In other words, the data is eventually consistent, meaning there is no read-after-write consistency. However, DynamoDB offers you an option to request the most up-to-date version of the data. By default, the Query operation performs eventually consistent reads, but you can optionally request strongly consistent reads. In the case of network delay or outage the strongly consistent reads might be less available.
Do we have the same concurrency model with RDBMS? The short answer is not. As we mentioned ACID, DynamoDB supports an “atomic counter” feature that you can increment or decrement a value atomically. DynamoDB also supports “conditional write” operations. Generally this will meet all the OCC requirements for an application.
RCU: unit of read capacity has been defined as 1 strongly consistent read per second for an item as large as 4KB. WCU: unit of write capacity represents 1 write per second for an item as large as 1KB. The read operations will be rounded up to the next 4KB. Now, what if your application’s read or write requests exceed the provisioned throughput? Your requests might be throttled. While DynamoDB provides Burst Capacity per-partition up to 5 minutes read/write capacity provided you have unused “burst credit”. However, you should not design your application to be dependent on those burst capacities. Words of caution: DynamoDB can and does use your burst capacity for background maintenance without prior notice. For example, if you have LSI the capacity units consumed by index operation are charged against the table provisioned throughput, things like queries against indexes or write data into indexes whenever you write data into table. You need to provision GSI (online indexing) separately, and consequently the capacity units are consumed from that.
Be careful though, your capacity units will still be consumed for 1) reads on non-existent items, 2) conditional writes failed on conditions. Now you probably are thinking about batch process to save some money if you have smaller size of the items. The result is probably not what you think depend on the operations. For instance, BatchGetItem reads a batch of two items of 1.5kB and 6KB. DynamoDB will first roundup individual items to the next 4KB then sum them together. So, in our case we will consume 4KB + 8KB = 12KB instead of 1.5KB + 6KB = 7.5KB and round up to 8KB RCU. Whereas Query operations just do the opposite. Scan considers the size of items evaluated instead of actually returned. Any data projection operation such as count, subset of attributes return have no impact on above mentioned calculation because DynamoDB has to read each of them. DeleteItem will use the size of the deleted item to calculate.
Last but not least, let’s talk about the index. There are two types of indexes you can create on DynamoDB so far: 1) GSI, 2) LSI.
GSI will asynchronously update index when you do the writes. Now, if GSIs don’t have enough write capacity (remember that GSI needs to be provisioned separately from table for RCU/WCU), table writes will be throttled. The general rule is that the data size in an partition is greater than 10GB, and eventual consistency is okay you can use GSI.
So, how do we design a N:M relationship using DynamoDB? One way to solve it is by using GSI. For example, we have a contact entity and group entity, and contact to group is many-to-many relationship. In other words, one contact can belong to many groups whereas one group can contain multiple contacts. Like in RDBMS we can use an intermediary table called ContactGroup (PartitionKey: UserId, RangeKey: GroupId) that we all familar with. The trick is to use GSI as secondary index with PartitionKey: GroupId and RangeKey: UserId. So, in this way we can user Query API to answer questions like, “give me all the contacts who’s GroupId is xyz by using GSI”, and “give me all the groups of contact A by using ContactGroup’s PartitionKey”
