A Deep Dive into AWS NoSQL Database “DynamoDB”, with simple and brief theoretical view , I like simplicity :) by the below elements:
1- What is DynamoDB briefly ?
2- Core Components and how it does it store data ?
3- Data Consistency
4- Indexing.
5- Pricing Model
6- Streams and Global Tables
7- Security
8- In-Memory Caching — DAX
9- Best Practice considerations
1- What is DynamoDB ?
- Very Fast, Secured , AWS Managed Serverless NoSQL (Non Relational) Database “Key-Value” store.
- Cost Effective “Pay as you go” model Or provisioned with reserved option.
- Fault Tolerance “multi AZs”.
- Multi-Region, Multi-Master Database using Global Tables.
- Backup and restore supports “PITR” (Point-In-Time).
- Supports In-Memory caching using “DAX” (DynamoDB Accelerator) with microsecond latency.
- Full ACID complaint like what we wrote in This Article.
2- Core Components:
- Core Components of DynamoDB are “Table , Item , Attributes”
Tables:
- Like any SQL DB , Is a collection of data which will host the other components hierarchically.
- Tables are independents where there is no `Joins` as in SQL by default.
- A table can have multiple partitions (Will be cleared later here).
- We should have at least one primary key in any item in a table.
- Two types of Primary key Simple as just k-v store or composite k-v and sort key
- Example:
- Simple Key Value:
{
"PersonID": 01, "FirstName": "Aly", "LastName": "Ragab"
}
---- Composite Key Value:
{
"PersonID": 01, "FirstName": "Aly", "LastName": "Ragab", "Address": { "Street": "123", "City": "Dubai"}}
Items:
- Items are the equivalent of Rows in SQL.
- Each table has one or more items, Can have one or more attributes.
Attributes:
- Attribute is the actual data of an item, as key-value store.
How DynamoDB stores tables physically ?
- It uses Partitions (Logical Partitions as Partition Key) and (Physical Partitions).
- Each physical partition is 10GB Volume , Managed internally by DynamoDB.
- One Partition supports `3000` RCUs (Read Capacity Unit)and `1000` WCUs (Write Capacity Unit).
- There is a “Partition Key” of the table primary key, which is considered as a logical partition which will affect the underlying physical partition.
What is the Primary Key ?
- Primary key must be created in table creation, It is a must to identify each item in a table.
- DynamoDB Supports two types of Primary Key (Partition Key or Hash Attribute) and (Partition Key or Sort Key).
1- Partition Key or Hash Attribute :
- A very simple primary key where DynamoDB uses for how it stores the data physically in the storage.
- It is being used internally as an input in a hash function processing and make the output to store it physically in the storage.
- No Duplications are allowed in this case.
2- Partition Key Or Sort Key :
- Composite Primary Key which contains two attributes (partition and sort keys)
- Sort Key known also as “Range Attribute”
- DynamoDB uses the portion of the partition key as an input to internal hash function for processing and then output it to the physical storage.
- Each item will have different Sort Key value which means that we can have duplications in items values.
- It’s possible for multiple items to have the same partition key value. However, those items must have different sort key values.
What about DynamoDB Data Types ?
1- Scalar Type:
- Only one Value as (string, number, boolean, binary, null)
2- Set Type:
- Contains multiple Scalar types as (string set, number set, binary set)
3- Document Type:
- Contains nested attributes as (list and map)
3- DynamoDB Data Consistency:
AWS makes DynamoDB in Fault-Tolerant state, which means that it replicates DynamoDB tables across all AZs in the given Region, So how we can read data from distributed tables ?.
Two types of consistency in DynamoDB:
- Eventual Consistency (Default one).
- Strongly Consistency.
1- Eventual Consistency:
- In this model we get the data from any read instance so fast, because it does not require a check if the data is the latest or no.
- BUT we may get a stale data in this case which means if we have heavy write and read we may read a stale data before the new data is replicated, the probability of having a stale data is not common but it is possible BUT How is that happen ?
In terms of write operation , DynamoDB uses an algorithm called “Paxos” which will always write the latest data to two storage nodes and will allow the third one to sync up a bit late “within milli-seconds”.
In terms of reading operation, DynamoDB will randomly go to any of the three nodes to read from, and in this case it may read from the node which is not synced yet.
and this is the case which we can have a stale data from DynamoDB.
2- Strongly Consistency:
- DynamoDB will read after checking the updated nodes which will not be fast as Eventual Consistency but more reliable.
- It is always checking the network delay, if there is then it will return error (HTTP 500).
4- Indexing:
- To allow fast access to items with its attributes other than the primary key then we can create one or more “Secondary Index” and make the “Scan” or “Query” requests against these indexes.
- Two types of Indexes (LSI — Local Secondary Index) and (GSI — Global Secondary Index)
1- LSI — Local Secondary Index:
- It is like Index in the SQL databases.
- Has the same Partition or Hash key as the primary index of the base table, But with different Sort/Range Key than the primary index of the base table
- Which means that the structure must be combined with Partition key and Sort key
- We can define up to 5 LSI
- For each partition key value, the total size of all indexed items must be 10 GB or less as it should be within the same size of the physical partition.
- When we query the LSI , We can choose either Eventual Consistency or Strong Consistency
- Can only query a single partition by hash key
- It is only created when we create the table, We cannot add a local secondary index to an existing table
- Uses the WCU and RCU of the main table, No Special throttling
2- GSI — Global Secondary Index:
- It is Global because queries on the index can span all of the data in the base table, across all partitions.
- It is stored in its own partition not like the LSI in the same partition.
- The primary key of GSI can be simple or composite (partition key and sort key)
- Queries on GSI support eventual consistency only.
- Can be created / Deleted at any given time.
- If the writes are throttled on the GSI, then the main table will be throttled as well.
So When to use LSI and GSI ? :)
Using LSI :
- When we build an app that needs the same partition key as the table.
- Avoiding additional costs.
- When we need strongly consistent reads.
Using GSI :
- When we build an app that needs the different partition key as the table.
- When we need eventual consistent reads.
5- DynamoDB Pricing Model :
- DynamoDB charges for Reading, Writing, and Storing Data.
- DynamoDB has two Pricing Models (Provisioned Capacity) and (On-Demand Capacity).
1- Provisioned Capacity:
- We pay for the provisioned capacity we choose which is calculated as number of reads/writes per second which affects the Read Capacity Unit “RCU” and Write Capacity Unit “WCU”.
- We have to have a predictable application traffic to DynamoDB.
- We can also use “Reserved Capacity” for getting a discounts as we normally do for EC2s and RDSs.
- Recommended for Production environment
- Auto-Scaling is recommended in case reaching to throttling.
2- On-Demand Capacity (Pay as you Go):
- No need to specify how much read/write
- DynamoDB charges you for the data reads and writes your application performs on your tables.
- No use of reserved capacity here.
- Used Testing Environment or very low heavy app.
What will happen if i have traffic more than i set as a Provisioned capacity ?
This comes from if i have set a specific WCU and RCU and i have kind of spikes or heavy requests suddenly, What will be the thing ?
- DynamoDB has two features which help to fix that (Burst Capacity) and (Adaptive Capacity).
1- Burst Capacity:
- When we provision a capacity large then the normal one, So DynamoDB will make use of that extra amount of the Un-used capacity for future use “In case of spike or throttling” for just 5 minutes.
- Burst can be consumed so quickly so it is not reliable one for heavily long run production traffic.
2- Adaptive Capacity:
- DynamoDB distributes the data across partitions and the throughput capacity is distributed equally across these partitions, However, when data access is imbalanced, a “hot” partition can receive a higher volume of read and write traffic compared to other partitions leading to throttling errors on that partition.
- Adaptive Capacity enables the application to continue reading and writing to hot partitions without being throttled, provided that traffic does not exceed the table’s total provisioned capacity or the partition maximum capacity.
- So it calculates the free WCUs of the other 2 partitions and add it to the hot partition so that it gets away from throttling.
- This is the best thing in Production environment because it is done automatically in real time.
6- Streams and Global Tables:
1- Data Streams:
- A feature which enable DynamoDB to stream data and being read by AWS Lambda,EC2,ES,Kinesis ..etc
- Useful for use cases like (Replication of data , Archival , Log processing or notifications).
2- Global Tables:
- In terms we have to design Multi-Master, Active-Active , Cross-Region replication.
- Use cases (DR purposes , implementing cross regional low latency) and for global applications or geo based applications.
- Streams must be enabled “New and Old” images opt
- Uses Eventual Consistency for Cross-Region read
- Uses Strong consistency for same region read
7- Security:
1- Encryption:
- It uses Encryption at Rest using KMS which is enabled by default.
- It Encrypts Primary Key,Secondary Indexes,Global tables,Backup and DAX Clusters.
- Encryption in Transit using VPC Endpoints within a VPC and TLS endpoints for encrypting data in transit.
2- Identity and Access Management:
- Like any other AWS resource, Authentication and Authorization is controlled by IAM.
- We can create IAM Role to allow certain IAM verb to DynamoDB and/or DAX Cluster
- Can work with Local identities or Federated one.
8- In-Memory Caching — DAX (DynamoDB Accelerator):
- As a best practice design for extreme heavy application requests to DynamoDB is using Caching, It offers DAX.
- Delivers up to 10X faster query performance.
- No code change other than connecting to DAX directly.
- It runs in fault-tolerance mode in multiple AZs.
- DAX Supports the below API calls:
GetItemBatchGetItemQueryScan
- It is just used for Read requests only not write by using ONLY eventual consistency model.
- If the request is “Strongly consistent reads” then DAX will pass through DynamoDB and the data is not cached.
- DAX continuously evaluates your CPU utilization to determine the volume of requests it can process while maintaining a healthy cluster state.
- DAX has two types of caching (Item Cache) and (Query Cache)
1- Item Cache:
- Storing the data as a result of (GetItem and BatchGetItem) with Default TTL 5 min.
- We can specify the TTL while creating the DAX Cluster
- Caching Life cycle is done by removing the older or less popular items
2- Query Cache:
- Stores result of “Query” and “Scan” operations with default TTL 5 min
9- Best Practice considerations:
There are a lot of consideration that should be considered and i will reference to it in the official documentation, The below two points are very important.
1- Avoid having hot keys and hot partition:
- When we have a table which handles so many Read/Write requests comparing with other tables, is called a hot partition which handles more than 3000 RCU and 1000 WCU.
- This is because of a Partition Key that does not distribute I/O requests evenly which can lead to a HOT partition which will result throttling.
- To avoid HOT partition and throttling, We can use “Adaptive Capacity” which works by automatically and instantly increasing throughput capacity for partitions that receive more traffic.
- To also avoid Hot partition by just better choosing of the right Indexing type based on application needs whether LSI or GSI.
- To also avoid Hot partition by just better choosing of the right data consistency “Strong or Eventually” based on application needs
- No for creating Secondary indexes for attributes that are queried frequently and use it for only heavy read attributes.
2- Dealing with Large Item attributes:
- Because DynamoDB limits the size of each item stored in a table so what we can do if we have a large attributes ?
- Storing metadata in DynamoDB and large data or attributes in S3 as we can store large data in S3 as objects and then store the object identifier in DynamoDB item
- We can also use the object metadata support in Amazon S3 to provide a link back to the parent item in DynamoDB.
- So writes on S3 then we can trigger a lambda function which will insert the S3 metadata in item attribute in DynamoDB.
