Advanced features of DynamoDB

This Blog has been moved from Medium to blogs.tensult.com. All the latest content will be available there. Subscribe to our newsletter to stay updated.

In this blog I will be explaining the advanced concepts of DynamoDB. If you are not familiar with the service, you can visit my DynamoDB core concepts blog here. I hope this blog will be useful to enhance your knowledge on DynamoDB and which would further add value to your application design processes.

Secondary Indices

What are indices in DynamoDB? DynamoDB provides fast access to items in a table by specifying Partition Key values. But if we need to query by attributes other than keys of the table, then secondary indices are useful to make such queries, secondary indices are two types.

• Local Secondary indices.
• Global Secondary indices.

Why do we need the local secondary index?

If we need to sort items of the table by any attribute other than tables’s sort key, then we can use local secondary index. While creating local secondary index, we can select a new attribute of the item as sort key but Partition Key remains same as the table.

We can only create local secondary indices at the time of table creation and once the table is created these indices becomes unmodifiable.

Why do we need the global secondary index?

We know that to query a DynamoDB table, we need to provide table Partition Key as a parameter but what if we want to query by some other attribute? This is when we need to use the Global secondary indices. Unlike local secondary index, we can select both Partition and sort keys in global secondary index but sort key is optional. So when we create a global secondary index, we need to specify the new Partition key other than the Partition key of the table and sort key can be same or different as the table’s sort key.

When we create a global secondary index, we also need to specify the attributes that will be projected into the index and three different options are available here :

KEYS_ONLY : Each item in the index consists the source table key values and index key values.
INCLUDE : In addition to the attributes described in KEYS_ONLY, the secondary index will include other non-key attributes that you specify.
ALL : The secondary index includes all the attributes from the source table because all of the table data is duplicated in the index.

In reality when you create global secondary key, DynamoDB creates a separate table and replicates all the table operations like insert, update and deletion asynchronously.

We can have upto 5 local and 5 global secondary indices but as local secondary indices have to be created while creating the table so we need to analyse how our query patterns will be before even creating the table.

Pagination

Pagination is a process of splitting results into separate pages, but DynamoDB handles pagination in slightly different way. DynamoDB processes query requests in two phases. First phase is the Query phase and in this phase, DynamoDB fetches records from the table using partition key and optional sort key and the second phase is the Filter phase where DynamoDB applies other attribute filters on the data fetched in the first phase.

DynamoDB provides a pagination parameter called limit and when we pass this parameter in query request, it first fetches items matched with key parameters up to a limit value in query phase and then applies attribute filters in filter phase hence the final items returned by DynamoDB after both phases will be less than or equal to the limit value. In Scan operation, during the query phase, DynamoDB directly fetches items up to the limit value and there is no matching of keys in this operation.

The maximum size of the items fetched in the query phase is limited to 1MB so even if we pass a very large limit value, DynamoDB returns only items that fits into 1MB limit.

Conditions

We can put conditions while creating or updating items. For example: put item, if it doesn’t already exist or update item if version attribute matches to the given value. To know more about conditions go to the AWS official documentation here.

What is hot partition in DynamoDB?

DynamoDB makes hardware partitions based on capacity units of the table. Capacity units of the table are distributed equally among all partitions in DynamoDB. If any partition has consumed more capacity units than provisioned capacity units, then that partition is called as hot partition.
How to avoid hot partitions? The main reason of hot partitions is partition key of the table, when a partition key value is getting more requests then obviously the item of that partition key consumes more capacity units, so selecting proper Partition key is important in DynamoDB to avoid hot partitions. Partition key should be unique and random.

Encryption At Rest

DynamoDB has provided encryption at rest feature for making table data more secure from an unauthorised access. DynamoDB encryption is a server side encryption and it is transparent to the user so we don’t have to change the application’s code. Once encryption is enabled on the table, all the data related to that table will be encrypted : the data in the table, local secondary indices, and global secondary indices.

Encryption will be done using AES-256 encryption. Encryption automatically integrates with AWS Key Management Service for managing the single service default key that is used to encrypt your tables. If a service default key doesn’t exist when you create your encrypted table then DynamoDB automatically creates a new AWS KMS key in your account. This key will be used for the tables that are created in the future.

Encryption at rest can be enabled only when the table is created. We can’t enable it after the table is created, also once encryption at rest is enabled on the table we can’t disable it back again.

TTL

Time To Live ( TTL ) is a feature that enables you to clean up items of the table automatically after a specific time. To enable TTL on the table, first we need to choose the name of the TTL attribute then we can use this TTL attribute to specify a timestamp at which point items should be deleted by DynamoDB from your table and DynamoDB never takes any additional charges for deleted items using TTL.

Note : DynamoDB deletes expired items within 48 hours after expiration, it depends upon the nature of the workload and the size of the table.

Backup

DynamoDB provides automated on-demand backup, restore, and point-in-time recovery. We have already written a blog on DynamoDB backup, and if you would like to read it, please click here.

Access control

You might have valid AWS credentials to access DynamoDB resources(tables, indexes, and streams), but until you have appropriate permissions, you cannot access DynamoDB resources. Click here to know more about AWS IAM policies.

In DynamoDB we can constrain users to certain actions while granting permission using IAM policy like allowing users to read certain items and attributes in a table or a secondary index and allowing users to write on certain attributes in a table, based upon the identity of that user.

We can achieve above scenarios using IAM policy condition element, by adding a condition element to an IAM policy, you can allow or deny access to items and attributes in DynamoDB tables and indexes. We can then apply this policy to IAM users, groups, or roles to restrict certain services or users. To know more about Access control go to an official documentation here.

Limits

Read & Write Capacity Unit Sizes :

One read capacity = One read per second (items size upto 4KB )

One write capacity = One write per second (items size upto 1KB )

Throughput limits : Except US East (N. Virginia)region, rest of the regions have the same limits.

US East (N. Virginia) Region:

• Per table — 40,000 read capacity units and 40,000 write capacity units
• Per account — 80,000 read capacity units and 80,000 write capacity units

All Other Regions:

• Per table — 10,000 read capacity units and 10,000 write capacity units
• Per account — 20,000 read capacity units and 20,000 write capacity units

Note : The throughput limit is at an AWS account level. All the account’s available throughput can be applied to a single table or across multiple tables.

Table Size : There is no limitation on table size in terms of the number of items and the number of bytes. Default limit on the Number of tables for AWS Account is 256/region. It is a soft limit so it can be increased by raising Service Limit Increase support case with AWS.

Item Size : item size should not exceed 400KB

Partition Key Size : All the items with same Partition Key goes to the same partition and the maximum partition size of DynamoDB is 10GB hence we can only store a maximum of 10GB data with a single Partition Key.