Advanced Design Patterns for Amazon DynamoDB

Part two

9 min readFeb 14, 2019

In Part One we covered some of the basic concepts that you need to know in order to start using DynamoDB. This is Part Two, which will cover some advanced data modeling patterns recommended by AWS at re:Invent 2018 sessions DAT404 and DAT401.

Firstly, here are some of the features released by AWS in 2018, and a quick summary of DAX and Transaction support.

New DynamoDB features in 2018

DAX in memory caching
Continuous backups
Point in time recovery
Encryption at rest
Support for transactions
On-Demand capacity

DAX in memory caching

The DAX service allows an in-memory cache cluster to be provisioned in front of a DynamoDB table. This reduces response times from milliseconds to microseconds.

It should be noted that reads are eventually consistent — please read the developer guide for further detail.

ACID Transaction support

DynamoDB now supports transactions — including across multiple tables. Features:

The new apis to use transactions are:

● TransactWriteItems

● TransactGetItems

With the transaction write API, you can group up to 10 Put, Update, Delete, and ConditionCheck actions and submit them as a single TransactWriteItems operation that either succeeds or fails as a unit. The same is true for multiple Get actions, which you can group and submit as a single TransactGetItems operation.

See here for an example using the new transaction API.

Tenets of DynamoDB data modelling

1. Understand the use case.

- Nature of the data

- Entity relationships

- Concurrency

- Archiving and recovery needs

2. Identify the access patterns

- Read/Write workloads

- Query dimensions and aggregations

3. Data modeling

- Use NoSQL design patterns

- 1:1, 1:n, m:n relationships

- 1 application = 1 table

- Identify primary key

- Partition key and Sort key

- Query dimensions using LSIs and GSIs

4. Review -> Repeat -> Review

Optimizing for large table scans

In most cases, full table scans can be avoided by designing the table Partition keys and GSIs to meet application usage patterns.

Parallel Scans

If a full table scan is required for a large table (2G+), use a parallel scan by specifying the following api parameters to the scan api:

Segment: <integer representing the current segment a worker is requesting>
 TotalSegments: <integer number of segments / worker threads>

It is up to the developer to manage the parallel threads/processes passing in the current segment value as each is started. This is quite straightforward to achieve with an asynchronous biased language such as Node.js

for (var segment = 0; segment < workers; segment++) {// call ddb scan api passing workers and segmentpromises.push(scanTable(segment));}// resolve all resultspromise.all(promises).then(results => {console.log(results);});

Optimizing for high volume writes

In an application usage pattern where a high volume of writes are expected to a low cardinality key value, you may need to augment the key value with a random number to ensure that table items are evenly distributed across partitions.

Sharded writes by GSI suffix

An example of this would be a voting system that needs to keep track of vote counts for a small set of candidates. Without augmenting the partition/index key value, a ‘hotkey’ situation will arise where a small number of partitions are dealing with a large volume of incoming data, and performance will suffer.

This means for example that each Candidate A_n item will keep track of roughly 10% of incoming updates for candidate A, and these can be distributed evenly across storage partitions on the underlying hardware. This is much more efficient for storage and retrieval.

In order to subsequently aggregate the total candidate votes for data using this sharding approach, a periodic operation such as a scheduled Lambda function can query the GSI for the sharded totals, sum them, and write an item back to the table with the aggregate total of votes for the given candidate.

This is another best practice design pattern for DynamoDB — perform calculations and aggregations out of process, and store back for retrieval by the consuming application. This can be done with Lambda.

GSI Overloading

This section goes against the grain of RDBMS data modeling, however It is a recommended best practice pattern for efficient data storage and querying in NoSQL databases such as DynamoDB.

Each global secondary index partition and sort key value does not need to be related to a single set/category of data satisfying a single query type. Instead, generic names can be assigned to these partition and sort keys, and multiple data sets can be stored and queried using the same GSI.

In this example, multiple types of data are stored in the same GSI, and the query is constructed to operate on the GSI based on the type of operation. In this case, a query is made for employees by Hire Date, by using the GSI Partition key value HR-CONFIDENTIAL to target items that contain the expected Data in the sort key — used as GSI query condition.

The same GSI could be used to query for employees with a Order Totals over 5000 by using the GSI Partition Key value QUOTA-2017-Q1, and adding a condition on the Data sort key > 5000.

Note that this involves multiple table items for a given employee, with columns that have different meaning based on query context, which is perfectly acceptable in a denormalized NoSQL table.

Remember here that we are using NoSQL for Internet scale performance, and this implies that we structure data in a way that is optimized accordingly. It is strongly recommended that the supported query types and related GSIs are documented and maintained as a reference point for future developers.

Sparse indexes

Sparse indexes refer to GSIs that have sort keys with values on only a small subset of the overall items in a given table. This means that a query to a GSI that uses this sort key will be much faster as the partition set will be far smaller than for the overall table.

An example of this is a table that contains the state of support cases. As you can see, only the items that contain ASSIGNEE and STATUS_DATE are included in the GSI which limits the scope of the search operation. In addition, to query by status, the STATUS_DATE attribute could be referenced in a query condition such as BEGINS_WITH: COMPLETED_

Refer to this link for query conditions types.

Sparse index details.

Composite sort keys

When designing a data model, consider modeling hierarchies of data via composite sort keys that relate directly to the type of query that the application will require. This basically means concatenating the values that will be used as GSI query parameters, and storing this data in the sort key.

This obviously means a lot of duplicate data, as there will be multiple items that contain the same result attributes, however this is perfectly acceptable in a NoSQL data model, and encouraged as it leads to optimal query performance. Use transactions to update multiple items to keep them in sync if required.

DynamoDB Stream and Lambda to enrich data

In many application use cases, a realtime aggregate result from a set of table items is required. An example may be a running total sales value for the day. In traditional databases this may be handled by a stored proc, or application side processing after retrieving the relevant data set.

With NoSQL, it is best practice to precalculate aggregates values out of band, and store them back into the table as a single item for quick retrieval.

The recommended mechanism for this is to use DynamoDB Streams in conjunction with a Lambda function (or a periodic Lambda).

The lambda function would update the relevant table item to maintain the running total, and potentially notify other services of the change.

There are many data enrichment use cases that would fit this model.

Adjacency Lists

Adjacency lists are a way of modelling many-to-many relationships without excessive data duplication. If we think in terms of a graph model it may help to visualise this.

Adjacency lists are modelled by representing nodes as the Partition key, and using the Sort key to define the edges (or relationship mapping). The remaining attributes in each item relate to the specific node and edge combination.

For example where a Bill and Invoice entity are many-to-many:

Refer to this link here.

Example mapping RDMS model to NoSQL

This is a case study example of how an entity model and known access patterns may be mapped to both relational and NoSQL data models.

Note that it often requires more effort to define an effective NoSQL data model than a relational data model, as more upfront access pattern analysis is required.

Entity Model

Access Patterns

Relational Model Example

Structured relational model suitable for OLAP, without predefined access patterns.

NoSQL Model Example

Efficiently stored data targeted at specific use case access patterns of an application. Ready for internet scale storage and traffic volumes.

Partitioning high capacity tables

When selecting Write Capacity Units and Read Capacity Units, you should select values that are representative of expected load. If you anticipate a spike of high volume traffic, you may be tempted to dial up WCU/RCU to high levels temporarily. You should take care with this, as when a partition is created, it is allocated a share of the WCU/RCU permanently. This is because physical partitions are not reconsolidated when WCU/RCU settings are reduced.

The following calculations give you an idea of capacity units and resulting partition counts.

So for example:

For a 100GB table, with 100RCU and 50WRU then dialled up to 5000RCU and 2000WRU:

● (5000 RCU / 3000) + (2000 WRU / 1000) = 1.67 RCU + 2 WCU = 3.67 partitions for throughput

● (100GB / 10) = 10 partitions for size

So 10 partitions will be used as this is higher than the throughput partitions. This means that if a table is dialed back to 1000 RCU, this would still be distributed across 10 partitions, which is no different to the original distribution

If however, the table is only 1GB in size, the 3.67 would be greater and rounded up to 4 partitions. This means that dialed back to 100 RCU, each partition would only have 25 RCU available. The original RCU was 100, which would have been allocated to one partition, however now an RCU value of 400 would be required to provide the same throughput to across the 4 partitions going forward.

This is generally only a significant issue with smaller table sizes, and/or when setting very high WCU/RCU allocations for temporary actions.

We have covered the basics of DynamoDB, as well as aggregated some new/advanced design patterns distributed by AWS into a summary in order to present a broad overview.

Hopefully you have found this overview helpful, and I encourage you to dive into some of the sources referenced for deeper understanding of each area.

Here’s a link to Advanced Design Patterns for Amazon DynamoDB Part one.

If you’re interested in learning more or thinking about working in technology at NAB, click here.

About the author: Andrew Vaughan is a Senior Manager Distinguished Engineer/Arch at NAB. Prior to this, he’s held senior developer roles at CBA, AMP and MLC.