DynamoDB Insert: Performance Basics in Python/Boto3
AWS DynamoDB is a hugely popular NoSQL store on AWS. It’s being used in production by pretty much everyone, from small startups to large corporates.
In this post, we’ll outline how during the development process of an API service that reads a few 10,000s of data records from an outside source and inserts them into DynamoDB, we went down from a few minutes of execution time (2–4 minutes) to a few seconds (2–8 seconds), using a few basic optimizations that can be applied quickly to the code and infrastructure.
To make a long story short, if you are writing a Lambda function to load data into DynamoDB, you want to make sure to apply the following optimizations:
Optimization #1: Memory (and vCPUs)
Set the Lambda to use the maximum of 10,240 MB (10 GB) of memory. This will also result in 6 vCPUs allocated to enable more concurrency, which is critical for speeding up the application's ability to load faster into the DB.
If you are using the serverless framework, it’s an easy flag right below the provider name:
Optimization #2: Batch Writing
Here is the text-book example from the official AWS docs. Ignore it!!!
You can probably already guess that loading item-by-item is not efficient: given a movies list, there should be a way to insert it in bulk instead of looping. And indeed, DynamoDB has a BatchWriteItem function that one could leverage.
This is how a load_batch function might look like, accepting a table name and a list of items:
But what does `get_item_for_create` (line 6) do? This brings us to the next optimization
Optimization #3: Handling (and Caching) Composite Keys
Unforuntalty, unlike in the text-book example, you would often not be able to just insert a python dictionary to DynamoDB as-is. You would have to take care of a few special values. For example, convert floats to Decimals:
But, more importantly, if the table schema contains composite keys, you would have to inject them into the dictionary. Sure, if you know the composite keys in advance, you might hard-code them as strings. But better engineering would have it that you write a generic function that can generate the composite keys (and values) based on the entity name by working with the DynamoDB table description API.
The following function generically builds the composite key for any DynamoDB entity:
Notice that you have to pull the table description using a dynamo DB client (line 3). This is a client you might create before calling the function and reuse later:
dynamodb_client = boto3.client('dynamodb')But more importantly, you really want to cache the table description somehow (and make sure to purge this cache when the schema changes), in order to prevent the API call to Dynamo on the creation of every single item:
In a simplistic memory cache, it would look like this:
table_composite_keys = dict()And now for the revised get_composite_keys function:
Then, you might use the following function to build the values of the keys so that you can inject them into your dictionary:
Until finally, you end up with the following get_item_for_create function, which takes care of everything:
Optimization #4: Concurrency
While the DynamoDB python client can handle 25 batch-write requests from a single thread, you might be able to improve this by concurrently pushing batch requests from multiple concurrent threads.
Unfortunately, Python on Lambda doesn’t enable most concurrency methods offered by Python. Some of the popular libraries are not implemented, and you cannot open a lot of “real” new threads since the CPUs are quite limited.
For example, these are a few errors you would run into by trying to implement concurrent/parallel code in Python on Lambda:
However, some approaches do seem to work — fortunately, concurrent.futures do seem to generate a performance increment. Given a entities_to_load dictionary that maps the table name to a dictionary with the item’s data and some MAX_PROC number representing the maximum number of threads, this is how you might use the load_batch function introduced in the previous section. Here the load_batch function is part of dal.dynamo module:
Conclusion
As is often the case, starting out with the quickstart examples of doing stuff (in this case, the DynamoDB put-item type of task) turns out to not to work in real-life scenarios, and pretty soon you are going to have to do some extra work to make things tick. Hopefully, this post will help you improve the performance of your application!
