Photo by Tobias Fischer on Unsplash

How to use DynamoDB with Apollo GraphQL and NodeJS Serverless offline and in production

Serverless lambda is stateless. It means it does not share any data between two calls of the same function, so now we need to find out how to actually store the data.

When it comes to AWS, we have multiple choices:

• Aurora — a MySQL/PostgreSQL-compatible service
• DocumentDB — a MongoDB-compatible service
• DynamoDB — a special kind of NO-SQL database
• ElastiCache — in-memory storage (based on Redis or Memcache)
• Any other data storage running on EC2

Today we will talk about DynamoDB. Facts you should know about DynamoDB before you proceed:

• It does not support JOIN-s, it is a non-relational database.
• It is a schema-less database, like MongoDB and unlike MySQL.
• It supports storing of deeply-nested objects in DynamoDB table, like in MongoDB.
• It is required to create tables explicitly with DynamoDB, like in MySQL and unlike in MongoDB.
• It does not have a query optimizer. It means, that in order to optimize your quires you will have to tell DynamoDB which index you would like to use in every query, explicitly. This can be a crucial factor against DynamoDB if you are building, for example, a rich query constructor available for users.
• The same problem exists when it is required to sort items in the result.
• Also, the size of the result of searching cannot be more than 1mb. If there is more — the database truncates the list of items and inform about this fact in the result.
• It does not support storage of Date objects, unlike in both MongoDB and MySQL (but you can always store a string representation of a date).
• It does not support integrity checks.
• It scales automatically, you don’t need to worry about provisioning: you just need to choose one of the available scaling strategies.
• You are allowed to create only one database on your account per a region, so if you want to create, let’s say, stage and production “databases”, you will have to come up with a prefix system of some sort when naming your tables.

Still sounds okay for your task? Then let’s go!

Step 1: Starter code

In the previous article, we have created a tiny lambda function and spun up Apollo GraphQL server inside. I am going to re-use the repository we have made. Feel free to clone it or come up with something of your own, it does not have to be Serverless: an Express application will also do.

Step 2: Local installation

To avoid additional money expenses while developing the application, it is a good practice to have a DynamoDB server installed locally. I strongly encourage you to use Docker for this.

So, open ./compose.yml and add there another service with a DynamoDB image:

It is also required to create a ./.db-files folder in order to make the volume work.

There are also other options, like a serverless plugin called serverless-dynamodb-local. It will require to install Java Runtime Environment though.

Step 3: Tables

Photo by chuttersnap on Unsplash

As an example, we will continue building our StarWars application based on swapi.co. We are going to weaponize our heroes because it is dangerous out there in a far-far galaxy where the star war happens.

DynamoDB uses tables in order to store data. So, let’s create one at ./graphql.app/src/db/tables/weapon.js :

DynamoDB is a schema-less database, but still, you need to define a shape of a so-called “partition key”. Yes, DynamoDB uses partitioning to be fast and scalable. For each item, a key value should be unique and random with a good spread, in order to allocate keys between partitions as equally as possible. Most of the time, the partition key is also your primary key.

Read/write capacity units mean how many operations on a table you are going to have per second. On localhost, these values do not matter. In production, there are some advanced settings available to tune up the auto-scaling and provisioning, so these two values can adaptively vary according to the load level. The higher those values the more you pay.

Configuring your provisioning strategy in the right way is important because if the database load breaches the limit defined by the capacity units, the requests will start failing, they will not queue. Automatic upscaling may also take some time, so it is better to add a little bit more of resources.

To be cool, we also create an index file for our table(s) in ./graphql.app/src/db/tables/index.js 🤘:

Step 4: Database class

Good news: we are not required to write API, it is already there: AWS SDK for JavaScript. Just install it:

npm install aws-sdk crypto-random-string

Let’s create a simple wrapper in order to manage our connection to the database.

In the ./graphql.app/src/db/index.js :

Important: make sure that “region” for production is the same as in the serverless.yml file.

I have decided to wrap database methods with promises, in order to make the consumer code slightly better. I hope you don’t mind.

The __DEV__ constant is defined using Webpack DefinePlugin in ./graphql.app/webpack.config.js like this:

Step 5: Exposition through GraphQL

In order to expose our database to the outer world, we will connect it to the GraphQL server.

So, let’s create all necessary files.

Types

For the entity itself in ./graphql.app/src/graphql/types/weapon.graphql :

And also a tiny type for the result in ./graphql.app/src/graphql/types/util.graphql :

Then we update the ./graphql.app/src/graphql/types/index.jsfile in order to plug the types in:

We also patch our characters types file in order to create a mutation.

So, in ./graphql.app/src/graphql/types/character.graphql :

Resolvers

The same goes for the resolvers. We create a new resolver file for the weapon entity:

In ./graphql.app/src/graphql/resolvers/weapon.js

And we patch the characters resolver.
In ./graphql.app/src/graphql/resolvers/character.js

Don’t forget to include all the resolvers in ./graphql.app/src/graphql/resolvers/index.js

Phewww… that was a lot. But the most annoying (I mean interesting) part is ahead: the data source.

Step 6: Database operations and the data source

We are going to create a data source file for our weapon entity: ./graphql.app/src/graphql/dataSources/weapon.js

Before discussing the methods, don’t forget to initialize the data source in our handler ./graphql.app/src/index.js :

How to add/replace an item

We are going to useputItem(). The method is an “insert-replace” operation, and you need to specify the primary key value in advance.

As you may have already noticed, the syntax is quite complex. You should not only specify a value of an attribute, but also its type as a key of an extra object. This looks even weirder if you came here from MongoDB, where you need just pass a data object as it is and that is it.

I bet, the guys from Amazon felt like this too, so they made another class called DocumentClient. It makes things a little bit fancier because the class tries to map JavaScript data types into DynamoDB types automatically. Feel free to explore it on your own.

Meanwhile, the most useful types are:

• S — string {S: "Hello"}
• N — number {N: "3.14"}
• L — a list of values of a different type {L: [{S: "I am a string"}, {N: "9.999"}]}
• M — a sub-object {M: {oneKey: {S: "Agent"}, anotherKey: {N: "0.07"}}}
• BOOL — a boolean (so obvious) {BOOL: true}

There are other specific types, see the documentation. Note, that :

• while put()-ting an item, you may specify any structure you like, and you don’t need to describe a schema,
• the actual attribute values should be always converted to a string.

How to delete an item

The delete operation is quite straight-forward, you just need to specify your partition key value:

How to get an item by id

This operation works pretty the same as the delete operation:

How to update an item

When it comes to updating and searching, things look like kind of SQL :) We need to specify an update expression and fill up placeholders. I guess, by using the placeholders DynamoDB allows us to prevent injections automatically (well, at least I have tried to make an injection and failed:) ).

How to search for an item

For searching there are two methods: scan() and query(). Let’s try the first one. The same expression and placeholders work here.

Whoooa, that was aaaaaalot of code. Now it is time to have some fun.

Step 7: Playing around

Let’s start the application:

docker-compose -f compose.yml up

and go to http://localhost:3000/graphql

Create a red lightsaber.

In the database manager http://localhost:8001/tables/weapon we can see it now:

Now give the sword to Darth Vader.

Now if we call for the entire structure, we can write the query like this:

Darth Vader now has a weapon.

But, naaaah, it is too dangerous to keep this weapon in wrong hands, it is better to have the sword destroyed once and for all:

Step 8: Deployment

The deployment is done as usual with serverless.

Authenticate like this:

npx serverless config credentials --provider aws --key #ACCESS_KEY_ID# --secret #SECRET_ACCESS_KEY#

Then deploy like this:

npx serverless deploy

But with DynamoDB on-board, we need to make two additional things.

Tables

Yes, again, but now in production. A table creation operation is quite heavy by itself, and keep in mind that we pay money for each millisecond while we execute the code inside our lambdas. Therefore, it is better to have our lambdas as fast as possible and create tables in advance.

We have several options here to create our table:

• in the AWS console opened in the browser,
• with an AWS command line tool,
• with a Cloud Formation script, like this one.

For now, let’s choose the first option: go to the DynamoDB in the AWS console (don’t forget to choose a proper region!) and create a new table.

Give permissions

By default, any service does not have access to the other services, the policy is initially set to “deny”. We need to allow our lambda to talk to the database.

Go to CloudFormation → Stacks

Click on the stack name, go to the “Resources” tab and find our lambda permission role.

Click on the “Physical ID”.

Now, for the real-real production, we should have pressed “Edit policy” and add as granular security permission as possible, by narrowing it to “allow several operations on weapons table from a certain account in a certain region”. But this goes really beyond the scope of the article, so we are going just click on “Attach policies” and add a pre-existedAmazonDynamoDBFullAccess policy.

So, click on the checkbox and then on “Attach policy” button.

Conclusion

Wow, that was a long article. Now we have learned how to run Apollo GraphQL server inside a lambda and use DynamoDB as a data storage.

Troubleshooting

Task timed out after N seconds

• Check if your code is valid, so there is no “pending” abandoned promise somewhere around. Otherwise, it is probably not a problem of your lambda.
• Make sure that your DynamoDB region is right (so you have not put eu-north instead of eu-north-1 by mistake).
• If nothing helped, try to increase your Read/Write capacity a little bit, or even switch to the “on-demand” strategy (don’t forget about your billing!).

The specified bucket does not exist

• go to AWS console, enter CloudFormation and delete the deployment,
• OR rename your service in serverless.yml.

Requested resource not found

Make sure that your DynamoDB table is created in the right region (we have used eu-central-1 above) and it is not in the “Creating” state.

Thanks for reading! If the article was helpful for you, don’t hesitate to make some 👏 :)

Extras

• here is the proof-of-concept repository made for the article,
• read the official documentation on how to make indexes,
• consider reading the official best practices with DynamoDB,
• a new “on demand” mode of provisioning,
• full documentation on aws-sdk,
• always mind the prices.

Happy DynamoDB usage experience!

← Previous article: How to use Apollo GraphQL server with Serverless offline and in production