Add a Serverless front-end to your ElastiCache Redis cluster

I have spent the last 18 month learning and using Redis. Redis is an awesome technology and it is made easier to use in the cloud with Amazon ElastiCache for Redis. I use Redis as more than just a cache, even though it is a great choice when you need to accelerate access to resources like files or database records. I use it as a resilient extension of my program memory, it includes implementation of common data structures like lists, sets and sorted sets. Because of its extreme performance and data type support, Redis is great for scenarios such as stream analysis and atomic increments. It is very easy to use Redis when you need to quickly aggregate data or deduplicate incoming records. But enough about Redis, by leveraging Serverless technologies like AWS Lambda and AppSync we can create applications that use Redis cheaply and quickly. In this blog post I will go through a practical example of Serverless technology together with Amazon ElastiCache for Redis.

The problem

I have been working on some projects recently where I used Redis (backed by DynamoDB) as an object store and a simple aggregation mechanism for processing realtime data streams. During the process I had to tackle a couple of challenges:

• I needed Public access to my Redis cluster over HTTP. Redis is using a TCP based protocol to communicate with clients and ElastiCache is configured by default inside an AWS VPC. With that in mind, if I wanted to access my Redis cluster from a mobile app or a browser I would have had to configure public access to the Redis port from my VPC. Also, by using HTTP in my client, I save myself from having to install and maintain local Redis libraries.
• I wanted to use Redis commands to access data instead of trying to create new abstractions on top of a perfectly good syntax.
• I wanted the option to authorize the clients that connect to my Redis clusters from an untrusted source (e.g. the Internet).

It also goes without saying that as a Cloud developer I expect to pay as little as possible and spend the least amount of time on setting up and installing my application.

The solution

As you can see in the image, the solution ended up pretty simple. Let’s look at the pieces:

• AWS AppSync — AppSync makes it easy to expose data sources as GraphQL endpoints, it also supports several authorization methods. The AppSync team recently announced Direct Lambda Resolvers reducing the number of steps it takes to map Lambda Functions to AppSync endpoints.
• AWS Lambda — AWS Lambda needs no introduction. It is a Serverless Functions service that manages all the aspects of running code in the cloud.
• Amazon ElastiCahe — Amazon ElastiCache is fully managed in-memory datastore that supports the Memcached and Redis open source engines.
• Honorable mention: AWS Serverless Application Model (SAM) — I’ve spent some quality time with SAM recently automating my deployment scripts and after a bit of a learning-curve I was able create a repeatable deployment process that helped me build and test my code.

How it works

To expose Redis commands through AppSync we will need a schema (you will deploy it as part of your SAM template later on).

We are using a Mutation to send an array of commands (technically an array of arrays) to the Lambda Function where they parsed and executed inside a Redis pipeline.

We are using a Query to read data from Redis (in pratice we are not testing whether the Redis commands passed to the Query are mutative or not).

Next, we map the Mutation and Query to our Lambda function by creating a Resolver for our Mutation and Query. Note that the resolvers are identical with the exception of the type of call to the Lambda function.

Putting it all together

This section will walk you through downloading, installing and testing this solution.

Requirements
* Git installed
* AWS CLI installed and configured with Administrator permission.
* SAM CLI — Install the SAM CLI
* Python 3.8.x is not required to run the sample but will be needed to make changes in the Lambda function

Downloading and installing
Clone this git repo locally and go to the project directory

$ git clone https://github.com/nirmash/appsync-redis-api.git
Cloning into 'appsync-redis-api'...
remote: Enumerating objects: 39, done.
remote: Counting objects: 100% (39/39), done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 39 (delta 19), reused 37 (delta 17), pack-reused 0
Unpacking objects: 100% (39/39), done.
$ cd appsync-redis-api/

Build the SAM deployment package

$ sam build
Building function 'RedisExecuteCommand'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
Build Succeeded
Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml
Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

Deploy the application.
Note: The SAM template will create a new AppSync API, a Lambda Function and an Amazon ElastiCache cluster.

$ sam deploy --guided

When prompted, select a stack name (the name of the CloudFormation stack that you will use to delete the application later). Your preferred AWS Region and the name for the AppSync API to be created.
Note: You will need the AppSync API name later to obtain the API ID.

Configuring SAM deploy
======================
   Looking for samconfig.toml :  Not found
   Setting default arguments for 'sam deploy'
   =========================================
   Stack Name [sam-app]: redisAppSyncApi
   AWS Region [us-east-1]: us-west-2
   Parameter RedisGraphQLApiName [GraphQL-Api]: RedisAppSyncApi

Follow the rest of the prompts. SAM will create a CloudFormation template and deploy the application components, this will take a few minutes.

... Deployment status will show here ...
CloudFormation outputs from deployed stack
--------------------------------------------------------------------Outputs
--------------------------------------------------------------------Key                 RedisExecuteCommand
Description         -
Value               arn:aws:lambda:us-west-2:..:redisAppSyncApiRedis
Key                 RedisQueryAPI
Description         -
Value               arn:aws:appsync:us-west-2:apis/xxxxxxxxxxxxx
--------------------------------------------------------------------

Validating and testing
Once the deployment is completed you will see the below output. We will then obtain the API Key for the App Sync API and execute some Redis commands against it.
To obtain the API Key you will need the API Id. You will do that by calling the aws cli.

$ aws appsync list-graphql-apis

Copy the apiId from the command output. To obtain the API Key, call another aws cli command. Also, copy the value of the GRAPHQL uri endpoint for later use.

$ aws appsync list-api-keys --api-id <your api Id>

Copy the id value from the command output. You now have what you need to test your new API. The image below shows the steps in a terminal window.

To make testing easier (and because it’s cool…) this repo includes a simple HTML test client that attempts to emulate the redis-cli experience. To get it to work, just double-click the index.html file in the root directory of the repo. Paste the GraphQL uri endpoint and API key you saved earlier into the HTML form and hit the connect button, once connected, `>>` will appear in the cli text area. You can now use some Redis commands to communicate with ElastiCache for Redis!

Cleaning up
You can use the AWS cloud formation CLI to remove the cloud components you deployed with SAM.

aws cloudformation delete-stack --stack-name <Stack Name you entered earlier>

Note: This call is asynchronous so don’t be alarmed when it completes immediately. You can check the AWS Console CloudFormation screen for the stack deletion status.

Summary
By adding AppSync and Lambda functions to ElastiCache for Redis, we can make it easy and cheap to connect clients to Redis directly and greatly simplify setup and ongoing maintenance.