Serverless WebSocket application on AWS using Terraform

When it comes to full-duplex client-server communication, WebSockets is a widely recognized protocol that has been around for while now. With support from a wide range of browsers and a bunch of readily available solutions, it has become a popular choice for developers. However, implementing a distributed WebSocket server in an application architecture involves various challenges, such as maintaining connections with multiple clients, scaling horizontally across multiple backends, and handling shared state, stale connections, and session stickiness.

This article presents an approach to building a web application with full-duplex client-server communication using a serverless AWS infrastructure. Its components are implemented using Terraform, providing an efficient deployment process that can be executed with a single command. The complete implementation is available for public use and can be accessed on GitHub at https://github.com/adrian-mezei/awsome-chat. And here comes a proposed solution that can serve as a valuable resource for those seeking to explore the possibilities of a serverless WebSocket AWS infrastructure using Terraform.

Architecture overview

• Backend
  A minimal NodeJs application that persists the identifiers of new connections to a DynamoDB table, sends incoming messages to every connected client, and deletes the connection identifiers from the database when a connection gets closed.
• Frontend
  A minimal JavaScript website that opens a WebSocket connection and makes it possible to send and receive messages through it.
• Infrastructure
  The resources in AWS that are required for the application defined using Terraform. This includes the frontend in an S3 bucket configured for static website hosting, API Gateway for maintaining WebSocket connections, and a Lambda Function for handling requests and persisting connections to a DynamoDB table.

Let’s have the following folder structure for the project:

.
├── backend/
├── frontend/
└── terraform/

Backend

The backend Lambda Function persists the WebSocket connection identifier into a DynamoDB table when a new client connects and deletes the identifier when the client disconnects. It also sends messages to connected clients using the postToConnection function of the ApiGatewayManagementApi. This is the crucial part of the solution. API Gateway maintains the connection, and the stateless backend can use connection identifiers to send HTTP messages to the clients through the management API of API Gateway. The frame of the handler function looks like this:

The complete implementation can be found in the backend/src/index.ts file of the GitHub repository. The function itself requires two environment variables to be set:

• TABLE_NAME: the name of the DynamoDB table, where the connection identifiers are persisted
• APIGW_ENDPOINT: the URL of the API Gateway where WebSocket messages can be sent

Both of these environment variables are assigned from Terraform during deployment.

Frontend

This simple HTML, JavaScript and CSS frontend serves the sole purpose of visualizing the communication through the WebSocket connection.

AWSome chat application in two browser windows

The URL of the WebSocket server (the API Gateway in this case) is configured during deployment through a Terraform template file, where deploy-time generated variables are substituted. Then all these static files are uploaded into an S3 bucket which is configured for static website hosting.

The complete implementation of this frontend can be found in the frontend folder of the GitHub repository.

Terraform

1. General variables
2. DynamoDB table
3. API Gateway API
4. CloudWatch Log Group
5. Lambda Function IAM Role
6. Lambda Function
7. S3 bucket
8. Terraform versions and outputs

AWS infrastructure of the AWSome chat application

1. General variables

This includes setting some commonly used data blocks like region and account along with some constant values like DynamoDB table and Lambda Function name. Also, defined a Terraform AWS provider provider, which in this case will use the configured default AWS CLI profile.

2. DynamoDB table

Set up the DynamoDB table for storing the connection identifier (a string) of the WebSocket sessions.

3. API Gateway

Let’s create the API Gateway API with WEBSOCKET protocol type, and an AWS_PROXY integration, so that incoming requests are forwarded to the Lambda Function. In this case, for simplicity, the same Lambda Function is used for all API Gateway routes. Then create the three routes defined by AWS:

• $connect: to initiate a persistent connection between the client and a WebSocket API
• $disconnect: to disconnect from the WebSocket API
• $default: for sending messages (this route is called by default when no other matching route is found)

To make the API callable, a deployment and a stage must also be created. A stage is a named reference to a deployment, which is a snapshot of the API.

4. CloudWatch Log Group

The Lambda Function requires CloudWatch Log Group where it can deliver its console logs.

5. Lambda Function IAM Role

The Lambda Function must have the proper IAM role, following least privilege principle, to perform the following actions:

• Create CloudWatch Log Streams and write log events to it
• Read and Write the DynamoDB Table
• Invoke API Gateway to send WebSocket messages to clients

6. Lambda Function

Now everything is ready for the Lambda Function to be created. A Terraform depends_on clause must also be added to have the CloudWatch Log Group and the IAM Role created before the Lambda Function. Furthermore, the API Gateway must be granted permission to invoke this function.

7. S3 bucket

Finally, the content of the website is stored in an S3 bucket, which is configured for public website hosting. The following resources are needed for this:

• The S3 bucket itself
• Public website configuration
• Static files uploaded (the config.js file is populated from a template with the endpoint of the previously created API Gateway)
• Bucket policy to allow public read access

8. Terraform versions and outputs

It is very important to pin down the version of Terraform and the used providers to avoid unintentional version updates that might break the code.

Finally, let’s define some output values like the S3 bucket URL, where the deployed website will be accessible.

Deployment

To have the whole application put together, the frontend, backend and Terraform resources should be place in their respective folders. This is important, since the static frontend files and the compiled backend file are referenced from the Terraform resources.

Also, AWS CLI and the Terraform AWS provider must be configured properly to be able to perform the deployment. When everything looks fine, the terraform apply command can be issued from the terraform folder.

Upon successful deployment, the bucket_url output parameter shows the URL of the deployed application. Let’s open the URL from multiple browser tabs and send some messages.

Destruction

It is always a good idea to clean up the AWS resources that are not used anymore. Since every single resource has been deployed by Terraform, this can be done by a single command: terraform destroy.

Final thought

While building a horizontally scalable serverless WebSocket solution would be quite cumbersome, API Gateway solves the most difficult part of the task: the connection management. Also, having an infrastructure deployed via Terraform makes it very easy to version control, duplicate and test the solution.

If you are interested in how we at SnapSoft build complex AWS infrastructures and modern, scalable, and serverless applications with top technologies like Terraform, get in touch — we’d love to talk to you. Also, if you are interested in serverless topics, join our AWS Serverless Budapest AWS User Group and visit our regular meetups.

About the author

Adrián Mezei is a Cloud Architect at SnapSoft Ltd., where he leverages his extensive experience in designing AWS architectures. He holds an AWS Solutions Architect Professional certificate and has been actively involved in this field for several years. As a leader of the AWS Serverless Budapest AWS User Group, Adrián consistently organizes meetups and events to educate and inform others about the latest advancements in AWS technologies. He has also been instrumental in leading the AWS infrastructure development of various international projects and has made significant contributions to SnapSoft Ltd.’s AWS Partnership. Adrián’s areas of expertise include infrastructure, serverless computing, networking, and databases.