Using Web Socket AWS API Gateway to allow your event driven Lambda based API to push data to your clients.

There are lots of blog posts touting the evolution of Real-Time or Event Driven API Architectures but how do you connect your clients to this type of an API?

Github Repo with Complete Solution
https://github.com/dsandor/example-ws-api-gateway

Note: I simplified this content to be focused on the actual Web Socket API Gateway and Push Response technology portion of the architecture. Subsequent articles will build on this article and integrate it into ReactJS and Vue.js frameworks.

The diagram above shows a Web Socket client submitting a request. Let’s assume this request is one that we would not want the user to wait for a response on. For example, maybe it is a chat type message or a bid for buying a commodity. Let’s walk through this diagram:

(1) Submit Long Running Request

The request is submitted via a web socket and is a simple JSON object like below:

{
"action": "submit-request",
"some-data": "12345abc"
}

(2) Accept Long Running Process

In this step, the JSON Object you put on the web socket above was routed to a lambda. In the AWS Console your WebSocket API looks something like below (I will go more in depth into this configuration later.)

Note that there is a route selection expression, that is basically telling us which property on the request to use to route. In my example I chose to use a property named action and in this route I am handling all messages with an action of submit-request. Those messages will be routed to the lambda defined in the submit-long-running-process.js.

(3) Ack Long Running Process

Our submit-long-running-process lambda in this example is simply sending the request on to an SNS Topic named example-long-running-proc-topic. The purpose of this lambda is to ingest the request into our AWS stack. At this point we can push the message through a series of event driven work processors. To simulate this I subscribed a Lambda to the example-long-running-proc-topic which will push a response to web socket client.

(4) Push Response to Client

The Lambda defined in push-result.js is triggered when a message is sent to the SNS Topic example-long-running-proc-topic. This Lambda is going to push a response to the web socket client.

Create the Lambdas and SNS Topic Infrastructure from the SAM Template

  1. Clone the code in the github repo https://github.com/dsandor/example-ws-api-gateway
  2. Install the dependencies:
    yarn install
  3. Deploy the Lambdas and SNS Topic with the command:
    yarn deploy

After performing the above commands you should have a stack named ws-test-example deployed to your AWS account. Check out the stack in the AWS CloudFormation console, it will look something like this:

Click on the stack name and you can then examine all the resources that our SAM Template “template.yaml” produced for you.

Create the WebSocket API Gateway with the AWS CLI

The following steps will create a WebSocket based API Gateway with the submit-request route mapped to the submit-long-running-process Lambda.

First, we will create the actual API. Note that we are setting the selection expression to route the web socket messages based on the action property of the request data. This is defined at the API level.

aws apigatewayv2 --region us-east-1 create-api --name "cli-created" --protocol-type WEBSOCKET --route-selection-expression '$request.body.action'

The response from CLI will look something like the following. You will need to note the ApiId as it will be used in all the subsequent commands.

{
"ApiEndpoint": "wss://aaaaaaaa.execute-api.us-east-1.amazonaws.com",
"ApiId": "aaaaaaaa",
"ApiKeySelectionExpression": "$request.header.x-api-key",
"CreatedDate": "2019-01-12T15:40:55Z",
"Name": "cli-created",
"ProtocolType": "WEBSOCKET",
"RouteSelectionExpression": "$request.body.action"
}

Now we can create the route for the send-request action.

aws apigatewayv2 --region us-east-1 create-route --api-id aaaaaaaa --no-api-key-required --authorization-type NONE --route-key submit-request

The result of the create route command will look like the following. We also need to take note of the RouteId for later commands.

{
"ApiKeyRequired": false,
"AuthorizationType": "NONE",
"RouteId": "bbbbbb",
"RouteKey": "submit-request"
}

Next we will create an integration in the API. This defines the Lambda that will be used by the submit-request route. This command does not yet link the Lambda Integration to the Route we will do that after this. In the command below you need to replace the api-id and you will need to the ARN for your submit-long-running-process lambda. You can get that by looking at the Lambda in your AWS Console or using the aws cli. Put the full ARN in the command where I put INSERT_THE_ARN_FOR_YOUR_LAMBDA_HERE.

aws apigatewayv2 --region us-east-1 create-integration --api-id aaaaaaaa --integration-type AWS_PROXY --integration-method POST --integration-uri arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/INSERT_THE_ARN_FOR_YOUR_LAMBDA_HERE/invocations

The output will look like the following. You will need to notate the IntegrationId value from this output.

{
"ConnectionType": "INTERNET",
"IntegrationId": "cccccc",
"IntegrationMethod": "POST",
"IntegrationType": "AWS_PROXY",
"IntegrationUri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/ARN_FOR_YOUR_LAMBDA/invocations",
"PassthroughBehavior": "WHEN_NO_MATCH",
"TimeoutInMillis": 29000
}

The final step is to link the Lambda Integration to the Route.

aws apigatewayv2 --region us-east-1 update-route --api-id aaaaaaaa --route-id bbbbbb --target integrations/cccccc --route-key submit-request

The output will look like this:

{
"ApiKeyRequired": false,
"AuthorizationType": "NONE",
"RouteId": "bbbbbb",
"RouteKey": "submit-request",
"Target": "integrations/cccccc"
}

Now you have a Web Socket API Gateway with a route. Check it out in the AWS API Gateway Console.

Let’s deploy our API now to a new stage named v1 with the following commands.

aws apigatewayv2 --region us-east-1 create-deployment --api-id aaaaaa

That will output the following:

{
"CreatedDate": "2019-01-12T17:04:20Z",
"DeploymentId": "dddddd",
"DeploymentStatus": "DEPLOYED"
}

Take note of the DeploymentId we will need it when we create the stage.

aws apigatewayv2 --region us-east-1 create-stage --api-id aaaaaa --stage-name v1 --deployment-id dddddd

You will get the following output when the stage has been created with the new deployment.

{
"CreatedDate": "2019-01-12T17:04:30Z",
"DefaultRouteSettings": {
"DataTraceEnabled": false,
"DetailedMetricsEnabled": false,
"LoggingLevel": "OFF",
"ThrottlingBurstLimit": 5000,
"ThrottlingRateLimit": 10000.0
},
"DeploymentId": "dddddd",
"LastUpdatedDate": "2019-01-12T17:04:30Z",
"RouteSettings": {},
"StageName": "v1",
"StageVariables": {}
}

Clicking on the Stage in the AWS Console will show you an information tab that states the WebSocket URL. Note this URL so we can test our API.

Test out our Web Socket API Gateway

We are going to use the nodejs command line tool wscat. You can install this with the following command:

npm i -g wscat

Now use wscat to connect to your new WebSocket AWS Gateway.

wscat -c wss://aaaaaaa.execute-api.us-east-1.amazonaws.com/v1

You will be connected to your API and placed in a little WebSocket terminal.

connected (press CTRL+C to quit)
>

From here we can send your message to the API and see what response we get back.

connected (press CTRL+C to quit)
>{"action": "submitrequest"}

If everything is installed and setup correctly you will receive a response pushed from your push lambda.

< {"connectionId":"TZ_gNda4UkwCLVQ=","requestId":"ff9764d3-30ee-4115-8b6d-8a6fcdfad036","someOtherData":{"foo":"bar"},"endpoint":"aaaaaa.execute-api.us-east-1.amazonaws.com/v1","dataPushedToSocket":"this property was added in the push lambda"}

How does the push lambda know who to respond to? It’s pretty easy. We simply attach the user’s connectionId and the web socket endpoint URI in downstream request.

const requestThatTakesALongTime = {
connectionId: event.requestContext.connectionId,
requestId: uuid.v4(),
someOtherData: { foo: 'bar' },
endpoint: event.requestContext.domainName + '/' + event.requestContext.stage
};

This allows the push lambda to lookup the user’s socket and push a response to it with the following code.

const apigwManagementApi = new AWS.ApiGatewayManagementApi({
apiVersion: '2018-11-29',
endpoint: message.endpoint
});
await apigwManagementApi.postToConnection({ ConnectionId: message.connectionId, Data: JSON.stringify(message) }).promise();

In the code snippet above we are creating an instance of the ApiGatewayManagementApi with the endpoint that was passed from the submit lambda. We also use the postToConnection function and pass in the ConnectionId and the Data we want to send.

Conclusion

There are a lot of moving parts here but overall the possibilities are amazing. I could eliminate half of this article if I knew how the Web Socket API could be defined in the SAM Template. Let’s hope that we get some better documentation surrounding that from the SAM team soon.

Useful References

AWS SAM Template Basics
AWS SAM Reference
AWS CLI Command Line Reference

Prerequisites

Install NodeJS
Install Yarn
Install AWS CLI