AWS API Gateway Integration Treks: SNS
There are a few reasons why an errant surfer would end up here: they were told to integrate a hyper-event-driven architecture or they want to explore the hideous, frightening, yet luring secrets of the notorious AWS service or…a quick “stress dump.” Certainly enough though, the wanderer might be caught by the tags SNS, AWS, API Gateway or have even googled them and reached out where every killer would bury his victim: way deep down in the search results page or even on the rarely visited the second page of search results.
So I’ll make some assumptions on your knowledge with the AWS API Gateway:
- you know what it is;
- you know more or less about its usage, but you want more;
- you already successfully stacked up some API Gateway + Lambda + Something service.
Now you’ll probably have already noticed that the large number of built-in integrations is never large enough and the great event-driven designs in your mind just don’t fit in into them seamlessly.
One of the integrations which I have found that the service is sorely lacking in is with SNS.
When addressing an event-driven architecture design, SNS (Simple Notification Service) to me has always worked as a collector of events to be forwarded. You’ll have multiple paths you can simultaneously fan-out (or filter) your event. These are:
- Amazon Simple Queue Service, when it comes to communicate with other entities, being them other aws services, devices, ec2 instances, etc. this is your choice
- AWS Lambda, a frequent “parallel” pathway to use in combination to SQS when want to fan-out to any micro-service or you have to log/insert data in DBs
- AWS Event Fork Pipelines, to perform work automatically in response to events triggered by a SNS publisher, pre-cooked pipelines of SQSs, Lambda and such;
- Webhook (HTTP/S), to feed preexisting APIs you want to attach to the new one).
You will find that you don’t integrate SNS the way you would with DynamoDB, but…
…No, I won’t create yet another lambda!
Your laziness is not what you think it is and, as a best practice, unless you’re forced to or want to add some logic to the fulfillment of the request, don’t put lambdas behind an API Gateway resource method, here’s why:
Money matters!
You pay API Gateways in a very simple way for REST APIs: more or less one dollar every 300mln requests worldwide (https://aws.amazon.com/api-gateway/pricing/) and every month you get the first million requests for free. Very straightforward. Although you may want to increase the complexity with API Keys, caching, etc., in the end you have a one-dimension cost: you pay per request.
Lambda costs, on the other hand, are three-dimension: requests, duration and memory usage.
Thus, you want to have your cost forecast clean and simple, just as much as you want to squeeze that lousy dollar to the bone.
And latency, too!
If you are an experienced Lambda user, you will know that every function has a warm-up time, a container refresh time and other hidden amenities that introduce jitters in your requests.
You already may have to struggle with time-burdening DynamoDB queries. So do NOT add unwanted, avoidable latencies in your requests!
… SNS is plain pay-per-request too!
Similarly to API Gateway, SNS pricing is one dimensional, too: $0.5 per million request (the first milion is again free) for each 64KB chunk of published data (which is a LOT!).
AWS is preventing us making events coming from “the base” (i.e. the Internet) meet our Notification architecture? Not exactly,
it’s just that it’s not as easy as integrating an API Method with a query on DynamoDB. Let’s look in the details at what you should do to have you REST request to set off a notification
STEP 1: get your resource and method ready
As this topic is a bit advanced, I won’t bother you (and me) explaining how to create a resource and subsequently a method. So let’s just say that I created a /whatever/command resource and associated a POST method to it. The POST method allows passing a body with the request so it may already be the message you want to spread to your notification list.
The integration will pass some parameters and the request body to a subscribed lambda, which will expect a proxy integration behavior (i.e. as if the SNS simply wouldn’t be there and and the request integrations point to a lambda with proxy integration), and then, as a default behavior, the body to everything else is subscribed to the SNS topic.
The figure above shows the final rendition of the method execution. You can see that I also set a query string parameter that the requester shall fill in. Let’s see now the Integration Request.
STEP 2: the Integration request
Following a list of the required configurations to fill in:
- Integration type: AWS Service
- AWS Region: your preferred one
- AWS Service: Simple Notification Service (SNS)
- AWS Subdomain: leave blank
- HTTP method: POST
- Path Override: /
- Execution Role: one that allows the API Gateway publishing to the SNS topic of your choice
- HTTP Header: ‘application/x-www-form-urlencoded’
The HTTP Method and HTTP Header fields tell the experienced AWS Actions API consumer that we are instructing our API Gateway to make a POST Publish request to the SNS service with some URL-encoded parameters.
STEP 3: create your SNS topic
This is the easy part. In you Amazon web console, got to the SNS page, create a new topic, choose your name for the new topic and subscribe to the endpoints of your choice.
STEP 4: the infamous mapping template
well, now this is the most frustrating part, especially if you want to tweak my template. As you may have already started to suspect, yes we will have to create a URL-encoded string (dammit!). This will more or less make what we’re going to put in the mapping template fairly unreadable.
yes it’s filthy! No, I couldn’t have made a huge $util.urlEncode of the whole message, because that method takes only a string as a parameter. Besides, there is no such thing as string formatting in Apache Velocity Template Language (VTL), nor their concatenation! The only good news is that I created a fairly general integration with SNS, so you probably won’t change much of what I will show you.
The things to notice in the picture above:
- the Content Type (for the request) is set to application/json;
- the “##” are a smart tweak to improve reading; in VTL, “##” tells the engine that all the rest of the line is a comment and it will be all discarded, newline character included, otherwise not allowed in a URL encoded string;
Let’s see the URL decoded string first (do not put this into the template!):
#set($allP=$input.params())
#set($body1=$input.body.replaceAll(" ",""))
#set($body2=$body1.replaceAll('\n', ''))
#set($body_lambda=$body2.replaceAll('"', '\\\\\\"'))
#set($body_def=$body2.replaceAll('"', '\\\"'))
Action=Publish
&TopicArn=$util.urlEncode(putyourtopichere)
&MessageStructure=json&MessageAttributes.member.1.Name=SN
&MessageAttributes.member.1.Value.DataType=String
&MessageAttributes.member.1.Value.StringValue=$util.urlEncode($input.params(‘SN’))
&Message={
"lambda":"{\"body\":\"$util.urlEncode($body_lambda)\",\"httpMethod\":\"$util.urlEncode($context.httpMethod)\",\"resource\":\"$util.urlEncode($context.resourcePath)\",\"pathParameters\":{#foreach($typein$allP.path.keySet())\"$type\":\"$allP.path.get($type)\"#if($foreach.hasNext),#end\"queryStringParameters\":{#foreach($typein$allP.querystring.keySet())\"$type\":\"$allP.querystring.get($type)\"#if($foreach.hasNext),#end#end},\"headers\":{#foreach($typein$allP.header.keySet())\"$type\":\"$allP.headers.get($type)\"#if($foreach.hasNext),#end#end}}",
"default":"$util.urlEncode($body_def)"
}The first part is a set of convenient declarations (the #set clauses), meant to prepare the body for the URL encoding.
The second part is still quite understandable, in it are the Action of the POST request to AWS SNS, the topic ARN (put your own topic!), the Message Structure and some Message Attribute, stuck with the message.
The third and last part is in fact a JSON, composed by an object containing two keys, each of those describing how we want the body to look like for the lambda or the rest of the subscribers.
Finally, here is the whole awful URL-encoded string as a whole to be put in the template:
#set($allP=$input.params())
#set($body1=$input.body.replaceAll(" ",""))
#set($body2=$body1.replaceAll('\n', ''))
#set($body_lambda=$body2.replaceAll('"', '\\\\\\"'))
#set($body_def=$body2.replaceAll('"', '\\\"'))
Action=Publish##
&TopicArn=$util.urlEncode('putyourtopichere')##
&MessageStructure=json##
&MessageAttributes.member.1.Name=SN##
&MessageAttributes.member.1.Value.DataType=String##
&MessageAttributes.member.1.Value.StringValue=$util.urlEncode($input.params('SN'))##
&Message=%7B##
%22lambda%22%3A%22%7B##
%5C%22body%5C%22%3A%5C%22$util.urlEncode($body_lambda)%5C%22%2C##
%5C%22httpMethod%5C%22%3A%5C%22$util.urlEncode($context.httpMethod)%5C%22%2C##
%5C%22resource%5C%22%3A%5C%22$util.urlEncode($context.resourcePath)%5C%22%2C##
%5C%22pathParameters%5C%22%3A%7B#foreach($type in $allP.path.keySet())##
%5C%22$type%5C%22%3A%5C%22$allP.path.get($type)%5C%22#if($foreach.hasNext)%2C#end#
#end%7D%2C##
%5C%22queryStringParameters%5C%22%3A%7B#foreach($type in $allP.querystring.keySet())##
%5C%22$type%5C%22%3A%5C%22$allP.querystring.get($type)%5C%22#if($foreach.hasNext)%2C#end##
#end%7D%2C##
%5C%22headers%5C%22%3A%7B#foreach($type in $allP.header.keySet())##
%5C%22$type%5C%22%3A%5C%22$allP.headers.get($type)%5C%22#if($foreach.hasNext)%2C#end##
#end%7D##
%7D%22%2C##
%22default%22%3A%22$util.urlEncode($body_def)%22%7DTesting
To show you how this works I’ll guide you through the testing of the simple integration we have made, a SQS queue has been subscribed to the topic, meaning that every message that will be published in it, will be directed to that queue. I named it after the parameter of the REST method so it’s kind of simple to address the right queue right from it (I’ll explain that in another trek ;) ).
What I’m going to show you now is the whole path as depicted below.
In short order, using the testing interface of the API Gateway console to call the method and start off the process.
The request will flow through the API Gateway, the SNS topic and the SQS queue. There you will receive message: no lambda, just one request, one SNS message and one SQS message.
Conclusions
In this post you have seen in this post a general way to get the AWS API Gateway working with SNS without having a lambda between the two, this means fast, scalable and cheap event creation.
By fast, I mean tens of milliseconds. By scalable, I mean that every service that we use is managed by AWS and declared to elasticly scale-up to your needs. By cheap, I mean that one million of the tests that I showed you here cost less than one dollar. And no maintenance.
What we accomplish here is a sound serverless event dispatch (similar to AWS EventBridge, but less expensive). What will consume those zillions of SNS messages, it’s up to you, sky’s the limit.
