AWS API Gateway with Custom Authentication

This week we built a cloud hosted microservice based on the serverless framework utilizing the AWS API Gateway, Lambda Functions, SQS and DynamoDB. It was an interesting experience because it was the first time I used all those components in conjunction.

As it always is, when you try something new, we experienced some throwbacks and were stuck on issues while getting everything to work together. One issue that we were stuck on for quite a while was a very weird behavior of the API Gateway.

An authorizer function can access any AWS resource to validate the authentcation

We implemented a custom authorizer function in lambda which had an implementation to validate our custom auth tokens that are sent via HTTP headers:

X-AgentID: 3284734
X-Auth-Token: 84H3K5j41k$ad4jf-49x

Custom authorizer functions are a great feature of the API Gateway. Every HTTP request that is sent to an endpoint is first validated against a Lambda function for authorization and then forwarded to the target function.

Flow of a GET request with API Gateway and an authorizer function

With the serverless framework it is quite easy to setup an authorizer function. In fact our whole API Gateway configuration is generated by serverless, similar to this example snippet from

Now we had 5 lambda functions set up to use our authorizer function and it worked really well until we realized that we had a strange issue. A call to one service authenticated corretly:

curl -H X-AgentID=3284734 -H X-Auth-Token: 84H3K5j41k$ad4jf-49x
 → 200

But a call to a second URL returned an “Access Denied”:

curl -H X-AgentID=3284734 -H X-Auth-Token: 84H3K5j41k$ad4jf-49x
 → 401

Now we changed the order of the calls an got the exact opposite. The call to metadata worked, but tasks returned a 401.

curl -H X-AgentID=3284734 -H X-Auth-Token: 84H3K5j41k$ad4jf-49x
→ 200
curl -H X-AgentID=3284734 -H X-Auth-Token: 84H3K5j41k$ad4jf-49x
→ 401

This got us puzzled for quite some time but after we finally fully understood the implementation of the authorizer function it all made sense. First let’s have look how the authorizer function for the API gateway is defined (very simplified version!):

module.exports.auth = (event, context, callback) => {
  // validate event.authorizationToken
  // determine principalId, i.e. authorized user
  return callback(null, 
generatePolicy(principalId, ‘Allow’, event.methodArn));

The function generatePolicy basically only packages the data in a JSON document. For a complete example of an authorizer handler with JWTs you can have a look here:

As you can see above the authorizer generates a response for a given methodArn which is a concrete HTTP method in the API Gateway, e.g.


That is perfectly ok for the first method — we have an authorization for the tasks method. We noticed that the authorizer was not called any more for the second (metadata) method and just the 401 was returned. As it turns out the authorizer response is cached.

The caching is done based on the auth header (X-Authorization by default) and therefore not called for subsequent method calls with the same header any more.

We had two options:

  • Disable caching? This worked: The authorizer was called for each method call and created a new policy for each HTTP method ARN. In the long disabling the caching was a not solution we wanted to use.
  • Create a policy in the authorizer func that contains the concrete ARNs of all the API methods that a user a can access. The policy can contain a list of ARNs where access is allowed.
  • Change the autorizer fu nction to return a policy which spans all the HTTP methods that a user can call. We opted to change a method ARN that was passed to the authorizer function and just replaced the path with a wildcard.

That was the solution! Now the authorizer is only called once per user and the policy document tell the API gateway to allow all subsequent calls to our API function based on that authorization.

All in all the setup with serverless, API Gateway and Lambda worked really well after we had correctly set up the authorization. Next time it might be a good idea to integrate AWS Cognito instead of using the custom authentication method.