Securing AWS APIs with an on-premise identity provider in a hybrid cloud

In this blog post we explore how we can use JWT authorizers in Amazon API Gateway to secure APIs on the cloud using a standard token-based authorization scheme provided by an existing identity provider.

This allows existing applications to integrate securely with both on-premise and cloud services as part of a progressive migration to the cloud.

Introduction

Many organizations in hybrid public-private cloud environments often find themselves with an on-premise identity provider. These organizations may either be in the midst of migrating the identity provider to the cloud, or have deliberately chosen to keep their identity provider on premises for regulatory or security reasons. The latter being common in public sector agencies and companies in highly regulated industries such as healthcare and finance.

Problems typically arise when organizations progressively migrate their workloads to the cloud, thus needing existing applications to securely integrate with both on-premise and cloud services at the same time.

Problem Scenario

Let’s see an illustration of such a problem.

How can the client application securely access APIs on the cloud?

Here we assume the on-premise identity provider refers to a service that manages identities and access control using JSON Web Tokens (JWTs) in conformance to OAuth 2.0 or OpenID Connect (OIDC) standards.

In this example, we have an organization that runs an on-premise monolithic application that they are actively refactoring into services to run on the cloud (e.g. as AWS Lambda functions or HTTP API servers on ECS).

This application integrates with the on-premise identity provider to authorize itself with other on-premise services using JWTs signed by the identity provider. However, with a cloud migration under way, this application now also needs to call protected APIs on the cloud, such as APIs served by Amazon API Gateway.

Amazon API Gateway offers several “AWS-native” ways of controlling access to its APIs, such as using the standard AWS IAM roles and policies (see the developer guide for more details). However, this requires us to learn and manage a separate identity and access management (IAM) solution, which itself often leads to a whole host of challenges. For example, having disparate IAM systems makes it difficult to maintain a global view of organization-wide user access. This may lead to lapses in access management which may in turn lead to overexposure of data, potential data breaches and other legal, financial or reputational damages.

Here we explore a solution that allows us to easily use “non-AWS-native” JWTs from our on-premise identity provider with AWS resources. Beyond mitigating the aforementioned problems of managing multiple identity providers, this solution also minimizes impact to existing applications and services as they can continue using the same token-based authorization mechanism for services on AWS.

All of this provides organizations much greater flexibility in their cloud migration journey.

Solution

Our proposed solution uses JWT authorizers to secure APIs on Amazon API Gateway using your organization’s on-premise identity provider. JWT authorizers use JSON Web Tokens (JWTs) to control access to your APIs on AWS, allowing or denying requests based on the validity of the token and optionally the scopes in the token.

JWT authorizers can support any identity provider so long as it issues access tokens that follow OAuth 2.0 and/or OIDC standards. This is what allows us to extend an existing OAuth 2.0 or OIDC authorization scheme used within an enterprise to also authorize services on the cloud.

Diagram of solution

1. The client application requests for a JWT access token from the identity provider. This is typically done using the OAuth 2.0 Client Credentials grant.
2. The client application then calls protected on-premise APIs using the access token.
3. Using the same access token (or another access token created by the same identity provider), the client application calls protected APIs on AWS.
4. The API Gateway uses the JWT authorizer to validate the access token. This involves calling the identity provider’s jwks_uri endpoint for its public key to ensure the authenticity of the access token.
   The standard JWT claims are also validated. This is important in ensuring the token is issued by the expected issuer (iss), is not expired (exp), and is issued for the intended recipient (aud or audience) -- among other claims. See this part of the documentation for the full details.
5. Finally, if the access token is valid, the API Gateway permits access to the upstream API (i.e. a Lambda function in this case).

Demo

Let’s see all this in action!

Prerequisites

• docker and docker-compose
• curl
• jq — command-line JSON processing tool

Setting up an “on-premise” identity provider

To simulate an on-premise identity provider, we use docker-compose to spin up an instance of Keycloak on our local machine. Keycloak is a popular open-source identity provider used in many large enterprises. Here it is used to issue OIDC-compliant access tokens.

We will also use ngrok to expose this local instance of Keycloak to the internet so API Gateway is able to reach it. You will need to sign up for a free account to tunnel https traffic. After signing up, please note your ngrok authtoken.

Copy the following docker-compose.yml file to a directory and replace <AUTHTOKEN> with the authtoken from the ngrok dashboard. Run docker-compose up -d in the same directory to start the services.

version: '3'
volumes:
  keycloak_data:
services:
  ngrok:
    image: ngrok/ngrok:latest
    ports:
      - 4040:4040
    command: http https://keycloak:8443 --authtoken <AUTHTOKEN>
  keycloak:
    image: jboss/keycloak:15.0.1
    depends_on:
      - keycloak-db
    environment:
      DB_VENDOR: POSTGRES
      DB_ADDR: keycloak-db
      DB_PORT: 5432
      DB_DATABASE: keycloak
      DB_USER: keycloak
      DB_PASSWORD: password
      KEYCLOAK_USER: admin
      KEYCLOAK_PASSWORD: password
  keycloak-db:
    image: postgres:9.6
    volumes:
      - keycloak_data:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: keycloak
      POSTGRES_USER: keycloak
      POSTGRES_PASSWORD: password

Next, point your browser to http://localhost:4040 to access the ngrok web UI. Click on the https link to access your local Keycloak instance via your uniquely generated ngrok.io address.

You should see the Keycloak landing page. Click "Administration Console" and log in with admin and password.

Configuring Keycloak

We will create a Keycloak client representing the “client application” in our diagram above. In the Keycloak admin UI, click “Clients”, then “Add Client” and create a new client with client ID hello-world.

Creating a new Keycloak client

Click “Clients” on the sidebar and click into the newly created hello-world client. Under "Access Type", select "confidential" and turn on "Service Accounts Enabled". This ensures we can use the OAuth 2.0 Client Credentials grant to request for an access token later on.

Enter * into the mandatory "Valid Redirect URIs" field and click "Save". Using a wildcard * redirect is not a good practice in production but suffices for a demo.

Configuring the newly created Keycloak client

Next, click the “Mappers” tab. Click “Create” and create a protocol mapper as follows.

Configuring Keycloak client with an audience mapper

This populates the access tokens with an aud claim containing hello-world. We will configure the API Gateway later to check for this.

Creating a Lambda function

Now we create a Lambda function that represents the AWS API that the client application calls.

Head over to the Lambda dashboard. Click “Create function” and select “Author from Scratch”. Name the function hello-world-service and leave everything else as-is. Click "Create function".

Creating a new Lambda function

Creating API Gateway endpoint

Under “Function overview”, click “Add trigger” and select “API Gateway”.

Then select “Create an API”. Select “HTTP API” type and select “Create JWT Authorizer” under the “Security” drop-down.

Under “Identity source”, enter $request.header.Authorization.

Under “Issuer”, enter the Keycloak issuer endpoint which is your ngrok endpoint from before followed by /auth/realms/master, e.g. https://4b08-123-456-78-90.ngrok.io/auth/realms/master.

Lastly, enter hello-world as the audience and click "Add".

Adding an API Gateway trigger

With this we have created an API endpoint with a JWT authorizer. When authorized, requests to that endpoint will then invoke the Lambda function.

Accessing the API Gateway endpoint

Try accessing the generated API Gateway endpoint from the UI. The endpoint URL should look something like https://<uuid>.execute-api.us-east-2.amazonaws.com/default/hello-world-service. You may use your browser, curl or any HTTP client. This represents the "client application" in the solution diagram.

You should get a 401 Unauthorized. This is because we are accessing a protected endpoint without a valid access token.

Let us now request for the access token from Keycloak. We will use curl here but you may also use a GUI tool like Postman.

curl -X POST https://<ngrok.io URL>/auth/realms/master/protocol/openid-connect/token \
-d 'grant_type=client_credentials' \
-d 'client_id=hello-world' \
-d 'client_secret=<retrieve from Keycloak UI>'

Be sure to replace the ngrok.io URL with yours.

You will also need to retrieve the client_secret from the Keycloak UI by going to the "Credentials" tab in the hello-world client and copying the "Secret".

If everything works, you will get as a response a JSON object with the access token. You can copy-paste the access token into a tool like jwt.io to see what the decoded token looks like. The token should contain the various standard claims that the JWT authorizer will validate, including the aud claim which itself should contain hello-world.

You can then pass this token as a Authorization: Bearer <TOKEN> header when calling the same API Gateway endpoint from before, as follows.

# Request for token and store in $TOKEN
TOKEN=$(curl -sX POST https://<ngrok.io URL>/auth/realms/master/protocol/openid-connect/token \
-d 'grant_type=client_credentials' \
-d 'client_id=hello-world' \
-d 'client_secret=<retrieve from Keycloak UI>' | \
jq -r .access_token)
# Call AWS API with $TOKEN
curl https://<uuid>.execute-api.us-east-2.amazonaws.com/default/hello-world-service -H "Authorization: Bearer $TOKEN"

Instead of a 401 Unauthorized, you should now see a "Hello from Lambda!" response! This means you have successfully called a protected API service hosted on AWS using an access token generated by an on-premise identity provider.

Note: If you are still getting a 401 Unauthorized, your token could have expired (default 5 minutes lifespan) between your requesting and using it.

Conclusion

In this blog post we discussed the challenges that arise from managing identities and access in a hybrid cloud environment.

We also explored a solution using JWT authorizers in Amazon API Gateway to leverage on-premise identity providers directly, alleviating organizations of the need to operate multiple identity solutions and minimizing impact on existing applications as organizations migrate progressively to the cloud.