First lesson: don’t leak your secrets in a medium post illustration

The best way to store secrets in your app is not to store secrets in your app

Disclaimer: the method described in this article relies heavily on AWS. That being said, the concept might still apply to other providers…

If you’ve ever built an app on the Internet you came across the struggle of keeping secrets (database credentials, api keys, etc) on your server.

Historically, developers have found several ways to achieve this. Each of them bearing a certain level of security and convenience. I’ll walk you through them so you can get a sense of where we are coming from.

Storing secrets in your code

# settings/instance_settings.py
STRIPE_API_KEY = '3ef8-843a-49dc-a34d' # Don't tell anyone! plz?

This is obviously the most convenient method. You write your API key in your code as a constant, you then push it to your source control, and voilà!

But, it’s also the least safe method. You just allowed an important secret to be spread on every developer’s computer (or external drives), on your source control provider’s storage, on your CI provider’s server, etc. This highly increases the risk that this secret could leak.

Storing secrets in the environment

stripe_api_key = os.environ["STRIPE_API_KEY"]

This is one I’ve used quite often. You’ll either need to manually set those environment variables on each server or use some kind of orchestrator to handle that for you, which is slightly less convenient than the previous method. Despite this, it has the benefit that only some trusted individuals have access to it.

There are still some problems: a simple misconfiguration (such as running a production server in debug mode) or a security bug could result in the leak of all the environment variables.

Storing secrets in the database

I’ll briefly address this one: you will still need to have your database credentials outside somewhere therefore defeating the purpose of putting your secrets there. Besides, having daily backups of all your services’ API keys also increase the risk of a leak at some point.

Using a secrets syncing service

There are SaaS that offer to take care of your API keys and other secrets for you. They keep them safe and let you query them from their service as needed. We’re still facing the same issue as the previous method: you’ll need a very secret API key in order to get all your API keys.

Storing secrets in your code …but encrypted

A modern version of the first method is to encrypt the secrets in your code, thus not exposing their values to your source control, other developers, and so on. However, to decrypt those secrets, the server still needs to manage a key. This comes back to the issue of distributing and maintaining one great secret that unlocks all the other secrets.

Why is it unsafe?

These methods all share the same characteristic; they still involve having secrets on your server and those could leak or get stolen

Worst case scenario

Imagine the following scenario. An attacker broke into your server using a vulnerability in Apache. He/she then starts searching for database credentials and API keys in your configuration file, your code, the environment. If your secrets are somehow hidden/encrypted enough, the attacker won’t find anything. But that doesn’t mean he/she will stop there. Using something like LiME, the hacker could still browse in the server’s memory dump to extract those secrets.

We can do better

Introducing:

Credentials-less service access using an IAM authenticated API Gateway as a proxy™

(I’m terrible at finding catchy names and this is not really trademarked)

Overview

In a nutshell, we’re using the instance role of our EC2 instance to make a secure call to an API Gateway, which then proxies it to the desired service, after having altered the request by adding the required credentials. Ew! That’s a lot of informations for a single sentence.

They say a picture is worth a 1000 words. CHALLENGE ACCEPTED

Still lost? Let’s dissect this

First, for this to work we need to use the native way for EC2 instances (or ECS Containers and Lambdas) to make calls to other AWS resources, which is by using IAM roles.

Second, we need to create a Rest API resource in API Gateway. There are some specific configurations that need to be applied so the body, headers and query parameters can be passed through. Additionally, we need to alter the request so we can inject the targeted service credentials. Finally, we have to activate the secret weapon: IAM based authorization.

The third and last step consist on signing the URL before issuing call to the desired proxied services.

¿Que?

For those of you not too familiar with the technologies I’ve just mentioned and still unsure of what this is all about, here’s a comparison of before and after using Credentials-less service access using an IAM authenticated API Gateway as a proxy™:

I’ll be using the well-known GitHub API and Postman for this demonstration:

Before (Direct call to GitHub):

After (Call proxied via API Gateway):

I can already hear you say:

You told us this was credentials-less and I see an access key, a secret key and a token !!!

This is because my laptop doesn’t have the secret weapon. Once you get this up and running on an EC2 instance, these temporary security credentials will be automatically retrieved from the instance metadata. If the concept of temporary credentials is new to you, here’s an excerpt from the documentation (emphasis mine):

An application on the instance […] is granted the permissions for the actions and resources that you've defined for the role through the security credentials associated with the role. These security credentials are temporary and we rotate them automatically. We make new credentials available at least five minutes prior to the expiration of the old credentials.

So, to be fair, a more exact description for this method would be Service access using an IAM authenticated API Gateway as a proxy without the use of long-term credentials. However, we can agree that “short-term temporary credentials used to sign calls to proxied services limited via IAM” are way less attractive to attackers than ordinary API keys and other long-term secrets.

This is a huge upgrade from a security perspective

Notable security improvements:

• Temporary credentials never have to be handled by developers
• They are queried programmatically so there’s no need to have them written into a configuration file
• If leaked or stolen, these temporary credentials will work for at most an hour
• All requests can be logged in CloudWatch for auditing
• You can restrict the targeted API using an IAM policy

The last bullet-point is especially useful when working with an API like GitHub that lacks a finer-grained permission model. Using the following IAM policy, I can restrict my API to only allow the GET method on the /repos/PokaInc/test-github-api/issues endpoint.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "execute-api:Invoke"
            ],
            "Resource": [
                "arn:aws:execute-api:us-east-1:123456789010:w974f1rs6e/dev/GET/repos/PokaInc/test-github-api/issues"
            ],
            "Effect": "Allow"
        }
    ]
}

But wait, there’s more !

Other benefits of using API Gateway as a proxy

• Usage statistics. This can be useful to identify trends and prevent rate-limiting from third-party services.
• Logging. When enabled, you can have detailed logs of each requests including its origin and parameters used.
• Caching. Having latency or rate-limiting issues? With three lines of CloudFormation code you can add a caching backend to your proxy.
• Local development. Developers that use an IAM user locally can access the proxied services directly instead of relying to hard-coded credentials on their computer.

At this point I’ll assume that you’re sold on the idea and you want to try out the method for yourself.

The code

Considering here at Poka we’re CloudFormation afi·cio·nados, I’ve created a proof-of-concept template available on GitHub.

Provisioning

To create the proxy, follow the instructions in README.md.

Using the proxy

This is also described in the readme, but I’ll embed it here so I can convince you it’s not black magic.

Let’s review this block of code:

return AWSRequestsAuth(
      [...]
      **boto_utils.get_credentials()    
)

This is where we would usually have seen hard-coded credentials, access to a configuration file or a query to the environment. Instead, this method asks the metadata service for temporary keys. If the EC2 instance role have sufficient permissions, the call will go through to GitHub, otherwise, API Gateway will reject it.

Conclusion

I hope this post helped raise awareness of poorly secured secrets in application servers and how using API Gateway as a proxy can benefit you beyond the added security.

Caveats (and their workarounds)

• What if I want to proxy an internal service not exposed to the internet?

The explanation could have its own dedicated blog post. But long story short, you can provision a VPC Enabled AWS Lambda as the target of your API Gateway proxy. The Lambda will have to pass the headers, query parameters, and body to the internal service, gather the response and return it to API Gateway.

• This works great with REST APIs but what about TCP services like a relational databases or a cache servers?

Again, this would deserve a complete blog post. In a nutshell, we can create a micro-service (accessible via, you guessed it, an IAM authenticated API Gateway) that will create credentials on-the-fly for the desired services. For exemple: if the micro-service is asked for PostgreSQL credentials, it can issue a CREATE ROLE command with a randomly generated password. Additionally, the role could have a time-to-live using the VALID UNTIL property.

Alternatives

You’re not running your production workload on AWS? This method can’t work for you? There are other tools out there that can help you tackle secrets management.

• http://engineering.nike.com/cerberus/
• https://www.vaultproject.io/
• https://docs.docker.com/engine/swarm/secrets/

References for this article:

http://blog.arkency.com/2017/07/how-to-safely-store-api-keys-in-rails-apps/

https://blog.rackspace.com/securing-application-secrets-with-ec2-parameter-store

https://www.envkey.com/

https://www.hashicorp.com/blog/Vault-announcement/

https://docs.docker.com/engine/swarm/secrets/

http://docs.aws.amazon.com/apigateway/latest/developerguide/permissions.html

http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html

http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html

Special thanks to Tea Rudolf, Etienne Talbot, Simon-Pierre Gingras, Maxime Leblanc and Marilou Simard-Baril for the corrections (Yes, I need that much people to review my English) and thanks to Julie Dorion-Bélanger for the illustrations.

Got questions/suggestions about this post? Write a comment below and I’ll do my best to answer it.