Secret Zero with Hashicorp Vault in AWS Lambda

We at Preply heavily rely on Hashicorp Vault to manage sensitive data. Recently, we’ve completely refactored our Vault setup and made it more reliable, observable, and secure. One of our main focuses was to get rid of long-lived tokens needed for login and get secrets from Vault.

As we’re completely hosted in AWS and run all our workload in Kubernetes, we decided to go with the mix of 2 options: both AWS and K8S auth methods. We did it for the whole infrastructure: EC2 fleet, CI/CD, Airflow, etc.

I’m Vlad and I am an Infrastructure Engineer at Preply. Here I want to describe “Secret Zero” approach with Vault and particularly highlight the way we integrated AWS services like Lambda. I’ll hope you enjoy it, let’s start!

What is the AWS auth method?

The AWS auth method provides an automated mechanism to retrieve a Vault token for IAM principals and AWS EC2 instances. It consists of 2 authentication types: iam and ec2.

AWS IAM is the new recommended method because it’s more flexible and can be used wherever the IAM instance profile is supported (in Lambda, for instance). Moreover, due to the nature of the AWS signature algorithm used in the IAM method, API requests automatically expire in 15 minutes in comparison to ec2-based where the identity document is relatively static and should be protected by the client nonces. It makes this method simpler.

However, the EC2 method provides you with more granular EC2 filtering. Instead of solely relying on the IAM instance profile, the EC2 method can validate instance/AMI IDs, VPC/subnet IDs, role tags, etc. It is specialized to handle EC2 instances, so it’s obviously more versatile for them.

All in all, both methods are valid and can be involved depending on the use case. We’ve primarily chosen the IAM method because of Lambda itself.

Basic flow: how to authenticate EC2 instances?

First, let me briefly describe how to authenticate EC2 instances with Vault via the IAM auth method. For that we need the following:

• IAM role that will be used to authenticate against Vault. It should have a trust policy that specifies the principal "Service": "ec2.amazonaws.com" is trusted to call AssumeRole.
• IAM policies for Vault server to properly resolve the full user/role path https://www.vaultproject.io/docs/auth/aws#recommended-vault-iam-policy
• (Optional) X-Vault-AWS-IAM-Server-ID header that helps mitigate different types of replay attacks
• (Optional) Configure STS endpoint if you want to use other than default one in us-east-1
• (Optional) Some “bound_ids” when creating the Vault role are needed to restrict its usage to certain entities. It can be an account, AMI, VPC, subnet, etc.

With all the above, creation looks pretty simple:

$ vault auth enable aws
$ vault write auth/aws/config/client iam_server_id_header_value=vault.example.com sts_endpoint=https://sts.eu-west-1.amazonaws.com sts_region=eu-west-1
$ vault write auth/aws/role/dev-role-iam auth_type=iam bound_iam_principal_arn=arn:aws:iam::123456789012:role/MyRole policies=prod,dev max_ttl=1h

And that’s it! Now you can log in to Vault via

$ vault login -method=aws header_value=vault.example.com role=dev-role-iam

In case you don’t want to use CLI you can optionally use your favorite language library. For Python, we use HVAC which stands for “Hashicorp Vault API Client”. Here is how simple AWS IAM auth can look like:

import boto3
import hvac

VAULT_ADDR = os.environ["VAULT_ADDR"]
VAULT_HEADER_VALUE = os.environ["VAULT_HEADER_VALUE"]

client = hvac.Client(url=VAULT_ADDR)
client.auth.aws.configure(
    access_key='SOME_ACCESS_KEY_FOR_VAULTS_USE',
    secret_key='SOME_ACCESS_KEY_FOR_VAULTS_USE',
    endpoint='https://sts.us-west-1.amazonaws.com',
)

session = boto3.Session()
creds = session.get_credentials().get_frozen_credentials()
client.auth.aws.iam_login((
    access_key=creds.access_key,
    secret_key=creds.secret_key,
    session_token=creds.token,
    header_value=VAULT_HEADER_VALUE,
    role='some-role,
    use_token=True,
    region='us-west-1',
)

What about Lambdas?

Looks easy, right? But things get more interesting when it comes to Lambda.

The first idea that comes to mind is to use the code above by injecting the vault library into every single lambda. But it quickly becomes a massive burden to manage and it’s not convenient at all.

Not so long ago AWS presented a new feature for Lambda — Lambda Extensions. It allows you to easily integrate Lambda with a third-party tool that has a needed extension. And it works as simple as adding an extra layer to your lambda or including to the image for functions deployed as container images. Extensions are supported across all major Lambda runtimes with a full list available in the official documentation.

Under the hood, AWS has created a special API that you can use to build your own extension. This API builds on the existing Lambda Runtime API, which enables you to bring custom runtimes to Lambda. The cool thing here is that the extension shares the execution environment of Lambda and can be triggered at different runtime stages: during function initialization, invocation, shutdown, and even after it. Of course, it is initialized before the runtime and the function itself.

Fortunately, Hashicorp has added such an extension for their Vault service.

To make this work you should add a layer to your Lambda with the following ARN: arn:aws:lambda:{region_name}:634166935893:layer:vault-lambda-extension:{layer_version}, where “{region_name}” is the AWS region that supports this layer and “{layer_version}” — is the version of this layer published by Hashicorp. By configuring the extension via Lambda environment variables you are setting up Vault address, path, role, etc. A full list of configuration options is available in the official repository.

As you may have already guessed Vault authentication is done via the AWS IAM auth method and secrets are fetched during Lambda invocation. For this method to be really “secret zero” we should also care about the way secrets are injected into Lambda runtime. By default, this extension stores secrets to the disk inside /tmp/vault/secrets.json that Lambda reads. Alternatively, there is a way to configure a special proxy server running alongside the extension which will add an authentication header and proxy requests to Vault. All you need to do is set VAULT_ADDR variable to http://127.0.0.1:8200 and use this endpoint in your Lambda function when getting secrets from Vault.

Another reason writing data to disk isn’t perfect is that the extension will only write to disk once per execution environment, rather than once per function invocation. It can lead to failure if your secrets are short-lived.

Using Serverless Framework a working example of such Lambda can look like this:

service: lambda-test
provider:
  name: aws
  region: eu-west-1
  role: arn:aws:iam::{account_id}:role/{role-name}
  layers:
    - arn:aws:lambda:eu-west-1:634166935893:layer:vault-lambda-extension:14
  vpc:
    securityGroupIds:
      - {sg_id}
    subnetIds:
      - {subnet_id}
  environment:
    VAULT_ADDR: http://127.0.0.1:8200 # connect to local proxy
    VLE_VAULT_ADDR: https://vault:443 # connect to Vault from proxy    
    VAULT_AUTH_PROVIDER: aws
    VAULT_AUTH_ROLE: lambda-iam-role
    VAULT_IAM_SERVER_ID: vault
    VAULT_STS_ENDPOINT_REGION: eu-west-1
    VAULT_SECRET_PATH: lambda/data/test
functions:
  handle:
    handler: handler.handle
    memorySize: 128
    timeout: 360
    runtime: python3.9
    events:
      - schedule: rate(5 minutes)

Performance impact has a tiny footprint: layer is less than 10MB in size with only 10ms of initialization latency as stated in the Hashicorp benchmark and that’s what we see from our monitoring as well.

What if we have a multi-account AWS organization?

When it comes to managing AWS Lambdas across different accounts, there is a way to do it. You can use AWS STS to assume IAM Roles in other accounts. For each account, you need to configure a separate role with a special STS config for Vault to assume that role from another account. This config can be created like this:

# vault write auth/aws/config/sts/123456789012 sts_role=arn:aws:iam::123456789012:role/VaultAccess

where 123456789012 is the AWS account id your Lambda is running in. Remember that Vault server must have sts:AssumeRole permission and the roles themselves — a Vault account as a trusted entity.

Future improvements

One of the action items we have is to decrease the load on the Vault server. It’s obvious that generating a request to Vault per Lambda invocation can quickly impact performance. So it makes sense to cache secrets for some short period of time, smaller than a secret’s lifetime. Fortunately, Vault extension also provides a mechanism to cache secrets in a local proxy. That’s the one that we still need to test but I have a good feeling. More information about caching configuration can be found here.

Thanks for reading so far. If you are interested in more, check out our Engineering blog.

Besides, we’re also hiring. Our careers page has now 14 open engineering positions both in Kyiv and Barcelona. Feel free to reach out!