Setting up AWS secrets backends with Airflow in a cost-effective way

Photo by Jason Dent on Unsplash

What is a secret and how are secrets managed in Airflow?

Apache Airflow is an orchestration platform to programmatically author, schedule, and execute workflows. By nature, Airflow manages a lot of data, including sensitive information often referred to as ‘secrets’. We are going to have a look at what these secrets represent, how they are managed in Airflow and the different AWS secrets backends you can use to manage these secrets leveraging AWS services. Lastly, we will go over use cases where secrets backends can become costly and will explore the different options available for cost savings.

The secrets in Airflow are variables and connections. These secrets need special treatment in order to be protected while being used such as:

1. Encryption. Secrets are encrypted when being stored in the database. It guarantees that secrets cannot be maliciously read without the encryption key.
2. Masking. Airflow masks secrets before printing them out in logs and displaying in any UI.

Secrets can be defined and stored into two different locations:

1. Environment variables. Secrets can be created and managed using environment variables.
   - How to manage variables using environment variables
   - How to manage connection using environment variables
2. Database. Secrets can also be stored in the database. In order to manage them, you can use the user interface or any other interface such as Airflow CLI or the Rest API.

When it comes to retrieving secrets, order matters. Airflow looks up secrets first in the environment variables and then, if not found, in the database.

However, Airflow offers an optional layer where secrets can be looked up before the environment variables: the secrets backend.

What is a secrets backend?

Airflow provides an option so you can enable an additional layer called secrets backend to retrieve Airflow connections or Airflow variables. These secrets backends are defined by the different providers and very often, depending on the provider, use a third party service to store and manage your secrets. You can see the list of secrets backends here. When a secrets backend is being used, Airflow will look up first secrets in this secrets backend before going over environment variables and the database.

In order to use a secrets backend, you need to specify it in the configuration in the section [secrets] as follows:

[secrets]
backend =
backend_kwargs =

More information about configuring a secrets backend in Airflow can be found here.

AWS Secrets Backends

This section and the rest of this blog will now focus on AWS specific secrets backends. AWS provides two different secrets backends in order to manage your secrets. We will go over these two different secrets backends with some examples on how to configure them, then we will compare them so you can chose the one that best suits your needs.

AWS Systems manager

AWS Systems Manager backend leverages AWS Systems Manager service to manage your secrets. To use it as a secrets backend you need to set the configuration as follows:

[secrets]
backend = airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
backend_kwargs = {
  # See documentation to see all the different options for backend_kwargs
  # https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-ssm-parameter-store.html
}

Let’s take an example of configuration to have a better understanding.

[secrets]
backend = airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
backend_kwargs = {
  "connections_prefix": "/airflow/connections",
  "variables_prefix": "/airflow/variables",
}

With the above configuration, any connection and variable have to be prefixed in AWS Systems Manager with /airflow/connections (for connections) and /airflow/variables (for variables). In other words, for a connection id smtp_default in Airflow, you want to store your connection at /airflow/connections/smtp_default in AWS Systems Manager. Similarly, for a variable key foo in Airflow, you want to store your variable at /airflow/variables/foo in AWS Systems Manager. AWS systems manager mandates to have a leading / for each parameter.

AWS Secrets Manager

AWS Secrets Manager backend leverages AWS Secrets Manager service to manage your secrets. To use it as secrets backend you need to set the configuration as follows:

[secrets]
backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
backend_kwargs = {
  # See documentation to see all the different options for backend_kwargs
  # https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-secrets-manager.html
}

Let’s take an example of configuration to have a better understanding.

[secrets]
backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
backend_kwargs = {
  "connections_prefix": "airflow/connections",
  "variables_prefix": "airflow/variables",
}

Very similarly to AWS Systems Manager, with the above configuration, any connection and variable have to be prefixed in AWS Secrets Manager with airflow/connections (for connections) and airflow/variables (for variables). For a connection id smtp_default in Airflow, you want to store your connection at airflow/connections/smtp_default in AWS Secrets Manager. For a variable key foo in Airflow, you want to store your variable at airflow/variables/foo in AWS Secrets Manager. As opposed to AWS Systems Manager, AWS Secrets Manager does not enforce each secret to start with a /.

Comparison

Which one to chose then? Even though AWS Systems Manager and AWS Secrets Manager are both a key/value store in which you can store your secrets, they still have some differences you should be aware and which should help you decide on which secrets backend to use. Let’s go over these differences.

There is obviously not a real answer to which one of the secrets backend you should use. That depends on many parameters such as your usage, your budget, your preference, etc. The price is always an important factor when making these decisions, but as you can see in the table above, AWS Secrets Managers also offers some nice to have features which could potentially save you valuable development time. In the next section we will go over an example where using a secrets backend can become very costly and the various strategies to reduce that cost.

Optimizing cost of Secrets backend

In order to understand how secrets backends can become costly, let’s take an example Airflow environment and then see the different solutions to reduce the cost. We will assume in the example below that we are using AWS Secrets Manager or AWS System Managers with advanced parameters or higher throughput (the price is the same in all these situations: $0.05 per 10,000 API interactions, we will ignore the storage price).

Example

While working with customers using Amazon MWAA, we had multiple instances where customers’ production system looked like below configuration.

• Number of DAGs: 5000
• Average number of variables per DAG: 20
• Average number of connections per DAG: 5
• DAG parsing interval (scheduler.min_file_process_interval) set to 30 seconds (by default)
• DAGs executed every minute on average

Solution 1. Avoid top level code

The first solution is part of the Airflow best practises: avoid writing top level code. Top level code is the code that is executed directly within the main body of a module, rather than being inside a function or a class definition. This top level code is executed every time the Airflow scheduler parses a given DAG. By default, this happens every 30 seconds (configurable through the configuration min_file_process_interval). Below is an example of what you should NOT do.

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.models import Variable

 var = Variable.get("foo")  # AVOID THIS!

@task
def example_task():
    print(var)

with DAG(
    dag_id="example",
    schedule=None,
    start_date=datetime(2021, 1, 1),
    catchup=False,
) as dag:
    task = example_task()

Fetching some secrets at the top level code of your DAG results in making an API call to AWS Secrets Manager or AWS Systems Manager for each secret every-time this given DAG is parsed. Let’s figure how much that would cost you per month with this example.

• Number of times one DAG is parsed per month: 2 (DAG is parsed every 30 seconds, so twice per minute) * 60 * 24 * 30 = 86400
• Number of API calls per month: 86400 * 5000 (number of DAGs) * 20 (number of variables) = 8640000000
• Cost per month: (8640000000 / 10000) * $0.05 = $43,200 (AWS Secrets Managers charges $0.05 per 10,000 API calls, see more pricing details here).

Instead, you should do.

from airflow import DAG
from airflow.decorators import task

@task
def example_task():
    var = Variable.get("foo")
    print(var)

with DAG(
    dag_id="example",
    schedule=None,
    start_date=datetime(2021, 1, 1),
    catchup=False,
) as dag:
    task = example_task()

Because you read variables at the top level code of your DAGs, Airflow scheduler actually calls AWS services when parsing your DAGs which adds up to an additional cost of $43,200 per month. This is just the parsing cost, it does not include the actual cost of your DAG executions. By following best practices and not writing code at the top level, Airflow scheduler would not call any secrets backend services during parsing. In this example, this means saving $43,200 per month.

Solution 2. Use lookup pattern for secrets backend

This solution involves a feature released as part of the Amazon provider package version 7.3.0, you must use such version or higher in order to leverage it.

Before exploring this solution, let’s figure how much your DAG executions would cost you per month. By not writing top level code as explained in the previous solution you have no extra parsing cost due to secrets backends, you still have naturally some cost engendered by the executions of your DAGs.

• Number of times one DAG is executed per month: 60 (DAG is executed every minute) * 24 * 30 = 43200
• Number of API calls per month: 43200 * 5000 (number of DAGs) * 25 (number of secrets: variables + connections) = 5400000000
• Cost per month: (5400000000 / 10000) * $0.05 = $27,000

In this example, as a result of your DAG executions, the cost associated to the secrets backend you are using is $27,000per month. Is it possible to lower this cost? Depending on your situation, you might. First, let’s see how the secrets backends work.

Every time a DAG is executed, for every connection defined in your environment and every variable you read in your DAG, Airflow makes an API call to AWS Secrets Manager or AWS Systems Manager. Regardless whether these secrets exist in the secrets backend service, you make these calls increasing your DAG parsing time as well as cost, if you set up additional trails, the additional copies of management events can incur costs. If among the 5 connections defined in your environment, only one is stored in the secrets backend service and the rest are defined in the environment variables or database, Airflow will still try to look up the 5 connections in the secrets backend service first. Only 1 will be retrieved successfully from the secrets backend service but your trail cost would include 5 requests made to the AWS service. The same logic goes for the variables, if only 5 variables are defined in the secrets backend service, 20 requests will still be made to the AWS service for every DAG execution.

If you are in such a situation, there is a solution for you! AWS secrets backends have options to specify through a pattern which connection or variable need to be looked up in the secrets backend service. These options are connections_lookup_pattern and variables_lookup_pattern, they need to be added as part of the backend_kwargsconfig. These options are available in both AWS Systems Manager and AWS Secrets Manager secrets backend. See example below.

[secrets]
# For the example I use AWS Systems Manager but AWS Secrets Manager can be used equally
backend = airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
backend_kwargs = {
  "connections_prefix": "airflow/connections",
  "connections_lookup_pattern": "_default$",
  "variables_prefix": "airflow/variables",
  "variables_lookup_pattern": "^aws_",
}

In this example, because you specify connections_lookup_pattern to _default$, Airflow will only look up connections which end with _default in the specified secrets backend service. It does not mean the other connections will not be resolved, it means Airflow will bypass the secrets backend and will directly look them up in environment variables and then, if not found, in the database. Same goes for variables_lookup_pattern, because variables_lookup_pattern is set to ^aws_, Airflow will only look up variables which start with aws_ in the specified secrets backend service.

from airflow.decorators import task

@task
def example_task():
    var = Variable.get("aws_var")  # Airflow will look it up first in the secrets backend service
    var = Variable.get("foo")  # Airflow will not look it up in the secrets backend service

Using these options improve your performance and reduce overall AWS cost.

Summary

Secrets backends are very powerful tools which can help you manage your secrets in third party services. They are fully integrated with Airflow and with just a simple configuration, they are ready to use. However, please be aware that in some cases they might become very expensive. If that’s the case, depending on your situation, you might have solutions to reduce that cost in a very easy way!