Composer, Sendgrid and Secrets
A client wanted to use GCP Composer (Airflow) to send notification emails via Sendgrid when a workflow fails. When one wishes to use Sendgrid, one must supply an ApiKey in order to prove that you are authorized to send emails. The client wanted to store the Sendgrid API key in GCP Secrets. This article walks us through the recipe to achieve this.
We’ll start with the notion that in order for a Workflow to automatically send a notification email when it fails, this must be declared in the DAG.
Here is a sample DAG that does just that:
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.operators.bash import BashOperator
from airflow.exceptions import AirflowFailException# These args will get passed on to each operator
# You can override them on a per-task basis during operator initialization
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email': ['jones@example.com'],
'email_on_failure': True,
'email_on_retry': True,
'retries': 1,
'retry_delay': timedelta(minutes=5),
}
with DAG(
'tutorial',
default_args=default_args,
description='A simple DAG',
schedule_interval=timedelta(days=1),
start_date=datetime(2021, 1, 1),
catchup=False,
tags=['example'],
) as dag:t1 = BashOperator(
task_id='print_date',
bash_command='date',
)def task_to_fail():
raise AirflowFailException("Forced Exception!")t2 = PythonOperator(dag=dag,
task_id='throw_exception',
python_callable=task_to_fail
)t1 >> t2
The key things I want you to see are:
- The DAG throws an exception and hence the workflow as a whole fails.
- The “
email_on_failure” flag is set to True. - An email address is supplied in the “
email” list which is where the notification emails will be sent.
If we run this DAG, we will find it fails as desired however there is more work to be done. We have not yet told Composer how to send an email. There are multiple ways that an email can be sent including SMTP, Sendgrid and others.
Airflow is primarily driven from a configuration file called airflow.cfg. This file is basically a name/value set of pairs split into related sections. One of those sections is specific to emails that are generated internally from Airflow. Two parameters are of prime interest to us. The first is called “email_backend”. This names the logic to be invoked when an email is to be sent. This allows us to provide customizations for how to send an email. Apache has provided just such a plugin which provides the hooks to Sendgrid. The second parameter of interest is called “email_conn_id” and is used to supply an Airflow connection name. This connection is supposed to hold the parameters required by the email subsystem referenced by the email_backend. This connection id will be used to obtain the information necessary to allow Airflow to use Sendgrid. For our purposes, we want the airflow.cfg to now include:
[email]
email_backend=airflow.providers.sendgrid.utils.emailer.send_email
email_conn_id=sendgrid_defaultLater on in the recipe, we will show how to set these values in a Composer environment.
When an email is sent by Airflow, the email will be seen to have originated from some source email address and with a name associated with that address. It would have been preferable if those parameters could have been part of the connection id but sadly that isn’t the case. Instead, two environment variables are used to supply these values:
- SENDGRID_MAIL_FROM=mycomposer@example.com
- SENDGRID_MAIL_SENDER=Composer
It may be necessary to configure Sendgrid to authorize emails originating from the source address.
At this point we could create an Airflow Connection definition and our solution would run; however, we wanted to employ GCP Secrets as our vault for secure information. To do that we need an additional entry in our airflow.cfg file.
To enable the use of Secret manager we need to modify the airflow.cfg to include an entry in the secrets section called secrets. This is used to name some plugin logic that provides a hook to where secrets are stored. Airflow provides a plugin for GCP Secret Manager. The entry in airflow.cfg should thus be:
[secrets]
backend=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackendWhen this is configured, Airflow will consult GCP Secret Manager when secrets are needed. Connection definitions are also considered secrets. If we create an appropriately named entry in Secret Manager, Airflow will use the value of that secret as a Connection Id. The value must be encoded in the Connection Id URL format.
For Sendgrid, we can create a value of the following format:
sendgrid://username:[SOME_API_KEY]@hostIt is only the API key that is examined in the value. The name of the secret is also extremely important. The structure of the name of the secret for a Connection Id is:
airflow-connections-[CONNECTION_ID]For our purposes, this will mean:
airflow-connections-sendgrid_defaultAnd that’s it … all the pieces are plugged together.
Now we will provide a walk through illustrating all the parts.
- Create a new Composer environment
Create a new GCP Composer environment. Accept the majority of defaults. In my tests, I used Airflow version 2.1.4. Before confirming, be sure and specify values for the Airflow Configuration Overrides and Environment Variables:
Airflow Configuration Overrides
[email]
email_backend=airflow.providers.sendgrid.utils.emailer.send_email
email_conn_id=sendgrid_default
[secrets]
backend=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
[logging]
logging_level=DEBUG(Note that the logging entry is not required but, it has been my experience, that during development/testing it is extremely valuable)
Environment Variables
- SENDGRID_MAIL_FROM=mycomposer@example.com
- SENDGRID_MAIL_SENDER=Composer
The creation of a Composer Environment can take 10–20 minutes.
2. Create a Secret in GCP Secrets manager
In the GCP Secrets manager, create a new secret called airflow-connections-sendgrid_default. Set the value of it to be
sendgrid://username:[SENDGRID_API_KEY]@host3. Authorize Composer to read the secret
At runtime, Composer will need to be able to read the value of the Secret from the Secret Manager. Ensure that the IAM permission is added to allow access.
4. Run a DAG that fails
If we now run a DAG that fails that is configured to send an email on failure, we will find that an email has been sent.
See also:
