Write error logs from Composer and create an alert policy on Stackdriver
You want to create a custom alert on Stackdriver for one of your DAG in Composer Airflow? You have heard about logging and monitoring but you don’t really know how to do that? If the answers are yes, check out this blog post!
Picture a scenario:
- You are working on GCP.
- You want to create a custom alert on Stackdriver for one of your DAG in Composer Airflow.
- You have heard about logging and monitoring but you don’t really know how to do that.
If this applies to you, then you’ve come to the right place! This tutorial will help you with just that!
Prerequisites:
- Google Cloud Project
- A Composer environment
For the purposes of this tutorial, we will be working on a Composer environment named composer-log-error-example. Note that its location is set to europe-west1.
PART 1 — CREATING THE ERROR LOGS IN COMPOSER
In the following example, we will make use of the Logging Client Library (already installed in Composer).
Here is a script example:
Explanation of some of the more notable points
As a side-note, if you’re looking for a bit more structured logs, instead of log_text, you may use log_struct:
Put your script in the DAGs folder and trigger your DAG.
Results
Go to Logging>Logs Viewer:
Select the Cloud Composer Environment resource, then the europe-west1 location and, finally, the name of the composer environment: composer-log-error-example.
In the second tab, select the airflow-worker logs. In the third tab, select Error.
Your error should appear in the logs.
PART 2 — CREATING AN ALERT SYSTEM ON STACKDRIVER
Great! We’ve now learned how to file our custom error logs. However, periodically checking the Log Viewer for them is hardly efficient. It’d be great if we could somehow create a notification system that triggered whenever they occurred.
To do this, we will first have to create a metric. A metric will keep track of the number of logs in our composer’s error log.
On the same page as before, and using same filters as before, click on the Create metrics button (top of the page).
Give your new metric a name, a description, labels and units. For the type, select Counter. Then click on Update Metric.
We will now create an alert based on our new metric.
First go to Monitoring>Alerting:
Click on create policy. Give a name to your new alert and click on Add Condition.
Write the name of your metric in Target. In our case, we will be looking for ‘composer_error’.
For Aggregator, select count.
In the Configuration section, set Condition to is above,Threshold to 0 and For to 1 minute.
Click on Add.
You can also add a Notification. The options range from e-mail, to Slack, to even SMS.
In the Documentation field, you can write text that will be send in the notification. You can use Markdown for the formatting.
Click on Save.
Your new alert is now created.
Test it! Trigger your DAG again.
After 1–2 minutes, an incident should appear in the alerting page.
If you had set up an e-mail notification, you should also receive something like this:
Conclusion
We’ve shown how to easily create some error-type logs for Composer. In addition, we’ve demonstrated how to use logging and monitoring to create an alert policy for them in Stackdriver.
What’s next?
To go further:
- GCP Official documentation:
Health Checking your Cloud Composer Environment
- GCP GitHub repository with examples:
Python Docs Samples- Connect to Preview
- google-cloud-logging library information:
Python Client for Cloud Logging (we recommend the following guide)
Originally published at https://www.syntio.net, March 20, 2020.
