Stackdriver Monitoring Automation Part 5: Alerting Policies
This post is part 5 in the Stackdriver Automation series. In this post, I will walk through the steps that you can use to automate the management of Stackdriver Monitoring Alerting Policies with Terraform.
SLI Alerting Metrics
Head over to part 2 for the details of the alerting scenarios implemented with these Alerting Policies. To recap, the main Stackdriver Alerting conditions, notifications and documentation that I selected were the following:
Conditions
For Availability & Quality:
- L7 load balancer error request count = 0/sec
For Latency:
- L7 Load balancer total latency > 100ms
- L7 load balancer Frontend RTT latency > 50ms
- L7 load balancer Backend RTT latency > 50ms
For Volume:
- L7 load balancer request count > 100/sec
Notification channels
- Email: website-oncall@example.com
- Email: support-website@example.com
Documentation
- Specific text describing each alerting policy
Notification Channels Configuration
The 5 alerting conditions described above translated into 5 alerting policies and 2 separate notification channels. First, I created the notification channels configuration. Don’t forget to change your project values if you run in your own project.
provider "google" {
project = "abab-cdcd-023991"
region = "us-central1"
zone = "us-central1-c"
}resource "google_monitoring_notification_channel" "email0" {
display_name = "Website Oncall"
type = "email"
labels = {
email_address = "website-oncall@example.com"
}
}resource "google_monitoring_notification_channel" "email1" {
display_name = "Website Support"
type = "email"
labels = {
email_address = "support-website@example.com"
}
}output "email0_id" {
value = "${google_monitoring_notification_channel.email0.name}"
}output "email1_id" {
value = "${google_monitoring_notification_channel.email1.name}"
}
Creating the Notification Channels
Creating the Notification Channels requires 2 simple steps. First, initialize Terraform.
$ terraform initInitializing provider plugins...The following providers do not have any version constraints in configuration,
so the latest version was installed.To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.* provider.google: version = "~> 2.1"Terraform has been successfully initialized!
Then, the standard Terraform apply does all the work.
$ terraform applygoogle_monitoring_notification_channel.email1: Refreshing state... (ID: projects/abab-cdcd-023991/notificationChannels/12532464389112270325)
google_monitoring_notification_channel.email0: Refreshing state... (ID: abab-cdcd-023991/notificationChannels/11497409249297891355)An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-placeTerraform will perform the following actions:~ google_monitoring_notification_channel.email0
labels.email_address: "website-oncall@example.com" => "website-oncall-a@example.com"~ google_monitoring_notification_channel.email1
labels.email_address: "support-website@example.com" => "support-website-a@example.com"Plan: 0 to add, 2 to change, 0 to destroy.Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.Enter a value: yesgoogle_monitoring_notification_channel.email1: Modifying... (ID: projects/abab-cdcd-023991/notificationChannels/12532464389112270325)
labels.email_address: "support-website@example.com" => "support-website-a@example.com"
google_monitoring_notification_channel.email0: Modifying... (ID: projects/abab-cdcd-023991/notificationChannels/11497409249297891355)
labels.email_address: "website-oncall@example.com" => "website-oncall-a@example.com"
google_monitoring_notification_channel.email1: Modifications complete after 1s (ID: projects/abab-cdcd-023991/notificationChannels/12532464389112270325)
google_monitoring_notification_channel.email0: Modifications complete after 2s (ID: projects/abab-cdcd-023991/notificationChannels/11497409249297891355)Apply complete! Resources: 0 added, 2 changed, 0 destroyed.Outputs:email0_id = projects/abab-cdcd-023991/notificationChannels/11497409249297891355
email1_id = projects/abab-cdcd-023991/notificationChannels/12532464389112270325
Alerting Policies Configuration
I took the outputs from the notification channels (email0_id and email1_id above) and added them as local variables in the alerting policies configuration. The 5 alerting conditions described above translated into 5 alerting policies. Please note that I’ve only included 2 alerting policies here for the sake of readability. You can find the full configuration files on the github repo.
After the standard initialization, the terraform apply does all the work and creates the Alerting Policies.
$ terraform applygoogle_monitoring_notification_channel.email0: Refreshing state... (ID: projects/abab-cdcd-023991/notificationChannels/6314605697446264142)
google_monitoring_alert_policy.alert_policy0: Refreshing state... (ID: projects/abab-cdcd-023991/alertPolicies/48947729567500594)
google_monitoring_alert_policy.alert_policy4: Refreshing state... (ID: projects/abab-cdcd-023991/alertPolicies/16071394893024794286)
google_monitoring_alert_policy.alert_policy2: Refreshing state... (ID: projects/abab-cdcd-023991/alertPolicies/5670542644520942633)
google_monitoring_alert_policy.alert_policy3: Refreshing state... (ID: projects/abab-cdcd-023991/alertPolicies/6104269770159121403)
google_monitoring_notification_channel.email1: Refreshing state... (ID: projects/abab-cdcd-023991/notificationChannels/10463049156064412054)
google_monitoring_alert_policy.alert_policy1: Refreshing state... (ID: projects/abab-cdcd-023991/alertPolicies/5985603450478517474)
Once the alerting policies were created, I used the Stackdriver Monitoring console to verify that the alerting policies had been successfully created.
This concludes part 5 of the series. Read more about Stackdriver Monitoring Automation with Terraform in the other posts in the series and references below.
- Stackdriver Monitoring Automation Part 4: Stackdriver Groups with Terraform
- Stackdriver Monitoring Automation Part 6: Stackdriver Uptime Checks with Terraform
References:
