Create Elastic Watcher and integrate with PagerDuty

In this blog post I’m going to take you through the process of setting up elastic watchers and also integration on PagerDuty.

Why watcher?

Watcher is an Elasticsearch feature that you can use to create actions based on conditions, which is periodically evaluated using queries on your data. It's helpful for analyzing mission-critical and business-critical streaming data.

In our system, watchers are really important. We want to notify ourselves about unexpected situations during the critical component initializations.

So what could these be? Message processors, Api Client initializations or could be an important message that we want to notify ourselves. As a solution, we preferred to use watchers.

Create Advanced Watch

There are two type of watchers.

• Threshold Alert
• Advanced watch

I'm going to explain in details how to create an advanced watcher.

You can configure them on Kibana watcher overview page. Click Create and choose Create advanced watch. An advanced watch requires a name and ID. Name is a user-friendly way to identify the watch, and ID refers to the identifier used by Elasticsearch.

Trigger — This schedule trigger executes the watch every 15 minutes.

Input — This input searches for errors in the * index and loads the response into the watch payload.

Condition — This condition checks to see if there are more than 1 error events (hits in the search response). If there are, execution continues for all actions.

Transform — If the watch condition is met, this transform loads all of the errors into the watch payload by searching for the errors using the default search type, query_then_fetch. All of the watch actions have access to this payload.

Actions — Notifies a 3rd party system about the problem. Configure with email, pagerduty, slack…

Elastic watcher configured successfully but we haven't configured any actions yet. It won't notify us with this setup. Somehow we need to configure it.

PagerDuty — Incident Management

PagerDuty is an incident management platform that provides reliable notifications, automatic escalations, on-call scheduling, and other functionality to help teams detect and fix infrastructure problems quickly.

As Beerwulf, we use PagerDuty for our incident management. It helps us quickly address incidents. We have a duty schedule that defined for our team.

If there is an incident on production, It immediately notify people on schedule. With this option, we instantly know that there is a degradation of services on production that affects customers.

We configured some triggers on PagerDuty which are NewRelic, Azure and Elastic Watcher. I'm going to explain integration on PagerDuty.

Integration

• Go to the Services menu and select Service Directory.
• If you are creating a new service for your integration, click New Service.
• If you are adding your integration to an existing service, click the name of the service you want to add the integration to. Then click the Integrations tab and click the New Integration button.
• If you have an integration key for Elastic Watcher, use it otherwise select Select a Tool and select Watcher from the dropdown.

Get your service key and update the service_key in the body of the action with the integration_key you copied.

The integration should be configured and you are all set. As soon as there is an issue that meets your watch’s condition, the incident will trigger within PagerDuty.