InfluxDB Task Alerts - Performance Monitoring

This article is intended for Performance Engineers who would love to see alert notifications during Performance monitoring using InfluxDB. Below are the below topics covered as part of this blog.

• Tasks in InfluxDB
• Setting up Tasks as Alerts for Performance monitoring
• Performance monitoring — Need for Alerting

Tasks in InfluxDB

Tasks are Flux queries which are customized and scheduled as periodic jobs that can also act upon on the result data by monitoring and performing necessary actions based on the conditions set. In this blog, we will focus only on setting up alerts from the InfluxDB Tasks and how they can be utilized in Performance monitoring.

Some pointers on the benefits of InfluxDB Tasks,

• Triggers alerts to Slack, emails etc.
• Execute queries at specific frequency to perform some action
• Downsampling of data

Setting up Tasks as Alerts

Let us dive into the various steps involved in setting up the Task as alerting system for our Performance test monitoring,

1. Login to the InfluxDB Cloud account, and go to Tasks

2. Create Task -> New Task

3. Fill the name of the task, how you would like the task to be scheduled (cron jobs are feasible as well) and enter the query.

Here, let us write the query that send Slack alerts when the 90th percentile response time spikes above the set threshold of 5 seconds.

4. Write the query and configure

Below is the complete configuration + query for the Slack alerting using the InfluxDB task.

Task Alert for Slack

Line-wise explanation

a. Lines 1:2 - Libraries to be imported, we are importing Slack and InfluxDB Secrets

b. Line 4 — Task name and the frequency (auto-generates) when we fill details in the Create Task -> New Task. Triggers using cron jobs can be set as well.

c. Lines 6 — Secured my Slack webhook using InfluxDB Secrets feature hence getting them invoked into the task. It is best to safeguard our Webhooks and InfluxDB provides Secrets feature to be utilized.

d. Lines 8:15 — Add details on configurations of sending the alerts to the Slack Webhook

e. Lines 17:27 — Query to be executed, this query states that the mean of the 90th percentile of the transactions to be retrieved followed by filtering them by the threshold of “5 seconds”. This part can be validated in the Data Explorer to ensure the results of the query are what we expect.

f. Lines 28:45 — Mapping the query results to if-else condition and subjecting to Slack Alerts when threshold is exceeded/met. Here, the condition validates for transactions that has 90th percentile greater than 5 seconds.

5. After configuring the query per above, save the Task. The task gets executed based on the set frequency and the logs can be viewed inside the task.

Task Logs

6. Task does have the option to turn ON/OFF anytime, so this does provide advantage to turn them ON only when we need or keep them ON always.

Below is the alert message that was triggered to the Slack channel when the 90th percentile threshold of 5 seconds was exceeded during the Performance test run.

Slack Alert when the transaction crossed its threshold

Similar to how we got the alerts triggered to Slack, we can get them in our emails too!

Why need an Alerting system during Performance tests?

1. InfluxDB Tasks provides us with LIVE notifications on the spikes observed in various performance metrics such as the response times, error rates, infra stats such as CPU, memory, disk etc. during the test run.
2. Performance tests certainly does gives us the adrenaline pump especially while we perform certain types of Performance tests where the chances of failure of the system is somehow expected. These tests usually keeps us on the verge of the seat so as to start with the mitigation activities in cases of any adverse effects or finding any root causes.
3. At times, there are chances of missing the exact timeframe when things start going downhill and alert notifications provide us with immense support.

Kindly refer to the below blog on setting up JMeter Live monitoring with InfluxDB, the query used in this blog is based on the same.

https://medium.com/@cuteberry.madhu/jmeter-live-monitoring-using-influxdb-79e2a63fb23e

Thank you :)