How to Develop and Deploy a Webhook Alert Action App with Custom Payload and Headers for Splunk

Ankit Tyagi
Jun 29, 2020 · 6 min read

Splunk is a wonderful tool with loads of features, yet a few of them are not available out-of-the-box — one of those missing features is the customizable webhook alert action.

While working on a project, I needed to send some custom metrics along with some headers from Splunk alerts and ended up exploring the default available webhook alert action.

Here’s what’s already available

Splunk, by default, provides webhook alert action for an alert; however, it is lacking some of the required features like:

  • Authentication mechanism
  • Custom header support
  • Custom payload support

With the above features being missing from the default webhook alert action on Splunk, it becomes much less useful because the webhooks mostly require some kind of authentication to receive the data and similarly, customized payload might be a requirement for some webhook APIs.

Splunk has done a great job with its pluggable platform, which allows you to write your own apps and deploy them to serve your custom requirements. Despite this, building a custom app can be tricky and time-consuming.

After reviewing the webhook alert action I decided to develop my own custom webhook alert action with all the required missing features.

customWebhook app to the rescue

Image for post

I developed a customWebhook alert action application with the following configurable options:

URL: Target webhook URL

e.g. https://server.com/api/v2/webhook

Headers (dict): Include any number of headers

e.g. {‘Authorization’: ‘Bearer <token>’, ‘Content-Type’: ‘application/json’…}

Payload (dict): Payload data with key-value pairs of your choice

e.g. {‘service_name’: ‘MyService’, ‘key1’: ‘value1’, ‘key2’: ‘value2’…}

I’ll walk you through easy and simple steps that would give you a kick start with your Splunk project.

Objectives:

1. Understand the Splunk app directory structure

2. Components with the respective directory structure

3. Configuration and logic code snippet

4. Deploy the application on the Splunk server

5. Create an alert with customWebhook alert action

Understand the Splunk app directory structure

snapshot of application directory structure
A snapshot of the application directory structure.

Let’s break it down.

This is an exhaustive list of configuration I used in the customWebhook app, The complete list is here: https://docs.splunk.com/Documentation/Splunk/8.0.4/AdvancedDev/ModAlertsCreate

Image for post

Configuration and logic code snippet

alert_actions.conf

# Configuration file path#$SPLUNK_HOME$/etc/apps/customWebhook_app/default/alert_actions.conf[customWebhook]is_custom = 1label = customWebhookdescription = Send CustomWebhook alert notificationsicon_path = customWebhook_icon.pngpayload_format = jsonparam.alert_source = Splunkparam.user_agent = Splunk/$server.guid$

app.conf

[ui]is_visible = 0label = customWebhook alerts[launcher]author = Ankit Tyagidescription = Send alert payload to custom webhookversion = 1.0[install]state = enabledis_configured = 1

restmap.conf

[validation:savedsearch]action.customWebhook = case('action.customWebhook' != "1", null(), 'action.customWebhook.param.base_url' == "action.customWebhook.param.base_url" OR 'action.customWebhook.param.base_url' == "", "No Webhook URL specified",  1==1, null())action.customWebhook.param.base_url = validate( match('action.customWebhook.param.url', "^https?://[^\s]+$"), "Webhook URL is invalid")

default.meta

[alert_actions/customWebhook]export = system[alerts]export = system[restmap]export = system

customWebhook.html

<!--$SPLUNK_HOME$/etc/apps/customWebhook_app/default/data/ui/alerts/customWebhook.html-->

<form class="form-horizontal form-complex">
<div class="control-group">
<label class="control-label" for="customWebhook_base_url">URL</label>

<div class="controls">
<input type="text" class="input-xlarge" name="action.customWebhook.param.base_url" id="customWebhook_base_url" placeholder="https://server.com/api/v2/webhook/" />
<br>
<span class="help-block">
Webhook URL (str) https://server.com/api/v2/webhook
</span>
</div>
</div>
<div class="control-group">
<label class="control-label" for="customWebhook_headers">Headers</label>

<div class="controls">
<input type="text" name="action.customWebhook.param.headers" id="customWebhook_headers" placeholder="{'Content-Type': 'application/json', 'CustomHeader': 'Value'}"/>
<br>
<span class="help-block">
Headers (dict) <br>
e.g. {'Content-Type': 'application/json', 'CustomHeader': 'Value'}
</span>
</div>
</div>
<div class="control-group">
<label class="control-label" for="customWebhook_payload">Payload</label>

<div class="controls">
<input type="text" name="action.customWebhook.param.payload" id="customWebhook_payload" placeholder="{'key1': 'value1', 'key2': 'value2'}"/>
<br>
<span class="help-block">
Payload data (dict)
e.g. {'key1': 'value1', 'key2': 'value2'} <br>
add the key from seach result like this. <br>
e.g. {'key1': 'value1', 'key2': '$result.key_name$'}
</span>
</div>
</div>
</form>

alert_action.conf.spec

#$SPLUNK_HOME$/etc/apps/customWebhook_app/README/alert_actions.conf.spec[customWebhook]param.alert_source = "Splunk"

customWebhook.py

##################################################################   A splunk plugin developed for sending alerts from Splunk#   to a webhook endpoint with custom payload#   Developed by:#       Ankit Tyagi#       SRE - Adobe################################################################## File location on Splunk server:
#$SPLUNK_HOME$/etc/apps/customWebhook_app/bin/customWebhook.py
from __future__ import print_functionfrom future import standard_librarystandard_library.install_aliases()import sys, jsonimport requestsdef process_args(*args): return json.dumps(args)def send_alert_data(*args): args = json.loads(process_args(*args)) for config in args: if 'headers' in config: headers = config.get('headers') headers = headers.replace("'", '"') headers = json.loads(headers) if 'payload' in config: data = config.get('payload') data = data.replace("'", '"') if 'base_url' in config: url = config.get('base_url') if 'alert_source' in config: alert_source = config.get('alert_source') data = json.loads(data) data['alert_source'] = alert_source data = json.dumps(data) try: res = requests.post(url=url, data=data, headers=headers) print("Server response: %s" % json.dumps(res.json()), file=sys.stderr) return 200 <= res.status_code < 300 except requests.exceptions.RequestException as e: print("ERROR Error sending message: %s" % e, file=sys.stderr) return False
if
__name__ == "__main__":
if len(sys.argv) > 1 and sys.argv[1] == "--execute": payload = json.loads(sys.stdin.read()) if not send_alert_data(payload.get('configuration')): print("Failed trying to send alert notification", file=sys.stderr) sys.exit(2) else: print("INFO alert notification successfully sent", file=sys.stderr) else: print("FATAL Unsupported execution mode (expected --execute flag)", file=sys.stderr) sys.exit(1)

Deploy the application on the Splunk server

Bundle the application components and files in a directory with the name: alert_<app_name> here we’d use alert_customWebhook_app as the name of the directory.

Splunk admin need to deploy the directory alert_customWebhook_app in $SPLUNK_HOME$/etc/apps/ on Splunk server, so finally we’d have, $SPLUNK_HOME$/etc/apps/alert_customWebhook_app

final directory structure of the application
the final directory structure of the application

Create an alert with customWebhook alert action

1. Go to search and put the search criteria and click the search button.

Note: you may want to define some custom variables to load into the alert, for that use Splunk’s ‘eval’. For e.g. in the screenshot below.

Image for post

2. Save the search as an alert

Image for post

3. Set up the alert configurations. Not all the options are mandatory, however, so change them accordingly. For example, I used cron scheduler, which does a search based on the search criteria I selected while creating this alert and compares the result with the threshold as shown in the next figure.

Image for post

4. Set up the trigger condition

Image for post

5. Action time

Now add the desired action to get notified about this alert. Select the action, customWebhook.

Image for post

6. After clicking on customWebhook alert action, a configuration page opens up

Image for post

Put in the webhook URL, Headers, and Payload information.

Headers and Payload are dictionary type.

The evaluated variables with ‘eval’ can be used to send the custom data in the payload, to do that add the variable like this:

{‘service_name’: ‘$result.service_name$’}

Save the alert

7. To further edit the alert or add new actions, go to alerts from the menu and click on your alert.

Image for post

What’s next?

This customWebhook app is capable of sending alerts custom data on a single URL. Next, I’ll be working on enhancing it to be able to send alerts data on multiple URLs at the same time. Please leave your suggestions in the comments.

References:

https://docs.splunk.com/Documentation/Splunk/8.0.4/AdvancedDev/ModAlertsCreate

https://docs.splunk.com/Documentation/Splunk/8.0.4/AdvancedDev/ModAlertsIntro

Adobe Tech Blog

News, updates, and thoughts related to Adobe, developers…

Ankit Tyagi

Written by

Site Reliability Engineer @Adobe. Here, my bits to the community of nerds that has been there for an epic!

Adobe Tech Blog

News, updates, and thoughts related to Adobe, developers, and technology.

Ankit Tyagi

Written by

Site Reliability Engineer @Adobe. Here, my bits to the community of nerds that has been there for an epic!

Adobe Tech Blog

News, updates, and thoughts related to Adobe, developers, and technology.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store