How to Develop and Deploy a Webhook Alert Action App with Custom Payload and Headers for Splunk

Ankit Tyagi

Follow

Jun 29, 2020 · 6 min read

Splunk is a wonderful tool with loads of features, yet a few of them are not available out-of-the-box — one of those missing features is the customizable webhook alert action.

While working on a project, I needed to send some custom metrics along with some headers from Splunk alerts and ended up exploring the default available webhook alert action.

Here’s what’s already available

Splunk, by default, provides webhook alert action for an alert; however, it is lacking some of the required features like:

• Authentication mechanism
• Custom header support
• Custom payload support

With the above features being missing from the default webhook alert action on Splunk, it becomes much less useful because the webhooks mostly require some kind of authentication to receive the data and similarly, customized payload might be a requirement for some webhook APIs.

Splunk has done a great job with its pluggable platform, which allows you to write your own apps and deploy them to serve your custom requirements. Despite this, building a custom app can be tricky and time-consuming.

After reviewing the webhook alert action I decided to develop my own custom webhook alert action with all the required missing features.

customWebhook app to the rescue

I developed a customWebhook alert action application with the following configurable options:

URL: Target webhook URL

e.g. https://server.com/api/v2/webhook

Headers (dict): Include any number of headers

e.g. {‘Authorization’: ‘Bearer <token>’, ‘Content-Type’: ‘application/json’…}

Payload (dict): Payload data with key-value pairs of your choice

e.g. {‘service_name’: ‘MyService’, ‘key1’: ‘value1’, ‘key2’: ‘value2’…}

I’ll walk you through easy and simple steps that would give you a kick start with your Splunk project.

Objectives:

1. Understand the Splunk app directory structure

2. Components with the respective directory structure

3. Configuration and logic code snippet

4. Deploy the application on the Splunk server

5. Create an alert with customWebhook alert action

Understand the Splunk app directory structure

Let’s break it down.

This is an exhaustive list of configuration I used in the customWebhook app, The complete list is here: https://docs.splunk.com/Documentation/Splunk/8.0.4/AdvancedDev/ModAlertsCreate

Configuration and logic code snippet

alert_actions.conf

# Configuration file path#$SPLUNK_HOME$/etc/apps/customWebhook_app/default/alert_actions.conf[customWebhook]is_custom = 1label = customWebhookdescription = Send CustomWebhook alert notificationsicon_path = customWebhook_icon.pngpayload_format = jsonparam.alert_source = Splunkparam.user_agent = Splunk/$server.guid$

app.conf

[ui]is_visible = 0label = customWebhook alerts[launcher]author = Ankit Tyagidescription = Send alert payload to custom webhookversion = 1.0[install]state = enabledis_configured = 1

restmap.conf

[validation:savedsearch]action.customWebhook = case('action.customWebhook' != "1", null(), 'action.customWebhook.param.base_url' == "action.customWebhook.param.base_url" OR 'action.customWebhook.param.base_url' == "", "No Webhook URL specified",  1==1, null())action.customWebhook.param.base_url = validate( match('action.customWebhook.param.url', "^https?://[^\s]+$"), "Webhook URL is invalid")

default.meta

[alert_actions/customWebhook]export = system[alerts]export = system[restmap]export = system

customWebhook.html

<!--$SPLUNK_HOME$/etc/apps/customWebhook_app/default/data/ui/alerts/customWebhook.html-->

<form class="form-horizontal form-complex">
    <div class="control-group">
        <label class="control-label" for="customWebhook_base_url">URL</label>

        <div class="controls">
            <input type="text" class="input-xlarge" name="action.customWebhook.param.base_url" id="customWebhook_base_url" placeholder="https://server.com/api/v2/webhook/" />
            <br>
            <span class="help-block">
                Webhook URL (str) https://server.com/api/v2/webhook
            </span>
        </div>
    </div>
    <div class="control-group">
        <label class="control-label" for="customWebhook_headers">Headers</label>

        <div class="controls">
            <input type="text" name="action.customWebhook.param.headers" id="customWebhook_headers" placeholder="{'Content-Type': 'application/json', 'CustomHeader': 'Value'}"/>
            <br>
            <span class="help-block">
                Headers (dict) <br>
                e.g. {'Content-Type': 'application/json', 'CustomHeader': 'Value'}
            </span>
        </div>
    </div>
    <div class="control-group">
        <label class="control-label" for="customWebhook_payload">Payload</label>

        <div class="controls">
            <input type="text" name="action.customWebhook.param.payload" id="customWebhook_payload" placeholder="{'key1': 'value1', 'key2': 'value2'}"/>
            <br>
            <span class="help-block">
           Payload data (dict)
                e.g. {'key1': 'value1', 'key2': 'value2'} <br>
                add the key from seach result like this. <br>
                e.g. {'key1': 'value1', 'key2': '$result.key_name$'}
            </span>
        </div>
    </div>
</form>

alert_action.conf.spec

#$SPLUNK_HOME$/etc/apps/customWebhook_app/README/alert_actions.conf.spec[customWebhook]param.alert_source = "Splunk"

customWebhook.py

##################################################################   A splunk plugin developed for sending alerts from Splunk#   to a webhook endpoint with custom payload#   Developed by:#       Ankit Tyagi#       SRE - Adobe################################################################## File location on Splunk server:
#$SPLUNK_HOME$/etc/apps/customWebhook_app/bin/customWebhook.pyfrom __future__ import print_functionfrom future import standard_librarystandard_library.install_aliases()import sys, jsonimport requestsdef process_args(*args):    return json.dumps(args)def send_alert_data(*args):    args = json.loads(process_args(*args))    for config in args:      if 'headers' in config:        headers = config.get('headers')        headers = headers.replace("'", '"')        headers = json.loads(headers)      if 'payload' in config:        data = config.get('payload')        data = data.replace("'", '"')      if 'base_url' in config:        url = config.get('base_url')      if 'alert_source' in config:          alert_source = config.get('alert_source')          data = json.loads(data)          data['alert_source'] = alert_source          data = json.dumps(data)    try:        res = requests.post(url=url, data=data, headers=headers)        print("Server response: %s" % json.dumps(res.json()), file=sys.stderr)        return 200 <= res.status_code < 300    except requests.exceptions.RequestException as e:        print("ERROR Error sending message: %s" % e, file=sys.stderr)        return False
if __name__ == "__main__":    if len(sys.argv) > 1 and sys.argv[1] == "--execute":        payload = json.loads(sys.stdin.read())        if not send_alert_data(payload.get('configuration')):            print("Failed trying to send alert notification", file=sys.stderr)            sys.exit(2)        else:            print("INFO alert notification successfully sent", file=sys.stderr)    else:        print("FATAL Unsupported execution mode (expected --execute flag)", file=sys.stderr)        sys.exit(1)

Deploy the application on the Splunk server

Bundle the application components and files in a directory with the name: alert_<app_name> here we’d use alert_customWebhook_app as the name of the directory.

Splunk admin need to deploy the directory alert_customWebhook_app in $SPLUNK_HOME$/etc/apps/ on Splunk server, so finally we’d have, $SPLUNK_HOME$/etc/apps/alert_customWebhook_app

Create an alert with customWebhook alert action

1. Go to search and put the search criteria and click the search button.

Note: you may want to define some custom variables to load into the alert, for that use Splunk’s ‘eval’. For e.g. in the screenshot below.

2. Save the search as an alert

3. Set up the alert configurations. Not all the options are mandatory, however, so change them accordingly. For example, I used cron scheduler, which does a search based on the search criteria I selected while creating this alert and compares the result with the threshold as shown in the next figure.

4. Set up the trigger condition

5. Action time

Now add the desired action to get notified about this alert. Select the action, customWebhook.

6. After clicking on customWebhook alert action, a configuration page opens up

Put in the webhook URL, Headers, and Payload information.

Headers and Payload are dictionary type.

The evaluated variables with ‘eval’ can be used to send the custom data in the payload, to do that add the variable like this:

{‘service_name’: ‘$result.service_name$’}

Save the alert

7. To further edit the alert or add new actions, go to alerts from the menu and click on your alert.

What’s next?

This customWebhook app is capable of sending alerts custom data on a single URL. Next, I’ll be working on enhancing it to be able to send alerts data on multiple URLs at the same time. Please leave your suggestions in the comments.

References:

https://docs.splunk.com/Documentation/Splunk/8.0.4/AdvancedDev/ModAlertsCreate

https://docs.splunk.com/Documentation/Splunk/8.0.4/AdvancedDev/ModAlertsIntro