Send z/OS Performance Data to Splunk via the SYSVIEW REST API

In this blog, we will discuss integrating SYSVIEW on z/OS with Splunk on the distributed side via a python program, for the purposes of this article, we are calling sysviewConnect. sysviewConnect leverages the SYSVIEW REST API to send z/OS metrics, alerts, and on-demand performance data to Splunk. The details discussed here are specific to Splunk. However, they can be applied to other data analytics engines, such as Elasticsearch, Logstash, and Kibana (ELK), and Apache Superset.

With the introduction of the SYSVIEW REST API in SYSVIEW 16.0 and in the age of data visualizations and analytics, z/OS performance metrics and alerts are now accessible to any platform outside of z/OS where TCP/IP is available. This allows the SYSVIEW REST API to become the portal that connects your z/OS LPARs to the data visualization and analytics engines of your choosing; in order to gain insight into your enterprise performance data.

Overview

The integration between SYSVIEW and Splunk is broken into three distinct components:

1. Data source — SYSVIEW is the data source that provides performance data for the vast z/OS components and its subsystems: CICS, IMS, MQ, and DB2, as well as TCPIP, JVM, USS, DATACOM, and IDMS.
2. Data sink — Splunk is the data sink that will receive and store SYSVIEW z/OS performance data for analysis and visualization purposes.
3. Data transport — sysviewConnect is the intermediary component that uses the SYSVIEW REST API to request z/OS performance data from SYSVIEW and send it to Splunk.

SYSVIEW integration with Splunk

Note: The sysviewConnect configuration files, illustrated above,are optional to control the metrics retrieved and sent, as well as the data sinks that will receive the SYSVIEW data. 

Data source — SYSVIEW

Prerequisites:

• SYSVIEW 16.0 with PTF SO14533
• SYSVIEW REST API configured and deployed

Details:

Any SYSVIEW XDI (eXternal Data Interface) eligible command can be used as the source to provide data to Splunk. XDI provides foundational support for the REST API.

To identify XDI eligible commands, issue primary command COMMANDS to list all SYSVIEW commands. Eligible commands will have XDI in the XDI column. At this time, 977(or 93%) of SYSVIEW commands are XDI eligible.

Identify XDI eligible commands

Data sink — Splunk

Prerequisites:

• Splunk with HTTP Event Collector (HEC)

Details:

In order to allow Splunk to receive data from SYSVIEW via the REST API, it must be configured with the HEC component. For details on how to setup and enable HEC, see Set up and use HTTP Event Collector in Splunk Web.

In our example, we configured HEC with the following token attributes:

• Name: alerts
• Token value: choose a secure token
• Source Type: _json
• Index: index_alerts
• Status: enabled

Splunk HEC Token

Data transport — sysviewConnect

Prerequisites:

• Python 3.8.2 or later

Details:

sysviewConnect is a sample python application that behaves as a pass-through layer between SYSVIEW and Splunk.

sysviewConnect is responsible for:

• getData() — Invoking [on an interval] the SYSVIEW REST API to execute the desired XDI commands
• formatData()— Manipulating the JSON results of the command execution to make it compatible with Splunk
• sendData()— Sending the formatted JSON data to Splunk

Note: The sample code provided performs minimal input validation and error handling. This is intentional to keep the sample short and concise. In practice, it is encouraged to add input parameter validation and error handling.

getData() — Invoking the SYSVIEW REST API

This method is responsible for invoking the SYSVIEW REST API with the user provided SYSVIEW XDI eligible command string. In this example, we are executing the ALERTS command to retrieve the MVS system data collection exception alerts.

• restAPIUrl — The URL to your SYSVIEW REST API instance. Example: https://lpar.yourcompany.com:40080
• displayPath — REST API command endpoint.
• ssid — The SYSVIEW instance SSID where the command is to be executed.
• sysviewCommand — XDI eligible command to be executed and whose data to be sent to Splunk. This can be a complete SYSVIEW command string with parameters, sorts, selects, and even stacked commands.
• username and password — Your SYSVIEW mainframe credentials.

The method returns null, or, a string containing the result of the command execution in JSON format.

formatData() — Data Manipulation

The data returned by getData()is in the format detailed here. Before the data fields can be sent to Splunk, they must be converted to a key:value pair format where each data row contains a sequence of field-name:field-value pairs.

sendData() — Sending the data to Splunk

This method is responsible for sending the data formatted by formatData() to Splunk.

• splunkUrl — The URL to your Splunk instance with the HEC endpoint suffix.
• authToken — The HEC authentication token. This is the token value specified when the HEC event collector was created.
• splunkIndex — The Splunk index associated with the HEC event collector.

Viewing the SYSVIEW data under Splunk

On the Splunk side, search under index index_alerts to view the ALERTS. The displayed Splunk events are the data rows resulted from executing the ALERTS command.

With the proper Splunk expertise, users can run reports, create dashboards, set up alerts, render graphs, and so on.

Sample-1 Viewing the raw ALERTs data

Sample-2 Visualizing the ALERTs for a given metric

Next steps

You can take sysviewConnect to the next level by executing the main code on n minute intervals to continuously send SYSVIEW data to Splunk.

Note: SYSVIEW data collection metrics are collected on a user specified interval x. Running sysviewConnect on an interval < x will result in duplicate data getting sent to Splunk.

In summary, we have demonstrated how users can utilize the SYSVIEW REST API to send z/OS metrics, alerts, and performance data to Splunk using a python program we called sysviewConnect.

For additional information about the SYSVIEW REST API, visit our technical documentation.

Happy coding!