Observability for CA Endevor with the Elastic (ELK) Stack
What is “Observability”?
The term observability has been gaining traction with the rise of Site Reliability Engineering as a practice but what does it mean exactly? Observability refers to the ability to infer the internal state of a system by examining its outputs. In the context of software systems, that often means log and event data, runtime metrics like resource usage, and so on. Typically, observability platforms provide you with the ability to transform, aggregate and store these outputs so you can perform analytics on them.
What are the benefits of doing this?
By applying observability to a system, you can look for patterns or indicators of problems in the system. You can also establish baseline behaviors to better understand what constitutes normal behavior and what is an anomaly. If you do see problems, you can easily search the aggregated data to get indicators that will help you track down the root cause of problems. Let’s look at a concrete example. Your shop is trying to control their MIPS consumption and you’d like to know if any spikes in developer build activity occur. By adding observability to a system like CA Endevor, you can detect this and alert on threshold breaches in order to see if there is a problem. If you see a pattern of repeated spikes in the number of builds, you may decide that you want to limit foreground execution and control the number of concurrent builds that can be executed to better control MIPS usage.
How can I add observability to CA Endevor?
To get started, you will need to choose a platform. The guide below will use the Elastic stack, sometimes referred to as the ELK stack. ELK stands for (E)lastic (L)ogstash (K)ibana, which are all components needed for the platform. Elastic is a database where events and data can be aggregated. Logstash is a tool for collecting system outputs, optionally transforming them and then storing them in the Elastic database. Finally, Kibana is a platform for building visualizations and dashboards from the data. There are other platforms and systems out there such as Splunk but the ELK stack is one that is open source (with commercial options available) and widely used.
Once you have your platform, you need to set it up to gather the data from CA Endevor. There are a number of ways you can do this and you may want to gather multiple outputs depending on your use case. If you are collecting CA Endevor SMF records, you can gather them using Zowe CLI and Logstash. Using IFASMFDP and Zowe’s batch commands, you can dump SMF log data and then fetch the outputs to feed into Logstash. Another option might be to scrape data from the Endevor REST API. For this introduction, we are going to use webhooks to capture all the events happening in Endevor in real time via the Logstash http input plugin.
Getting started
The first step will be to deploy the ELK stack. As with any enterprise grade system, installing and configuring software such as the ELK stack can be very involved. For the purpose of this introductory article, however, we will focus only on getting the various components installed with minimal configuration so that you can play and get inspired as to how you might integrate Endevor to your organization’s observability platform. You may also find it easier to opt for a trial hosted version at elastic.co or to leverage a containerized instance.
Hint: ensure you will have no firewalls or other network issues between your Endevor instance and the ELK install. This likely rules out using laptops — you will want to set this up on a virtual or other server behind your firewall that you have access to.
To begin, you must download all the software from the Elastic site. You need to download Elasticsearch, Kibana and Logstash. Once downloaded, I created a parent directory and simply extracted each component into their own directory under the parent.
In each component directory, you will see a “bin” folder containing scripts to run and perform other operations. The main script to start the component is simply the script named after the component (so for instance for Kibana, you will find bin/kibana.bat for Windows or bin/kibana for Linux installations).
If you want to change ELK component ports or network bindings, you can edit the *.yml for the component name in the “config” or “conf” subdirectories. For my install, I only needed to update the Kibana.yml so that the server host was the DNS name or IP of the server on which I installed all the components instead of the default “localhost” so that it would be accessible from outside the machine:
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.# The default is ‘localhost’, which usually means remote machines will not be able to connect.# To allow connections from remote users, set this parameter to a non-loopback address.server.host: “my.server.fully.qualified.host.name”
Apart from that, I simply started Elasticsearch using the script, followed by Kibana and verified I could connect using the Kibana URL: http://my.server.fully.qualified.host.name:5601/
Logstash requires a bit more configuration. In particular, Logstash uses various plugins, filters and other components to capture data from your system. You need to specify in your config/logstash.conf file which input plugins are active, which filters you want (if any) and where to output the results. The configuration file I used looks like this:
# Sample Logstash configuration
input {
http {
host => "0.0.0.0"
port => 31311
}
}
filter {
mutate {
rename => { "actiondate" => "date" }
rename => { "actiontime" => "time" }
}
truncate{
fields => "time"
length_bytes => 6
}
mutate {
add_field => { "eventdatetime" => "%{date}%{time}" }
}
date {
match => [ "eventdatetime", "yyyyMMddHHmmss" ]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
}
}Let’s break it down to understand what this does. The first section, “input”, indicates what plugins you want to gather input from. In my case, I set up the http input plugin which sets up a listener on the specified port and address on the logstash server (0.0.0.0 means bind to any NIC on the server). This listener is going to be the end point I post webhooks to.
The next section, “filter”, describes the filters I want applied to transform the incoming webhook data. I use the mutate plugin to rename and standardize the date/time and actiondate/actiontime on the package and element events. I also shave off the milliseconds part of the event time using the truncate plugin. I then concatenate the event date & time to a new field called eventdatetime using mutate and finally use the date plugin to set the new field to the @timestamp field for the captured data. @timestamp is a standard ELK field that indicates the date/time an event occurs.
The final section, “output”, says to take the transformed captured webhook data and store it in the elasticsearch database at the specified host and port. In this case it’s on the same machine and I’m using the default port of 9200.
You should be able to cut & paste the above to a config/logstash.conf file which you will need to create. Afterwards, save your config/logstash.conf file and then start logstash using the bin/logstash script as follows (Note: this assumes you are in the bin directory where the startup scripts are):
logstash -f ../config/logstash.confAt this point, you are ready to start sending CA Endevor data.
Setting up CA Endevor Webhooks
There are two main steps for setting up CA Endevor to send webhook data. The first is to define the event logging to a USS directory. Logs can be organized by event type, system, etc. and once that is defined, then as events happen in CA Endevor, small files containing event data that will serve as webhook payloads will be written to the file system. To set up Endevor to log events, create the PARMLIB member for the SCL statements. Within the PARMLIB member, code one Package Logging statement, or one Element Logging statement, or both. For example:
PACKage LOGging ON
TO PATH
u/users/endevor/user5/pkglog .ELEment LOGging ON
TO PATh
/u/users/endevor/user08/elmlog
FROm
ENV QA .
Of course, you may want to adjust to filter logging for only certain systems or events and if so you need to adjust the SCL. A detailed description on how to set this up can be found here in the CA Endevor documentation under the “Activity Logging for Webhook Support” topic.
Note: if you are not the Endevor administrator, you will likely need to enlist them to set this up. It may also already be set up in which case you simply need to ask them to provide the directories where events are being logged.
With the PARMLIB member set up, you next need to specify the name of the PARMLIB member as the value for the C1DEFLTS parameter, E2ELOGMBR. You should also create the USS directories associated with the SCL to store the log files (in the example, this is u/users/endevor/user5/pkglog and /u/users/endevor/user08/elmlog). Ensure that the Alternate ID is enabled and can write to the USS directories. If the Alternate ID is enabled, then the log files are written under the security context of the Alternate ID. If the Alternate ID is not enabled, the log files are written under the security context of the user ID of the person who is performing the action. In either case, the ID must have the proper authority to be able to write to this directory. Note: event logging uses the permission value that is specified by the ENABLE_ALTID_USS_SECURITY option in the Option Features Table “ENCOPTBL”. Where this option is not enabled, the default permission value for End-to-End logging files is “700”.
The next step is to set up the Mainframe Webhook Server component to watch for event files at the various locations you are logging them. As files are written, they will be read, posted as a webhook to any defined end-point URLs and finally the files will be deleted. The webhook server is a Java *.war or web archive file that is meant to be deployed to an application server like Tomcat. If you are running Endevor web services, you can deploy to the same Tomcat server the web services are deployed on. This is as simple as copying the mfwebhookserver.war file from the install media to the <TOMCAT_HOME>/webapps directory. One thing that should be noted though is that the event logs will be written to the USS directories in EBCDIC. As such, your JCL for running Tomcat should be adjusted so that the following Java option is included in the start up command:
-Dmf.webhook.file.encoding=CP1140Detailed instructions on deploying the webhook server can be found here in the CA Endevor documentation under the “Mainframe Webhook Server” topic.
Note: again, if you are not the Endevor administrator, you will likely need to enlist them to set this up. It may also already be set up in which case you simply need to ask them to provide the URL for the mainframe webhook server so you can configure the webhooks.
Once you’ve deployed the webhook server, you can go to the URL of your tomcat, with the mfwebhookserver context root (e.g. https://my.tomcat.host.name:port/mfwebhookserver) and create a new hook:
Make sure to specify the directory you are logging events to in Endevor in the “Watched Directory” field and the URL of the logstash listener in the “Payload URL” field. Check the activate box and click “OK”. Now events being logged by CA Endevor will be posted to the ELK stack and you can begin to add visualizations, alerts and get insights into your Endevor installation.
Creating a Simple Visualization
The first step to creating a visualization is to create an Index Pattern for your data so that it becomes visible to Kibana. This is done via the “Stack Management” menu item in the ELK web interface (http://my.server.fully.qualified.host.name:5601/) after you click the menu icon on the top left. If this is the first time, accessing the interface, you may see a prompt to set up some initial data. If you see this, click “Explore on my own”. Once you click the menu icon and select “Stack Management” from the list you will be in the Stack Management page. Click “Index Patterns” from the list of sections on the left (under the Kibana heading). Now click “Create Index Pattern”. For the name, enter logstash* and click “Next Step”. On the next page, select @timestamp in the “Time field” drop down and then click “Create Index Pattern”. At this point your data will be visible to Kibana.
Now if you go back to the Kibana home page (click the Elastic Icon on the top left and then click the Kibana tile to go to the homepage), you can create a dashboard by clicking on the “Dashboard” tile. Then click the “Create Dashboard” button and “Create Visualization” on the following screen. You should see your index pattern on the left, along with fields that were detected from the various webhook payloads. Let’s say you are working with package webhook data. A first visualization you could create is the breakdown of package activity over various timespans. This could be done by dragging the @timestamp field to the “Horizontal Axis” target on the right and then the # Records field to the “Vertical Axis” target and finally the action.keyword to the “Break down by” target. Now you should have something like this:
Of course, this is just a starting point. You can add other panels to the Dashboard and other visualizations. You can save the Dashboards for easy retrieval and create thresholds and alerts. From here, I recommend experimenting with the visualizations and seeing what use cases you can come up with.
Summary
CA Endevor, like any other system in your tool chain contains a large amount of valuable data that can help you get insights into your DevOps processes. Tools like the Elastic stack can be invaluable for analyzing this data and realizing important insights. With the possibilities afforded by Zowe, the CA Endevor REST API and webhooks, you can easily integrate Endevor into these tools as detailed above.
Learn more about Zowe at this site; more Zowe blogs here.
If you have any questions or feedback or would like to talk about any of your CA Endevor use cases, please feel free to contact me at Vaughn.Marshall@broadcom.com or leave a comment below.
