Working with File Observers in WAIOps 3.1.1
Authors: Ricardo Olivieri, Muhammad Abubaker
In our previous article, we described how to use REST observers to feed topology information from your environment into IBM Watson AIOps (WAIOps). In this article, we introduce File observers, which can also be used for ingesting topology information. We will walk you through the steps for defining a File observer instance using the WAIOPs console and for defining the input topology file.
Prerequisites
- To get the most out of this article, you should have WAIOPs v3.1.1 successfully installed on your OpenShift cluster (v4.6 or v4.7). You should also know your way around OpenShift.
- You should expose the topology routes in your OpenShift cluster where Watson AIOps is installed. If you haven’t done this, please take a look at the
Expose topology routessection in our previous article. - You should obtain the username and password credentials for the Topology (aka Agile Service Manager) user. You obtain these values by inspecting the
evtmanager-topology-asm-credentialssecret found in the namespace where you installed WAIOPs.
Access OpenAPI documentation for the File Observer
Let’s first access the OpenAPI documentation for the File observer colocated with your WAIOPs v3.1.1 installation. From the OpenShift console, proceed to Networking > Routes and search for the evtmanager-topology-file-observer route. Please take note of the value under the Location column for this route (e.g. https://evtmanager-topology.cp4waiops.apps.capstan.cp.fyre.ibm.com/1.0/file-observer).
To obtain the URL for the File observer OpenAPI page, append swagger to the Location value of your evtmanager-topology-file-observer route (e.g. https://evtmanager-topology.cp4waiops.apps.capstan.cp.fyre.ibm.com/1.0/file-observer/swagger). Navigating to this URL should take you to a page similar to the one captured in the screenshot below:
We recommend you explore the OpenAPI specification for the File observer to get acquainted with the REST API methods that are available. Note that you can also access the OpenAPI specification for the File observer on the official IBM WAIOPs documentation.
Anatomy of input file for File Observer
The main component types that make up an input file for File observer are Vertex V (nodes) and Edges E (connections between nodes). Lines in the file starting with a # character are ignored since they are treated as comments. For more information on the structure of a topology source file, please see Configuring File Observer jobs and Properties.
Create and upload a simple topology file
As an introductory example, we will define a simple topology comprised of one person and one department, where the person manages the department. The example file shown below captures such topology by utilizing two vertices connected by one edge.
# Vertex: components/nodes
V:{"uniqueId":"person1","name":"person1","entityTypes":["person"],"matchTokens":["person1"],"tags":["my_app"]}V:{"uniqueId":"department1","name":"department1","entityTypes":["organization"],"matchTokens":["department1"],"tags":["my_app"]}# Edges: Relationships between vertices:
E:{"_fromUniqueId":"person1","_edgeType":"manages","_toUniqueId":"department1"}
The _edgeType element in the edge component defines the relationship (e.g. manages) between resources in a topology. For a list of all pre-defined edge types supported out-of-the-box, please see Edge types.
Let’s upload to WAIOPs our example topology file using the following HTTP POST invocation:
curl -k -u "<username>:<password>" -X POST --header 'Accept: application/json' https://<release-name>.topology.<namespace>.<domain>/1.0/file-observer/files --form 'job_file=@<topology-file>' --header 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255'where:
release-nameis the name of your deployment (e.g.evtmanager).namespaceis the namespace where WAIOPs was installed (e.g.cp4waiops).domainis the domain name for your cluster.topology-fileis the full path on your local system of the topology file to upload. You should take note of the name of this file (e.g.department.topology) as we will need it while creating a File observer job instance.
For example:
curl -k -u "evtmanager-topology-cp4waiops-user:<password>" -X POST --header 'Accept: application/json' https://evtmanager-topology.cp4waiops.apps.capstan.cp.fyre.ibm.com/1.0/file-observer/files --form 'job_file=@<path>/department.topology' --header 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255'If the upload was successful, you should not see a response from the execution of the command. Please note that if the file already exists, you cannot POST a new version of the file with the same name; attempting to do so throws the following error:
{"_error":{"message":"File already exists.","level":"warning"}}To update the topology file after you’ve uploaded it, you can use the same REST invocation but change the HTTP verb from POST to PUT:
curl -k -u "evtmanager-topology-cp4waiops-user:<password>" -X PUT --header 'Accept: application/json' https://evtmanager-topology.cp4waiops.apps.capstan.cp.fyre.ibm.com/1.0/file-observer/files --form 'job_file=@<path>/department.topology' --header 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255'After updating a topology file, you should see a response from the server similar to this:
["Updated successfully"]
Create and configure a File observer job
After uploading the input topology file, you can create an instance of the File observer. On the WAIOPs console, click Data and tool integrations, proceed to the Advanced tab, click Manage observer jobs, and click the Add a new Job tile. You should then see several observer options to choose from (see screenshots below).
Choose the File observer tile and provide the configuration values shown below: FileObserverTest for Unique ID; for File Name, use the name of the file you uploaded previously (i.e. department.topology); and for Provider, use a name of your liking such as testApp.
Note that to create a File Observer job, you must first upload the topology file (e.g. department.topology) to WAIOPs (as we did in the previous section). Otherwise, there will be an error while creating the File observer job.
Click Save after entering the configuration values. You should then see a tile like the one shown below for your newly created File observer. Right after creating the File observer job, it will start executing and processing the uploaded topology file.
You can see the outcome of its execution by choosing the View history menu option.
Using Topology viewer, you can visualize the topology configuration specified in the input file. You can use the keyword my_app to search for the topology (note that my_app is the tag value we assigned to both elements, person1 and department1, in the topology file).
Clicking on either View topology button reveals the following topology diagram.
By default, File observers only run once, right after creation. To run them again, you have to manually kick off their execution. Having said that, note that you can schedule the execution of a File observer to occur at regular intervals (e.g. every hour).
Create and upload a more complex topology file
Let’s now go over a second example which is more closely aligned to applications and servers. In this example, we have a topology file named application.topology that contains one server, host1, which is connected to four different applications: app1, app2, app3, and app4. Also, we define a relationship between app1 and app4, where app1depends (invokes) on app4.
# Vertex: components/nodes
V:{"uniqueId":"app1","name":"app1","entityTypes":["component"],"matchTokens":["app1"],"tags":["my_awesome_app"]}
V:{"uniqueId":"app2","name":"app2","entityTypes":["component"],"matchTokens":["app2"],"tags":["my_awesome_app"]}
V:{"uniqueId":"app3","name":"app3","entityTypes":["component"],"matchTokens":["app3"],"tags":["my_awesome_app"]}
V:{"uniqueId":"app4","name":"app4","entityTypes":["component"],"matchTokens":["app4"],"tags":["my_awesome_app"]}
V:{"uniqueId":"host1","name":"host1","entityTypes":["host"],"matchTokens":["host1"],"tags":["my_awesome_app"],"connections":"4"}
# Edges: Relationships between vertex:
E:{"_fromUniqueId":"host1","_edgeType":"connectedTo","_toUniqueId":"app1"}
E:{"_fromUniqueId":"host1","_edgeType":"connectedTo","_toUniqueId":"app2"}
E:{"_fromUniqueId":"host1","_edgeType":"connectedTo","_toUniqueId":"app3"}
E:{"_fromUniqueId":"host1","_edgeType":"connectedTo","_toUniqueId":"app4"}
E:{"_fromUniqueId":"app1","_edgeType":"dependsOn","_toUniqueId":"app4"}Using the command below, let’s upload this new topology file to WAIOPs:
curl -k -u "evtmanager-topology-cp4waiops-user:<password>" -X POST --header 'Accept: application/json' https://evtmanager-topology.cp4waiops.apps.capstan.cp.fyre.ibm.com/1.0/file-observer/files --form 'job_file=@<path>/application.topology' --header 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255'Next, we will update the File Name value accordingly in the File Observer definition. Otherwise, the new topology that we just uploaded won’t be processed. On the WAIOPs console, click Data and tool integrations, proceed to the Advanced tab, click Manage observer jobs, and click on the three dots on the top right corner of the FileObserverTest job tile. Select View & edit option and then update the File Name value in the definition.
As before, once the topology definition is processed, we can see the corresponding resources and relationships using Topology viewer (for searching, use the my_awesome_app keyword since that is the tag assigned to the elements of this new topology):
Let’s say that you intended to have uploaded the previous topology having app3 with a critical status. Replace the entry for app3 in the application.topology file with the following:
V:{"uniqueId":"app3","name":"app3","_status": [{"status": "CPU High","state": "open","severity":"critical"}],"entityTypes":["component"],"matchTokens":["app3"],"tags":["my_awesome_app"]}Use then the following HTTP PUT curl command to upload the topology file:
curl -k -u "evtmanager-topology-cp4waiops-user:<password>" -X PUT --header 'Accept: application/json' https://evtmanager-topology.cp4waiops.apps.capstan.cp.fyre.ibm.com/1.0/file-observer/files --form 'job_file=@<path>/application.topology' --header 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255'To process the uploaded file, you should kick off the FileObserverTest job by clicking on the Run button (otherwise the changes won’t be shown on the topology view).
Once the FileObserverTest job finishes running, using Topology viewer, search again using the my_awesome_app keyword.
To remove the critical status from app3, you can use a REST observer (for details on how to define and use REST observers, see our previous article Working with REST Observers in Watson AIOps 3.1.1). Once you have obtained the resource ID for app3, you can use a curl command similar to the following to clear its critical status:
curl -k -u "evtmanager-topology-cp4waiops-user:<password>" -X PUT --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255' --header 'JobId: RESTListenerTest' -d '{
"uniqueId": "app3",
"_status": [
{
"status": "CPU High",
"state": "clear"
}
]
}' 'https://evtmanager-topology.cp4waiops.apps.capstan.cp.fyre.ibm.com/1.0/topology/resources/ISDFYLXkTYWe5F7VFF1RVw'Reload the topology and you should now see that app3 is no longer in a critical state.
Conclusion
In this article, we discussed how to leverage File observers to feed topology information into WAIOPs. We went over several examples for defining resources and relationships between those resources. File observers are an excellent choice for uploading a large topology definition into WAIOPs. As with REST observers, File observers are great too for ingesting topology information when there is no out of the box topology observer. Finally, as shown in the article, after processing a topology file with a File observer, you can use a REST observer to augment (e.g. update the status of an element) the topology.
