File Observers in CP4AIOps v4.6

Authors: Ricardo Olivieri, Muhammad Abubaker, Tanush Sharanarthi

In our previous article, we described how to use REST observers to feed topology information from your environment into IBM CP4AIOps. In this article, we introduce File observers, which can also be used for ingesting topology information. We will walk you through the steps for defining a File observer instance using the CP4AIOps console and for defining the input topology file.

Pre-requisites

• To get the most out of this article, you should have CP4AIOps v4.6 successfully installed on your OpenShift cluster. You should also know your way around OpenShift.
• You should expose the topology routes in your OpenShift cluster where CP4AIOps is installed. If you haven’t done this, please take a look at the Expose topology routes section in our previous article.
• You should obtain the username and password credentials for the Topology (aka Agile Service Manager) user. You obtain these values by inspecting the aiops-topology-asm-credentials secret found in the namespace where you installed CP4AIOps.

Access OpenAPI documentation for File observer

Let’s first access the OpenAPI documentation for File observer colocated with your CP4AIOps v4.6 installation. From the OpenShift console, proceed to Networking > Routes and search for the aiops-topology-file-observer route. Please take note of the value under the Location column for this route (e.g. https://aiops-topology-file-observer-cp4aiops.apps.o2-151312.cp.fyre.ibm.com/1.0/file-observer).

To obtain the URL for the File observer OpenAPI page, append swagger to the Location value of your aiops-topology-file-observer route (e.g. https://aiops-topology-file-observer-cp4aiops.apps.o2-151312.cp.fyre.ibm.com/1.0/file-observer/swagger). Navigating to this URL should take you to a page similar to the one captured in the screenshot below:

We recommend you explore the OpenAPI specification for File observer to get acquainted with the REST API methods that are available. Note that you can also access the OpenAPI specification for File observer on the official IBM CP4AIOps documentation.

Anatomy of input file for File observer

The main component types that make up an input file for File observer are Vertex V (nodes) and Edges E (connections between nodes). Lines in the file starting with a # character are ignored since they are treated as comments. For more information on the structure of a topology source file, please see Configuring File Observer jobs and Properties.

Create and upload a simple topology file

As an introductory example, we will define a simple topology comprised of one person and one department, where the person manages the department. The example file shown below captures such topology by utilizing two vertices connected by one edge.

# Vertex: components/nodes
V:{"uniqueId":"person1","name":"person1","entityTypes":["person"],"matchTokens":["person1"],"tags":["my_app"]}
V:{"uniqueId":"department1","name":"department1","entityTypes":["organization"],"matchTokens":["department1"],"tags":["my_app"]}
# Edges: Relationships between vertices:
E:{"_fromUniqueId":"person1","_edgeType":"manages","_toUniqueId":"department1"}

The _edgeType element in the edge component defines the relationship (e.g. manages) between resources in a topology. For a list of all pre-defined edge types supported out-of-the-box, please see Edge types.

Let’s upload to CP4AIOps our example topology file using the following HTTP POST invocation:

curl -k -X 'POST' -u "<username>:<password>" --header 'Accept: application/json' --url 'https://aiops-topology-file-observer-<namespace>.<domain>/1.0/file-observer/files' -H 'accept: application/json'  -H 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255' -H 'Content-Type: multipart/form-data'  -F 'job_file=@/<path>/department.txt;type=text/plain'

where:

• namespace is the namespace where CP4AIOps was installed (e.g. cp4aiops).
• domain is the domain name for your cluster.
• topology-file is the full path on your local system of the topology file to upload. You should take note of the name of this file (e.g. department.txt) as we will need it while creating a File observer job instance.

For example:

curl -k -X 'POST' --url 'https://aiops-topology-file-observer-cp4aiops.apps.o2-151312.cp.fyre.ibm.com/1.0/file-observer/files' -u "aiops-topology-cp4aiops-user:<password>" -H 'accept: application/json'  -H 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255' -H 'Content-Type: multipart/form-data'  -F 'job_file=@/<path>/department.txt;type=text/plain'

If the upload was successful, you should not see a response from the execution of the command. Please note that if the file already exists, you cannot POST a new version of the file with the same name; attempting to do so throws the following error:

{"_error":{"message":"File already exists.","level":"warning"},"messageBody":{"message":"File already exists.","level":"warning"}}

To update the topology file after you’ve uploaded it, you can use the same REST invocation but change the HTTP verb from POST to PUT:

curl -k -X 'PUT' --url 'https://aiops-topology-file-observer-cp4aiops.apps.o2-151312.cp.fyre.ibm.com/1.0/file-observer/files' -u "aiops-topology-cp4aiops-user:<password>" -H 'accept: application/json'  -H 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255' -H 'Content-Type: multipart/form-data'  -F 'job_file=@/<path>/department.txt;type=text/plain'

After updating a topology file, you should see a response from the server similar to this:

["Updated successfully"]

Create and configure a File observer job

After uploading the input topology file, you can create an instance of File observer. On the CP4AIOps console, click Integrations, click on Add integration button on the upper right corner. Proceed to the Topology category, click the Observer jobs link, and click the Add a new job button. You should then see several observer options to choose from (see screenshots below).

Choose the File observer tile and provide the configuration values shown below: FileObserverTest for Unique ID; for File Mode choose Use server file option ; for File Name, use the name of the file you uploaded previously (i.e. department.txt); and for Provider, use a name of your liking such as testApp.

Note that to create the above File observer job, you should first upload the topology file (e.g. department.txt) to CP4AIOps (as we did in the previous section). Otherwise, there will be an error while creating the File observer job.

Click Save after entering the configuration values. You should then see a tile like the one shown below for your newly created File observer. Right after creating a File observer job, it will start executing and processing the uploaded topology file.

You can see the outcome of its execution by choosing the View history menu option.

If a File observer job fails its execution, the Details column shows a potential reason. One of the most common reasons for failures is providing a wrong file name during the configuration of a File observer job.

In CP4AIOps v4.6, you can also upload a topology file by drag-and-drop functionality while defining a new File observer job. Please note that the extension of the topology file must be a.txt.

View the uploaded topology

Using Resource management section of the CP4AIOps console, you can visualize the topology configuration specified in the input file. You can use the keyword my_app to search for the topology (note that my_app is the tag value we assigned to both elements, person1 and department1, in the topology file).

Clicking on either of the resources available would reveal the following topology diagram.

By default, File observers only run once, right after creation. To run them again, you have to manually kick off their execution. Having said that, note that you can schedule the execution of a File observer job to occur at regular intervals (e.g. every hour).

Create and upload a more complex topology file

Let’s now go over a second example which is more closely aligned to applications and servers. In this example, we have a topology file named application.txt that contains one server host1, which is connected to four different applications: app1, app2, app3, and app4. Also, we define a relationship between app1 and app4, where app1depends on (invokes) app4.

# Vertex: components/nodes
V:{"uniqueId":"app1","name":"app1","entityTypes":["component"],"matchTokens":["app1"],"tags":["my_awesome_app"]}
V:{"uniqueId":"app2","name":"app2","entityTypes":["component"],"matchTokens":["app2"],"tags":["my_awesome_app"]}
V:{"uniqueId":"app3","name":"app3","entityTypes":["component"],"matchTokens":["app3"],"tags":["my_awesome_app"]}
V:{"uniqueId":"app4","name":"app4","entityTypes":["component"],"matchTokens":["app4"],"tags":["my_awesome_app"]}
V:{"uniqueId":"host1","name":"host1","entityTypes":["host"],"matchTokens":["host1"],"tags":["my_awesome_app"],"connections":"4"}
# Edges: Relationships between vertex:
E:{"_fromUniqueId":"host1","_edgeType":"connectedTo","_toUniqueId":"app1"}
E:{"_fromUniqueId":"host1","_edgeType":"connectedTo","_toUniqueId":"app2"}
E:{"_fromUniqueId":"host1","_edgeType":"connectedTo","_toUniqueId":"app3"}
E:{"_fromUniqueId":"host1","_edgeType":"connectedTo","_toUniqueId":"app4"}
E:{"_fromUniqueId":"app1","_edgeType":"dependsOn","_toUniqueId":"app4"}

Using the command below, let’s upload this new topology file to CP4AIOps:

curl -k  -X 'POST' --url 'https://aiops-topology-file-observer-cp4aiops.apps.o2-151312.cp.fyre.ibm.com/1.0/file-observer/files' -u "aiops-topology-cp4aiops-user:<password>" -H 'accept: application/json'  -H 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255' -H 'Content-Type: multipart/form-data'  -F 'job_file=@/<path>/application.txt;type=text/plain'

Next, we will update the File Name value accordingly in the File observer definition. Otherwise, the new topology that we just uploaded won’t be processed. On the CP4AIOps console, Click on the three dots on the top right corner of the FileObserverTest job tile we created. Select View & edit option and then update the File Name value in the definition.

As before, click Run, and once the topology definition is processed, we can see the corresponding resources and relationships using Resource management(for searching, use the my_awesome_app keyword since that is the tag assigned to the elements of this new topology):

Let’s say that you intended to have uploaded the previous topology having app3 with a critical status. Replace the entry for app3 in the application.txt file with the following:

V:{"uniqueId":"app3","name":"app3","_status": [{"status": "CPU High","state": "open","severity":"critical"}],"entityTypes":["component"],"matchTokens":["app3"],"tags":["my_awesome_app"]}

Use then the following HTTP PUT curl command to upload the topology file:

curl -k -X 'PUT' --url 'https://aiops-topology-file-observer-cp4aiops.apps.o2-151312.cp.fyre.ibm.com/1.0/file-observer/files' -u "aiops-topology-cp4aiops-user:<password>" -H 'accept: application/json'  -H 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255' -H 'Content-Type: multipart/form-data'  -F 'job_file=@/<path>/application.txt;type=text/plain'

To process the uploaded file, you should kick off the FileObserverTest job by clicking on the Run button (otherwise the changes won’t be shown on the topology view).

Once the FileObserverTest job finishes running, using Resource management, search again using the my_awesome_app keyword.

To remove the critical status from app3, replace the entry for app3 in the application.txt file with the following:

V:{"uniqueId":"app3","name":"app3","_status": [{"status": "CPU High","state": "clear","severity":"clear"}],"entityTypes":["component"],"matchTokens":["app3"],"tags":["my_awesome_app"]}

Use then the following HTTP PUT curl command to upload the topology file and make sure that a response[“Updated successfully”] is returned back:

curl -k -X 'PUT' --url 'https://aiops-topology-file-observer-cp4aiops.apps.o2-151312.cp.fyre.ibm.com/1.0/file-observer/files' -u "aiops-topology-cp4aiops-user:<password>" -H 'accept: application/json'  -H 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255' -H 'Content-Type: multipart/form-data'  -F 'job_file=@/<path>/application.txt;type=text/plain'

Run the FileObserverTest again for changes to take affect on node app3.

Reload the topology and you should now see that app3 is no longer in a critical state.

Conclusion

In this article, we discussed how to leverage File observers to feed topology information into CP4AIOps. We went over several examples for defining resources and relationships between those resources. File observers are an excellent choice for uploading a large topology definition into CP4AIOps. File observers are great for ingesting topology information when there is no out-of-the-box topology observer. Finally, as shown in the article, after processing a topology file with a File observer, you can also augment (e.g. update the state of an element in) the topology.