ASM tutorial: Learning to troubleshoot a REST API connection
This blog is a tutorial providing a step-by-step interaction to troubleshoot a REST API connection. While this blog describes the REST-based interaction between an ASM (Agile Service Manager) Observer and a VMWARE vCenter, most of the tests were performed using curl. It is general enough for any REST interaction.
ASM vCenter Observer.
ASM provides Out of the Box integration to the managed system through its Observer. One such Observer is the vCenter Observer. The vCenter Observer interacts with the vCenter through its REST interface to obtain the required topology information. This blog’s tests are done on vCenter version 6.5, which is one of the supported versions by ASM.
The issue.
One problem with working with the REST interface-based Observer, through the GUI, is that we might not see an immediate error as we run the Observer. The observer execution may be completed successfully, and yet no vCenter component topology is displayed in ASM. The observer log file does not provide us with enough information to tell what is wrong, so we need to interact with the vCenter REST interface manually to troubleshoot.
The tool.
We can use many tools to talk to a REST interface; some tools provide a nice Graphical User Interface. This blog uses curl, as it is widely available in a Linux platform.
The troubleshooting Steps.
Check the log files.
Let us start with the observer log. Checking the observer log files gives the following warning message:
WARN [2021–03–19 00:39:15,344] [cfd95b7e-3bc7–4006-a4a8-a73a79c71255:LAB-vCenter/VCenterHostTask] c.i.i.t.o.v.j.r.t.VCenterHostTask — Could not obtain Host Info data via RESTThe observer job exited after encountering those WARNING, which explained why there is no vCenter information in ASM.
Ensure that the REST interface is available.
We need to ensure that the vCenter responded to a REST API call; we can start by querying the base address. We can find the base address by reading the vCenter documentation. It usually is at https://[hostname or ipaddress]/rest.
$ curl -sik -H ‘Accept:application/json’ -X GET https://[vcenter-address]/restThe flag:
-s (for silent), is to exclude progress meter. This is useful for concise output.
-i this is to include the HTTP response headers in the output. As we will see later, the response header can help in debugging.
-k (or — insecure) allows curl to accept a self-signed SSL server connection.
-H (or — header) for specifying the header of the query. We want a JSON formatted output.
-X (or — request) this is where we specify our request.
The server responded with the following:
If we run the above command, we may get the following output:
HTTP/1.1 200 OK
Date: Fri, 19 Mar 2021 00:56:24 GMT
Content-Type: application/json
Transfer-Encoding: chunked
{
“value”: {
“components”: {
“metadata”: {
“method”: “GET”,
“href”: “https://[vCenter Address]/rest/com/vmware/vapi/metadata/metamodel/service/operation/id:com.vmware.vapi.rest.navigation.component/id:list"
},
“method”: “GET”,
“href”: “https://[vCenter Address]/rest/com/vmware/vapi/rest/navigation/component”
},
“resources”: {
“metadata”: {
“method”: “GET”,
“href”: “https://[vCenter Address]/rest/com/vmware/vapi/metadata/metamodel/service/operation/id:com.vmware.vapi.rest.navigation.resource/id:list"
},
“method”: “GET”,
“href”: “https://[vCenter Address]/rest/com/vmware/vapi/rest/navigation/resource”
}
}
}With this, we know that the Server’s REST Interface is working fine.
The endpoint
As we are troubleshooting an ASM discovery, we needed to know the endpoint that ASM was querying. One way to do this is to force an error by specifying a wrong username and password, to get the following message in the log file.
ERROR [2021–03–19 00:36:31,005] [cfd95b7e-3bc7–4006-a4a8-a73a79c71255:CSPLAB-vCenter/VCenterHostTask] c.i.i.t.o.v.j.r.t.VCenterHostTask — Could not gain access to VMware VCenter endpoint ‘https://[vcenter address]/rest/vcenter/host’ — check your keyStore path in job parameter. Reason:- An VMware VCenter session-id Token could not be obtained, check if the credential are correct.Cannot deserialize instance of `java.lang.String` out of START_OBJECT token
Just what we wanted. The endpoint was: `https://[vcenter-address]/rest/vcenter/host`. We can check the VMWare REST documentation and verified that, indeed, this was a valid endpoint.
Verify without Authorization.
We need to specify some user information for Authentication/Authorization before we can query in more detail. To verify, let us try without any user information first.
$ curl -sik -H ‘Accept:application/json’ -X GET https://[vcenter-address]/rest/vcenter/hostHTTP/1.1 401 Unauthorized
Date: Fri, 19 Mar 2021 01:05:00 GMT
Content-Type: application/json
Transfer-Encoding: chunked
{
“type”: “com.vmware.vapi.std.errors.unauthenticated”,
“value”: {
“messages”: [
{
“args”: [],
“default_message”: “This method requires authentication.”,
“id”: “vapi.method.authentication.required”
}
]
}
}
As expected, we received This method requires authentication response. Note the header. It said the same thing: 401 — Unauthorized.
Get the session ID.
For vCenter and other REST providers, the authorization is normally a two steps process. First, we specify our username and password to get a Session-ID, then use the Session-ID in the query.
$ curl -X POST — insecure — header ‘Content-Type: application/json’ — header ‘Accept: application/json’ — header ‘vmware-use-header-authn: test’ — header ‘vmware-api-session-id: null’ -u ‘username:password’ ‘https://[vcenter-address]/rest/com/vmware/cis/session'
The server responded with the session-ID:
{“value”:”d2de956a5f0e072aa546e91fcf70f3dc”}
Use the Session ID
We run the same query, but this time, we added the session-ID.
$ curl -sik -H ‘Accept:application/json’ -H “vmware-api-session-id:d2de956a5f0e072aa546e91fcf70f3dc” -X GET https://[vcenter-address]/rest/vcenter/host
And we may get the following response.
HTTP/1.1 500 Server Error
Date: Fri, 19 Mar 2021 00:51:24 GMT
Content-Type: application/json
Transfer-Encoding: chunked
{
“type”: “com.vmware.vapi.std.errors.internal_server_error”,
“value”: {
“messages”: [
{
“args”: [
“com.vmware.vapi.std.errors.Unauthorized”
],
“default_message”: “Provider method implementation threw unexpected exception: com.vmware.vapi.std.errors.Unauthorized”,
“id”: “vapi.bindings.method.impl.unexpected”
},
{
“args”: [],
“default_message”: “Not authorized to perform this operation.”,
“id”: “com.vmware.api.vcenter.unauthorized”
}
]
}
}
Yes! We just found out about the issue! The Session-ID and hence the username and password did not have enough privilege to run the query.
Note that, unlike the query response without providing any username, we now have a server error response header, and the message was also different.
An authorized user, please.
With the above information, we can go back to the vCenter admin. Asked for a credential with a proper permission, and rerun the query with the the authorized user, and here is the query and the response, it is a good and valid response:
$curl -sik -H ‘Accept:application/json’ -H “vmware-api-session-id:292a717da90093ccfb46ec66e8789e65” -X GET https://[vcenter-address]/rest/vcenter/host
HTTP/1.1 200 OK
Date: Fri, 19 Mar 2021 01:52:51 GMT
Content-Type: application/json
Transfer-Encoding: chunked
{
“value”: [
{
“host”: “host-151”,
“name”: “172.16.3.133”,
“connection_state”: “CONNECTED”,
“power_state”: “POWERED_ON”
}, . . . — cut for brevity — . . . {
“host”: “host-89”,
“name”: “172.16.3.124”,
“connection_state”: “CONNECTED”,
“power_state”: “POWERED_ON”
}
]
}
Let there be light, or an ASM topology.
We know now that our manual query works. We can go back to ASM, rerun the vCenter Observer job; lo and behold; the topology is now displayed:
Summary
This blog shows an example of using curl to troubleshoot a REST Interaction specifically for vCenter, including using a Session ID generated by the Interface after providing a correct user name and password.
I posted this blog initially on an internal IBM technical site. The content is general enough for anybody that needs to troubleshoot a REST Interface connection, so I rewrote and posted it here.
