Using the Attunity AEM REST API to send metrics to Elastic Stack

We use Attunity Replicate as our change data capture (CDC) software. We are also big users and fans of the Elastic Stack and use it for all our logging and metrics.

Attunity provide their enterprise manager software (AEM) which provides graphs etc of metrics such as CDC latency, but for us it’s much better to have all our logging and metrics in one place in Elastic. The ability for us to have an overall picture of what’s happening in our entire architecture and use X-Pack features such as alerting and machine learning to warn us of abnormalities that we can resolve before they become issues for our customers is powerful.

This post takes you through how to use the AEM API’s to capture detailed metrics for your Attunity tasks and send this data to the Elastic Stack.

I’m using Python to call the API’s with the following packages.

Attunity Documentation

The AEM setup and user guide is available via the Attunity support portal (registration/support may be required). Section 11 of that document provides full details on all of the AEM REST API’s provided. It’s well worth a read particularly if you want to automate your Attunity operations into a build/deployment pipeline (I feel another blog post coming along!).

Authentication

In order to access the API’s you first need to authenticate and then use the session token returned in the header of your next API calls. The session token last for 5 minutes.

Servers API

In order to get to the low level metrics we need a list of Attunity Replicate tasks. Tasks are assigned to servers and you can have one or more servers in your Attunity deployment all under the control of AEM. Calling the servers API allows you to get a list of the servers in JSON.

Tasks API

With each server returned we can then call the tasks API for each server. This will then give us the task names which we can then call the task details API to obtain the metrics for each task that we want to log to Elastic.

Elastic Stack Setup

I added an index for the data to be logged into. Using some of the data type detection capabilities of Elasticsearch worked well enough for my needs.

Indexing The Data

Next I built a JSON payload with placeholders for the metrics I had obtained from the API and did a find a replace.

The function at the end sends the JSON to Elasticsearch.

Example Data

Here is a sample document from elasticsearch.