Instana integration using StatsD and Prometheus

Authors: Rajesh K. Jeyapaul, Shivangi Mehra, Neha Ghongade

Problem statement: You wish to integrate any telemetric data with Instana using a customized dashboard, including data from a script.

Scenario: You have a script that fetches some OS level data (apart from the metrics, which Instana Host sensor supports) which you want to monitor using Instana UI.

Solution: The problem can be solved by using Statsd or Prometheus data exporter. Send the scripted data to the Statsd/Prometheus server and integrate it with the Instana agent. The scripted data can now be monitored using a customized dashboard.

Architectural diagram

A. Prerequisites

1. Instana agent is installed and is reporting to your Instana server.
2. Statsd server should be installed on the same VM/Environment where the Instana agent is installed.
3. Prometheus server can be either remote or local.

B. Agent configuration

Instana agents are available for both standalone VMs and cluster environment such as a Kubernetes cluster, or an OpenShift cluster.

• Host level (Standalone)

The following configurations are used to deploy and configure Instana agents in a Standalone VM:

1. Download the agent package. Follow these steps to install the package.
2. After downloading the required deb/rpm package, run the following command to install the agent:

Debian derivatives: sudo apt install ./<Package_name>

Redhat derivatives: rpm -i <Package_name>

For example,

rpm -i instana-agent-static-j9–20211020–1148.x86_64.rpm

Go here for agent configuration.

Notes

• Automatic Installation (One Liner) is not supported for power architecture. You need to install the agents manually.
• To install instana agents on AIX, follow these steps.
• This is applicable for customers who have an Instana license. Those who don’t have an Instana license can register for the trial version.
• Cluster level

For environments with network access, you can use any of the following methods to deploy and configure the Instana agents:

1. Helm — For more information, see helm way agent installation.
2. YAML — For more information, see yaml way agent installation.
3. Operator — For more information, see operator way agent installation.

Notes :

• The image pull from Docker Hub will be at runtime when you apply the helm chart (if the installation is done by using Helm), or apply the yaml (if the installation is done by using YAML or Operator).
• For clusters running in an air-gapped environment, images need to be downloaded manually and pushed to your local registry. You need to update the image path in the yaml file (if the installation is done by using YAML or Operator) or the helm chart (if the installation is done by using Helm).
• Required images for cluster level (air-gapped environments) can be downloaded from docker hub. If the installation is done using Helm or YAML, you need to pull the agents and leader-elector images, and push to your local registry. If the installation is done by using Operator, you need to pull the agents and Operator, and push to your local registry.

C. Configuring Prometheus /Statsd

You can configure agent using either Statsd or Prometheus:

1. Statsd server Installation

• Click here to setup Statsd on your environment.

Architectural diagram

As shown in the above diagram, once you have the script running it fetches some details and sends that data of your script to the Statsd server. Now the Statsd sensor in Instana will start monitoring the data.

1. Your script is running in some CI/CD pipeline, sending data to the Statsd server using the following format

echo “<metric name>:<value>|<metric type>”

e.g.,

`echo “total_virtual_memory:94573|c” | nc -u -w0 127.0.0.1 8125

Here, the total_virtual_memory indicates the name of the metrics followed by value and then type of metrics like counter, gauge, timer, histogram.

2. Configure your Instana agent with Statsd plugin details as mentioned here.

update <agent_install_dir>/etc/instana/configuration.yaml file with below details:

3. On the Instana server you’ll see those metrics listed on the Host dashboard page as shown,

4. Follow the steps to create a dashboard to monitor those specific metrics which we are fetching using script,

• Switch to the main page of Instana Server
• Click on Create Dashboard

• Select Statsd metrics specific to your Host by applying filtering mechanism as shown below.

2. Prometheus server Installation

• Click here to setup Prometheus on your environment

For monitoring Prometheus on Instana, we do not need a Prometheus server to be installed. The sensor directly captures the metrics from the endpoints
exposed by the system.

Architectural diagram

According to the diagram, a Prometheus server (locally or remotely installed) collects data through the endpoint exposed by the monitored system. It is then available for the Prometheus sensor of Instana to automatically detect and monitor this data.

1. Your script is running in some CI/CD pipeline, sending data to Prometheus server using below format

cat <<EOF | curl — data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance
# TYPE <metric name> <metric type>
# HELP <metric name>
<metric name> <metric value>
EOF

e.g.,

cat <<EOF | curl — data-binary @- http://pushgateway.example.org:9091/metrics/job/VirtualMemory/instance/Total_Vir_Mem
# TYPE Total_virtual_memory counter
# HELP Total_virtual_memory
Total_virtual_memory 94573
EOF

Here, the Total_virtual_memory indicates name of the metrics followed by value and type of metrics like counter, gauge, timer, histogram, etc. is mentioned in type section. The Help section includes the description of metrics.

2. To configure Prometheus plugin on your Instana agent, update your YAML file present in <agent_install_dir>/etc/instana/configuration.yaml with the below details.

3. On the Instana server you’ll see the Prometheus dashboard linked on the Host dashboard page and metrics related to Prometheus are listed as shown below,

4. Follow the steps mentioned to create a dashboard to monitor those specific metrics data which we are fetching using the script,

• Switch to the main page of the Instana server
• Click on Create dashboard

• Select the Prometheus metrics specific to your host by applying filtering mechanism as shown below.

Click here to watch demo for above integration.

I hope this has been a helpful exploration of this subject. If you would like to find out more about Instana and its monitoring capabilities please view this document.