Updated Instana plug-in for OpenTelemetry

Gaozhenz

Published in

IBM Cloud

4 min readMay 15, 2024

Co-Authors

Zhen Gao (gaozhenz@cn.ibm.com) — Developer for Instana OTEL

Zhi Li (bjlizhi@cn.ibm.com) — Developer for Instana OTEL

Rui Liu (liurui@cn.ibm.com) — Tech lead for Instana OTEL

OpenTelemetry-based data collectors work with prebuilt generic sensors in Instana to inject the monitoring data.

The enhancements on the Instana OpenTelemetry include:

Improved handling and display of complex metrics with histograms: Histograms are used to show complex metrics. Compared to other types of diagrams, histograms have several advantages. Histograms such as bar charts can present highs, lows, concentrations, and trends within the data to indicate common value ranges or concentrations within specific intervals. Histograms help to swiftly identify outliers or anomalies within the data, as they might appear as extreme values in the distribution.
Attributes of metrics are retained: All data types are retained with different dimensions.
Better equipped to manage the dependencies of the apps: The generic sensor and entity plug-in for OpenTelemetry can handle the dependencies of every app that runs in containers.

Design

Histograms:

Both DELTA mode and CUMULATIVE mode are supported.
The real-time average value is displayed in the built-in dashboard.

Note: If data goes through otlp-acceptor at the backend, the real-time average value for CUMULATIVE mode is lost as the date is not supported on the backend.

The _max and _min data points are displayed.

Attributes:

All OpenTelemetry attributes for data points are retained.
Resource attributes for service.name, pid, container.id, host.name are retained as tags. All other tags for resource attributes are discarded because too many tags might severely decrease the user experience of custom dashboards.

Dependency with entities:

For in-container apps, the container.id parameter is identified and dependency is established with the corresponding container entity;
The dependency between the Generic OpenTelemetry entity and the OTelProcess entity is optimized.

Sample payload of generic OpenTelemetry plug-in

A sample payload of the Generic OpenTelemetry plug-in is shown in the following snippet:

{
  "pid": "1234",
  "remote": "28322@d4d7c18968f34007a04b815b27582c62",
  "resource":{
    "service.name":"my-service",
    "pid": "1234",
    "hostname": "dk-liunoone.fyre.ibm.com"
    "container.id": "90202a241f1bc13f96068735ddf4ccbd7b8af7c44c9e2210d835a44c58f39eb4",
    "sdk.opentelemetry.type": "sensor-sdk:dameng"
  },
  "metrics": {
    "sums": {
      "sdk.logs/db.client.connections.idle.min{pool.name=\"HikariPool-1\"}": 10
    },
    "gauges": {
      "hikaricp/queueSize{logRecordProcessorType=\"BatchLogRecordProcessor\",}": 0
    },
    "histograms": {
      "hikaricp/db.client.connections.create_time_count{pool.name=\"HikariPool-1\", }": 59,
      "hikaricp/db.client.connections.create_time_sum{pool.name=\"HikariPool-1\", }": 55.000000,
      "hikaricp/db.client.connections.create_time_min{pool.name=\"HikariPool-1\", }": 0.000000,
      "hikaricp/db.client.connections.create_time_max{pool.name=\"HikariPool-1\", }": 8.000000,
      "hikaricp/db.client.connections.create_time_mean{pool.name=\"HikariPool-1\", }": 3.800000,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"0.0\",}":23.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"5.0\",}":35.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"10.0\",}":1.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"25.0\",}":0.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"50.0\",}":0.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"75.0\",}":0.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"100.0\",}":0.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"250.0\",}":0.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"500.0\",}":0.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"750.0\",}":0.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"1000.0\",}":0.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"2500.0\",}":0.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"5000.0\",}":0.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"7500.0\",}":0.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"10000.0\",}":0.0,
      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"+Inf\",}":0.0
    }        
  }
}

Deploying OpenTelemetry Demo in OCP

To deploy OpenTelemetry Demo in OCP, complete the following steps:

Log on to OpenShift CLI (oc).
Create the namespace otel-demo:
kubectl create namespace otel-demo
kubectl project otel-demo
Install the otel-demo application by using Helm:
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
Install helm chart with the release name my-otel-demo:
helm install my-otel-demo open-telemetry/opentelemetry-demo
Expose services using kubectl port-forward:
kubectl port-forward svc/my-otel-demo-frontendproxy 8080:8080
Create a route for services for my-otel-demo:
my-otel-demo-frontendproxy
my-otel-demo-grafana
my-otel-demo-jaeger-query

A route that creates my-otel-demo-frontendproxy is shown in the following example:

Deploying Instana agent on OCP

To deploy Instana agent on Red Hat OpenShift Container Platform:

helm install instana-agent \
   --repo https://agents.instana.io/helm \
   --namespace instana-agent \
   --create-namespace \
   --set openshift=true \
   --set agent.key=*************** \
   --set agent.downloadKey=*********** \
   --set agent.endpointHost=ingress-magenta-saas.instana.rocks \
   --set agent.endpointPort=443 \
   --set cluster.name='cluster101' \
   --set zone.name='zone101' \
   instana-agent