Deploying Open Source Observability Stack in AWS — Part — II

In the previous part we have seen the tools available in open source for observability stack which can be deployed in AWS. In this part we will see the open source tool OpenTelemetry to collect application traces and visualize in Grafana.

What is OpenTelemetry ?

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.

Why do you need OpenTelemetry and what it can do ?

Telemetry data is required for observability products to function. There are several best of breed products that specialize in either logs, metrics or traces (aks APM). This best of breed approach leads to vendor lock in. With OpenTelemetry, a single agent is responsible for collection so that IT operations has more flexibility in deciding where this telemetry data has to be routed to be analyzed. For example in the next article, we would be routing telemetry data to Elastic stack with zero code changes and without involvement of any Engineering/Development team being involved.

You can use OpenTelemetry to :

• Per language, a single vendor-independent instrumentation library that supports both automatic and manual instrumentation.
• A single collection binary that can be used as an agent or gateway in a variety of ways.
• An end-to-end solution for creating, emitting, collecting, processing, and exporting telemetry data.
• Configuration allows you to transmit data to numerous destinations in parallel, giving you complete control over your data.
• Data collection that is vendor-agnostic thanks to open-standard semantic norms.
• The ability to support several context propagation formats in parallel to aid with standard migration.

What OpenTelemetry does not include ?

Like Jaeger and Prometheus, OpenTelemetry is not an observability backend. Instead, it allows data to be exported to a number of open-source and commercial back-ends. It has a pluggable architecture that makes it simple to integrate new technological protocols and formats.

Demo Environment Setup

Now lets see a demo to configure opentelmetry in java applications and view our traces in grafana.Have created a demo application and you can access it here.

It’s worth noting that the software itself has no observability features. It doesn’t log anything (apart from what Spring Boot logs), doesn’t write metrics, traces, or anything else. The Spring Boot Actuator was not even turned on.

Configuring the Collector

This configuration has four parts:

1. The receivers section controls the protocols and formats of traces that can be received. This example only enables support for Jaeger Thrift over compact Thrift protocol, but any OpenTelemetry trace receiver is supported.
2. The processors section adds the batch processor, which batches up data before uploading to help with performance and compression. This is a best practice.
3. The exporters section adds the otlp (OpenTelemetry Protocol) exporter, which is the gRPC-based protocol to be used with Grafana Cloud. The endpoint is the url for your Grafana Cloud stack. The headers section adds the HTTP Authorization header in Basic format. We will come back to the base64-encoded credentials in a bit.
4. The service section brings the previous three sections together by building the final pipeline for trace data.

The Authorization header is configured with Basic scheme, which is a base64-encoded version of the text <user>:<password>. There are a multitude of ways to create this value, including commands and web pages, but here is a simple example command:

$ echo -n “<your user id>:<your api key>” | base64

Gather your Grafana Cloud URL and Tempo credentials from the My Account page. To see your username and URL, choose the stack and click Send Traces. Security -> API Keys is where you maintain your API keys.

Modify your configuration file.

Go ahead and open the app at this point:

$ docker-compose up –build

And then point your browser at http://localhost/v1/orders. You should get the order details.You may have to wait for all the containers to spin up, so wait a few seconds and retry if you get an error.

→ Configuring java application to add tracing automatically

We use the -javaagent option to attach an instrumentation agent to the JVM and set some system parameters that inform the agent where to send trace spans to add automatic tracing to the programme.The agent JAR was acquired from the opentelemetry-java-instrumentation tags page.You’ll almost certainly want to use the most recent and greatest.

[“java”,“-Dapplication.name=${APP_NAME}“, \

“-Dotel.traces.exporter=otlp”, \

“-Dotel.exporter.otlp.traces.endpoint=http://otel-collector:4317”, \

“-Dotel.service.name=opentelemetry-demo”, \

“-Dotel.metrics.exporter=none”, \

“-javaagent:/opentelemetry-javaagent-all.jar”,\

“-jar”,“/app/app.jar”]

What’s going on in the background? The agent is familiar with a large number of popular Java libraries and frameworks. The agent injects tracing instrumentation into specific areas via bytecode injection during classloader. Automatic tracing is now available in the app.

Let’s see the result

The Docker compose includes a otel-collector container. This otel-collector will collect the traces from our java application and send traces to Grafana.

We launch the application

Here we browse application traces in Grafana.

We will make a few requests to the API and see the corresponding traces in Grafana.

To summarize, we instrumented an application with OpenTelemetry collector and collected application traces, these applications were sent to Grafana for visualization.

In the next part of the article, we will be sending OpenTelemetry traces to Elastic and Kibana for visualization.

We’re a proud Opensearch partner. Read all about our Opensearch practice.

Case Study: OpenSearch for Behavioural Analytics