Sending Traces with the Grafana Agent for Grafana Tempo

We will bring a simple example of sending traces to the Grafana Tempo backend.

Goal

To send Trace for the Grafana Tempo through Grafana Agent.

Introduction

Grafana tempo is an open source trace backend developed by Grafana Labs, Tempo can be very powerful for tracing visualization and add well to your observability.

I present a demonstration of sending traces from a python demo application to Grafana Tempo, through the Open Telemetry integrated to the Grafana Agent.

This is the scenario used.

Tecnologies used in this practice

• Python
• Flask lib
• OpenTelemetry lib
• OLTP Protocol
• Grafana Agent
• Grafana Tempo
• Grafana

Here is the demo application that was used.

import re
import os
import logging
from flask import Flask, render_template, request, redirect, url_for, jsonify, make_response
from prometheus_flask_exporter import PrometheusMetrics
from opentelemetry import trace
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import Compression
from opentelemetry.sdk.resources import SERVICE_NAME, Resource

app = Flask(__name__)

#  OpenTelemetry
resource = Resource.create({SERVICE_NAME: "grafana-tempo-demo"})
trace.set_tracer_provider(TracerProvider(resource=resource))

# OTLP Exporter
otlp_exporter = OTLPSpanExporter(endpoint="localhost:4317", insecure="true")
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

# PrometheusMetrics
metrics = PrometheusMetrics(app)
metrics.info('app_info', 'App Information', version='1.0.0')

#logs 
#log formatter
class SpanFormatter(logging.Formatter):
    def format(self, record):
        trace_id = trace.get_current_span().get_span_context().trace_id
        if trace_id == 0:
            record.trace_id = None
        else:
            record.trace_id = "{trace:32x}".format(trace=trace_id)
        return super().format(record)
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
formatter = SpanFormatter('level=%(levelname)s msg=%(message)s TraceID=%(trace_id)s')
handler = logging.FileHandler('app2-teste.log')
handler.setFormatter(formatter)
logger.addHandler(handler)

# Flask Instrumetation 
FlaskInstrumentor().instrument_app(app)

@app.route('/')
def index():
    logger.info("hello-word")
    return "hello-word"


if __name__ == "__main__":
  app.run(host="0.0.0.0", port=5555)

This application send metrics, logs and traces, the traces will exported through oltp port 4317 for the Grafana Agent using Open Telemetry Lib.

The Grafana Agent Config

Install Grafana Agent

We are going to use an Ubuntu Linux in this example.

Import GPC key and Grafana Package

sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg - dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

Update the repositories

sudo apt-get update

Install Grafana Agent

sudo apt-get install grafana-agent

Configure the Grafana Agent

In this configuration we will be using the http protocol to export the traces to Grafana Tempo, the port used will be 4318.
Open telemetry was also configured as a trace receiver by the application.

traces:
  configs:
  - name: default
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: "0.0.0.0:4317"
          http:
            endpoint: "0.0.0.0:4318"
      jaeger:
        protocols:
          grpc: 
    remote_write:
      - endpoint: "http://localhost:4318"
        protocol: "http"
        insecure: true  
    batch:
      timeout: 5s
      send_batch_size: 100

Visualize traces in Grafana

Configure the Datasource for the Grafana Tempo in Grafana.

Here is the traces visualization.

Conclusion

Through this integration we can capture traces from different data sources and send them to Grafana in Tempo, bringing a good view of your application, in addition to the ability to link metrics and logs to traces if configured.

References

Official documentation Grafana Agent : https://grafana.com/docs/agent/latest/
Helm Config Grafana Tempo: https://github.com/grafana/helm-charts/blob/main/charts/tempo/values.yaml