Sending Traces with the Grafana Agent for Grafana Tempo
We will bring a simple example of sending traces to the Grafana Tempo backend.
Goal
To send Trace for the Grafana Tempo through Grafana Agent.
Introduction
Grafana tempo is an open source trace backend developed by Grafana Labs, Tempo can be very powerful for tracing visualization and add well to your observability.
I present a demonstration of sending traces from a python demo application to Grafana Tempo, through the Open Telemetry integrated to the Grafana Agent.
This is the scenario used.
Tecnologies used in this practice
- Python
- Flask lib
- OpenTelemetry lib
- OLTP Protocol
- Grafana Agent
- Grafana Tempo
- Grafana
Here is the demo application that was used.
import re
import os
import logging
from flask import Flask, render_template, request, redirect, url_for, jsonify, make_response
from prometheus_flask_exporter import PrometheusMetrics
from opentelemetry import trace
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import Compression
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
app = Flask(__name__)
# OpenTelemetry
resource = Resource.create({SERVICE_NAME: "grafana-tempo-demo"})
trace.set_tracer_provider(TracerProvider(resource=resource))
# OTLP Exporter
otlp_exporter = OTLPSpanExporter(endpoint="localhost:4317", insecure="true")
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)
# PrometheusMetrics
metrics = PrometheusMetrics(app)
metrics.info('app_info', 'App Information', version='1.0.0')
#logs
#log formatter
class SpanFormatter(logging.Formatter):
def format(self, record):
trace_id = trace.get_current_span().get_span_context().trace_id
if trace_id == 0:
record.trace_id = None
else:
record.trace_id = "{trace:32x}".format(trace=trace_id)
return super().format(record)
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
formatter = SpanFormatter('level=%(levelname)s msg=%(message)s TraceID=%(trace_id)s')
handler = logging.FileHandler('app2-teste.log')
handler.setFormatter(formatter)
logger.addHandler(handler)
# Flask Instrumetation
FlaskInstrumentor().instrument_app(app)
@app.route('/')
def index():
logger.info("hello-word")
return "hello-word"
if __name__ == "__main__":
app.run(host="0.0.0.0", port=5555)
This application send metrics, logs and traces, the traces will exported through oltp port 4317 for the Grafana Agent using Open Telemetry Lib.
The Grafana Agent Config
Install Grafana Agent
We are going to use an Ubuntu Linux in this example.
Import GPC key and Grafana Package
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg - dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
Update the repositories
sudo apt-get update
Install Grafana Agent
sudo apt-get install grafana-agent
Configure the Grafana Agent
In this configuration we will be using the http protocol to export the traces to Grafana Tempo, the port used will be 4318.
Open telemetry was also configured as a trace receiver by the application.
traces:
configs:
- name: default
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
jaeger:
protocols:
grpc:
remote_write:
- endpoint: "http://localhost:4318"
protocol: "http"
insecure: true
batch:
timeout: 5s
send_batch_size: 100
Visualize traces in Grafana
Configure the Datasource for the Grafana Tempo in Grafana.
Here is the traces visualization.
Conclusion
Through this integration we can capture traces from different data sources and send them to Grafana in Tempo, bringing a good view of your application, in addition to the ability to link metrics and logs to traces if configured.
References
Official documentation Grafana Agent : https://grafana.com/docs/agent/latest/
Helm Config Grafana Tempo: https://github.com/grafana/helm-charts/blob/main/charts/tempo/values.yaml