Honeycomb
“Always bee Tracing”
In this series of blog posts, we are looking at the following OpenTelemetry backends.
For a more storage alternatives, checkout the Awesome OpenTelemetry repository.
In the Introduction to OpenTelemetry Backends part , we talked about OpenTelemetry and in particular the exporter component of the OTEL collector and why we need it to send traces to backends.
We will showcase how you simultaneously can use OpenTelemetry to send data to different systems/backends with ease. Once the data is stored, we will explore the traces using each vendors UI to visualize and analyze the data.
In this blog post, we will use an existing Kubernetes cluster together with an OpenTelemetry collector and an OpenTelemetry instrumented Golang application, that we used in our OpenTelemetry on Kubernetes post.
Honeycomb
Honeycomb is a fast analysis tool that reveals the truth about how users experience your code in complex and unpredictable environments. Within seconds, find patterns and outliers across billions of rows of data to definitively solve problems.
Sign Up
Honeycomb offers a Free tier, which I’ll use for this demo.
Clicking on Start Free Plan gets you to the Create account page.
Next it asks you to Create Team
Instrument your service with OpenTelemetry
Once the team is created, you are ready to send data to Honeycomb.
Before doing that, we need to have an application that will send the data. If you don’t have one, Honeycomb provides example applications that you can use.
In our case, we will use the same Golang application we used in our OpenTelemetry on Kubernetes post.
If you recall the diagram at the top of this post, we will use an OpenTelemetry collector to export traces to a storage backend, in this case Honeycomb.
Export to Honeycomb from OpenTelemetry Collector
What we are interested in right now, is to know how we can configure our OTLP exporter to send trace data to Honeycomb.
From the documentation, we can read:
You must configure an OTLP exporter, passing in your Honeycomb API Key as a header:
exporters:
otlp:
endpoint: “api.honeycomb.io:443”
headers:
“x-honeycomb-team”: “YOUR_API_KEY”To create an API key, go to Account — Team Settings — Environments and API Keys
Copy the API key and save it somewhere safe, we will soon need it.
Configure OpenTelemetry Collector
We should now have everything we need to configure the exporters to send traces to Honeycomb. To do that, we will add a new exporter (OTLP). In addition to the OTLP exporter, we also use the existing Jaeger exporter and the Logging exporter
Open your otel-collector.yaml file.
Paste in the endpoint and the API key that you created earlier.
In addition to this, we need also need to enable it, which is done by adding it to pipelines in the service section.
Save the file and apply the new changes by running:
kubectl apply -f otel-collector.yaml
After a little while, the pods are using the new version of your configuration.
otel-collector-58bd844bfc-pqftv 1/1 RunningYou can check the logs of the opentelemetry collector by running:
kubectl logs deployment/otel-collectorIf you look closely to the output, you will see that the exporters that we configured (jaeger and otlp) are started:
{"level":"info","ts":1666076795.0685623,"caller":"builder/exporters_builder.go:90","msg":"Exporter is starting...","component_kind":"exporter","component_type":"jaeger","component_name":"jaeger"}
{"level":"info","ts":1666076795.068976,"caller":"builder/exporters_builder.go:95","msg":"Exporter started.","component_kind":"exporter","component_type":"jaeger","component_name":"jaeger"}
{"level":"info","ts":1666076795.0699496,"caller":"builder/exporters_builder.go:90","msg":"Exporter is starting...","component_kind":"exporter","component_type":"otlp","component_name":"otlp"}
{"level":"info","ts":1666076795.07015,"caller":"builder/exporters_builder.go:95","msg":"Exporter started.","component_kind":"exporter","component_type":"otlp","component_name":"otlp"}We are ready to send traces to the collector
Send traces to Honeycomb
Time to send some trace data to our OpenTelemetry collector.
As mentioned earlier, we will use the Golang application that we used in the OpenTelemetry on Kubernetes post.
Remember, that the application access the Kubernetes cluster through a NodePort on port 30080. The Kubernetes service will bind the
4317port used to access the OTLP receiver to port30080on the Kubernetes node.By doing so, it makes it possible for us to access the Collector by using the static address
<node-ip>:30080. In case you are running a local cluster, this will belocalhost:30080.
The application contains an (SDK) instrumented application written in Go, that simulates an application.
go run main.go
2022/10/18 09:09:44 Waiting for connection...
2022/10/18 09:09:44 Doing really hard work (1 / 10)
2022/10/18 09:09:45 Doing really hard work (2 / 10)
2022/10/18 09:09:46 Doing really hard work (3 / 10)
2022/10/18 09:09:47 Doing really hard work (4 / 10)
2022/10/18 09:09:48 Doing really hard work (5 / 10)
2022/10/18 09:09:49 Doing really hard work (6 / 10)
2022/10/18 09:09:50 Doing really hard work (7 / 10)
2022/10/18 09:09:51 Doing really hard work (8 / 10)
2022/10/18 09:09:52 Doing really hard work (9 / 10)
2022/10/18 09:09:53 Doing really hard work (10 / 10)
2022/10/18 09:09:54 Done!Let’s check out the telemetry data generated by our sample application
Exploring with Jaeger
Let’s first explore the data using Jaeger.
Use port-forwarding to access Jaeger UI.
kubectl port-forward svc/simplest-query 16686:16686Open web-browser and go to http://127.0.0.1:16686/
Under Service select test-service to view the generated traces.
Exploring with Honeycomb
Let’s begin exploring Honeycomb.
First we need to login to the UI.
The home page gives you real-time view into what’s happening in your production systems right now. If you click on View pre-built charts in Home it will give you an explanation of each chart. Very handy 👍
Clicking on New Query takes you to the following page.
If you just click on Run Query it will show you the raw data from your application.
It also tells us: You didn’t specify any visualizations! Doing so will visualize your results via graphs.
I didn’t really know what I would want to visualize, so for now I just put in Heatmap (duration_ms), which then resulted in this graph:
If you want to see more detailed information about a trace, just create a new query and click on one of the traces.
And there you have it!
To get a more granular view, we can use Group By and Filter.
The UI provides suggested queries, but I have noticed, that it’s a bonus if you know your way around the SQL syntax.
There is obviously MUCH more to Honeycomb and to get started , and I suggest that you begin with the Quick Start page.
Conclusion
In this post we signed up to Honeycomb. We configured an OpenTelemetry collector to use the OTLP exporter to send data to Honeycomb. We then used an OTEL instrumented application (Golang) to send traces to the collector.
Once the traces were exported, we explored them using the Jaeger UI and the Honeycomb UI.
If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇
