Lightstep
A beginners guide to OpenTelemetry backends
In this series of blog posts, we are looking at the following OpenTelemetry backends.
For a more storage alternatives, checkout the Awesome OpenTelemetry repository.
In the Introduction to OpenTelemetry Backends part , we talked about OpenTelemetry and in particular the exporter component of the OTEL collector and why we need it to send traces to backends.
We will showcase how you simultaneously can use OpenTelemetry to send data to different systems/backends with ease. Once the data is stored, we will explore the traces using each vendors UI to visualize and analyze the data.
In this blog post, we will use an existing Kubernetes cluster together with an OpenTelemetry collector and an OpenTelemetry instrumented Golang application, that we used in our OpenTelemetry on Kubernetes post.
Lightstep
Monitoring, observability, and incident response for the world’s most reliable systems
Connect a service
Once the account is created, you will be asked to connect your service to Lightstep.
Exploring the Sandbox
If you don’t have service yet, you can still explore Lighstep using a sandbox, which includes an interactive guided demo with two different scenarios:
- Investigate a Metric Spike
- Debug an Error
Using the Sandbox, gives you a quick way to get the feeling experience of the product. Since it’s interactive, you can debug an iOS error or resolve a performance regression.
We will exit the Sandbox and go back to the first page.
Instrument your service with OpenTelemetry
Once the team is created, you are ready to send data to Lightstep.
Before doing that, we need to have an application that will send the data.
When you click on Connect Your First Service, you will get a guide on how to do that.
In our case, we will use the same Golang application we used in our OpenTelemetry on Kubernetes post.
If you recall the diagram at the top of this post, we will use an OpenTelemetry collector to export traces to a storage backend, in this case Lightstep.
Export to Lightstep from OpenTelemetry Collector
The Lighstep website provides good documentation how to use the OpenTelemetry Collector to send data to Lightstep Observability. It even gives us an example how to configure the OpenTelemetry collector file like this:
exporters:
otlp/ls:
endpoint: ingest.lightstep.com:443
headers:
"lightstep-access-token": "${LIGHTSTEP_ACCESS_TOKEN}"From this we can see that we need to create an ACCESS_TOKEN. This can be done in the Project Settings page:
Copy the API key as we will need it in the next step.
Configure OpenTelemetry Collector
We should now have everything we need to configure the exporters to send traces to Lightstep.
Open your otel-collector.yaml file. In the exporters section, next to the existing jaeger exporter, add the otlp/ls exporter. Paste in the endpoint and the API key that you created earlier.
In addition to this, we need also need to enable it, which is done by adding it to pipelines in the service section.
Save the file and apply the new changes by running:
kubectl apply -f otel-collector.yaml
After a little while, the pods are using the new version of your configuration.
You can check the logs of the opentelemetry collector by running:
kubectl logs deployment/otel-collectorIf you look closely to the output, you will see that the exporters that we added (otlp/ls) is started:
{"level":"info","ts":1666114207.1594775,"caller":"builder/exporters_builder.go:90","msg":"Exporter is starting...","component_kind":"exporter","component_type":"otlp","component_name":"otlp/ls"}
{"level":"info","ts":1666114207.1594837,"caller":"builder/exporters_builder.go:95","msg":"Exporter started.","component_kind":"exporter","component_type":"otlp","component_name":"otlp/ls"}We are ready to send traces to the collector
Send traces to Lightstep
Time to send some trace data to our OpenTelemetry collector.
As mentioned earlier, we will use the Golang application that we used in the OpenTelemetry on Kubernetes post.
Remember, that the application access the Kubernetes cluster through a NodePort on port 30080. The Kubernetes service will bind the
4317port used to access the OTLP receiver to port30080on the Kubernetes node.By doing so, it makes it possible for us to access the Collector by using the static address
<node-ip>:30080. In case you are running a local cluster, this will belocalhost:30080.
The application contains an (SDK) instrumented application written in Go, that simulates an application.
go run main.go
2022/10/18 20:17:26 Waiting for connection...
2022/10/18 20:17:26 Doing really hard work (1 / 10)
2022/10/18 20:17:27 Doing really hard work (2 / 10)
2022/10/18 20:17:28 Doing really hard work (3 / 10)
2022/10/18 20:17:29 Doing really hard work (4 / 10)
2022/10/18 20:17:30 Doing really hard work (5 / 10)
2022/10/18 20:17:31 Doing really hard work (6 / 10)
2022/10/18 20:17:32 Doing really hard work (7 / 10)
2022/10/18 20:17:33 Doing really hard work (8 / 10)
2022/10/18 20:17:34 Doing really hard work (9 / 10)
2022/10/18 20:17:35 Doing really hard work (10 / 10)
2022/10/18 20:17:36 Done!Let’s check out the telemetry data generated by our sample application
Exploring with Jaeger
Let’s first explore the data using Jaeger.
Use port-forwarding to access Jaeger UI.
kubectl port-forward svc/simplest-query 16686:16686Open web-browser and go to http://127.0.0.1:16686/
Under Service select test-service to view the generated traces.
Exploring with Lightstep
Let’s begin exploring Lighstep. First we need to login to the UI
On the left-hand side, I see that my service (test-service) is selected, which means that data was sent from my application to Lighstep.
Clicking on the Operations tab opens up the Explorer to the Latency Histogram. It directly asks me if I want to start a tutorial to learn more about how to compare latency, which I finds very useful.
At the bottom of the screen, you will find the traces that were sent from our application.
Clicking further on one of the traces gives you details about the trace. Again, it asks if you want to start a tutorial to learn more how to read the data. Super useful!
Here, it explains how to use the UI and how you can understand the data.
And there you have it!
Conclusion
In this post we signed up to Lightstep. We configured an OpenTelemetry collector to use the OTLP exporter to send data to Lightstep. We then used an OTEL instrumented application (Golang) to send traces to the collector.
Once the traces were exported, we explored them using the Jaeger UI and the Lighstep UI.
If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇
