New Relic — OpenTelemetry

A beginners guide to OpenTelemetry backends

In this series of blog posts, we are looking at the following OpenTelemetry backends.

1. Honeycomb
2. Lightstep
3. New Relic
4. Tempo (Grafana Cloud)

For a more storage alternatives, checkout the Awesome OpenTelemetry repository.

In the Introduction to OpenTelemetry Backends part , we talked about OpenTelemetry and in particular the exporter component of the OTEL collector and why we need it to send traces to backends.

We will showcase how you simultaneously can use OpenTelemetry to send data to different systems/backends with ease. Once the data is stored, we will explore the traces using each vendors UI to visualize and analyze the data.

In this blog post, we will use an existing Kubernetes cluster together with an OpenTelemetry collector and an OpenTelemetry instrumented Golang application, that we used in our OpenTelemetry on Kubernetes post.

New Relic

Monitor, debug, and improve your entire stack.

Sign Up

New Relic gives you 16 tools for free and among them APM with Distributed Tracing, which I’ll use in this demo.

Creating an account is easy, just supply a Name and an email address and you are ready to start.

Once you are logged in, you should see the following page:

Right now, it’s all empty, but we will soon send traces here.

Instrument your service with OpenTelemetry

Once the account is created, let’s see how we can send data to New Relic. The New Relic documentation points us to the OpenTelemetry repository on how we can instrument an application.

In our case, we will use the same Golang application we used in our OpenTelemetry on Kubernetes post.

If you recall the diagram at the top of this post, we will use an OpenTelemetry collector to export traces to a storage backend, in this case New Relic.

Export to New Relic from OpenTelemetry Collector

What we are interested in right now, is to know how we can configure our OTLP exporter to send trace data to New Relic.

From the documentation, we can read:

Be ready to configure the OTLP exporter to add a header (api-key). The value is the license key for the New Relic account you want to send data too.

Let’s get this API key. You can find this in your account settings.

Pick the Ingest — License as Key Type. Copy the key as we will soon need it.

Next thing we need is the endpoint, here are the endpoints for US and EU (OTLP). A full list can be found here.

Configure OpenTelemetry Collector

We should now have everything we need to configure the exporters to send traces to New Relic.

To do that, we will add a new exporter (OTLP). In addition to the OTLP exporter, we also use the existing Jaeger exporter and the Logging exporter.

Open your otel-collector.yaml file.

Paste in the endpoint and the API key that you created earlier.

In addition to this, we need also need to enable it, which is done by adding it to pipelines in the service section.

Save the file and apply the new changes by running:

kubectl apply -f otel-collector.yaml

After a little while, the pods are using the new version of your configuration.

otel-collector-58bd844bfc-v78dl    1/1     Running

You can check the logs of the opentelemetry collector by running:

kubectl logs deployment/otel-collector

If you look closely to the output, you will see that the exporters that we configured (jaeger and otlp) are started:

{"level":"info","ts":1666076795.0685623,"caller":"builder/exporters_builder.go:90","msg":"Exporter is starting...","component_kind":"exporter","component_type":"jaeger","component_name":"jaeger"}
{"level":"info","ts":1666076795.068976,"caller":"builder/exporters_builder.go:95","msg":"Exporter started.","component_kind":"exporter","component_type":"jaeger","component_name":"jaeger"}
{"level":"info","ts":1666076795.0699496,"caller":"builder/exporters_builder.go:90","msg":"Exporter is starting...","component_kind":"exporter","component_type":"otlp","component_name":"otlp"}
{"level":"info","ts":1666076795.07015,"caller":"builder/exporters_builder.go:95","msg":"Exporter started.","component_kind":"exporter","component_type":"otlp","component_name":"otlp"}

We are ready to send traces to the collector

Send traces to New Relic

Time to send some trace data to our OpenTelemetry collector.

As mentioned earlier, we will use the Golang application that we used in the OpenTelemetry on Kubernetes post.

Remember, that the application access the Kubernetes cluster through a NodePort on port 30080. The Kubernetes service will bind the 4317 port used to access the OTLP receiver to port 30080 on the Kubernetes node.

By doing so, it makes it possible for us to access the Collector by using the static address <node-ip>:30080. In case you are running a local cluster, this will be localhost:30080.

The application contains an (SDK) instrumented application written in Go, that simulates an application.

go run main.go

2022/10/19 22:35:11 Waiting for connection...
2022/10/19 22:35:11 Doing really hard work (1 / 10)
2022/10/19 22:35:12 Doing really hard work (2 / 10)
2022/10/19 22:35:13 Doing really hard work (3 / 10)
2022/10/19 22:35:14 Doing really hard work (4 / 10)
2022/10/19 22:35:15 Doing really hard work (5 / 10)
2022/10/19 22:35:16 Doing really hard work (6 / 10)
2022/10/19 22:35:17 Doing really hard work (7 / 10)
2022/10/19 22:35:18 Doing really hard work (8 / 10)
2022/10/19 22:35:19 Doing really hard work (9 / 10)
2022/10/19 22:35:20 Doing really hard work (10 / 10)
2022/10/19 22:35:21 Done!

Let’s check out the telemetry data generated by our sample application

Exploring with Jaeger

Let’s first explore the data using Jaeger.

Use port-forwarding to access Jaeger UI.

kubectl port-forward svc/simplest-query 16686:16686

Open web-browser and go to http://127.0.0.1:16686/

Under Service select test-service to view the generated traces.

Exploring with New Relic

Let’s begin exploring New Relic.

First we need to login to the UI.

There you can find your service, in my case it’s called test-service. It also detects that the provider is opentelemetry.

When I click on my service, it automatically jumps to the APM & Services section on the left side.

From there, I click on Distributed Tracing

I can even click further to get more details about the trace.

And there you have it!

Conclusion

In this post we signed up to New Relic. We configured an OpenTelemetry collector to use the OTLP exporter to send data to New Relic. We then used an OTEL instrumented application (Golang) to send traces to the collector.

Once the traces were exported, we explored them using the Jaeger UI and the New Relic UI.

If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇

🚀Join FAUN & get similar stories in your inbox each week