Curious About How to Send OpenTelemetry Data to Dynatrace? So Was I!
OpenTelemetry Vendor Neutrality for the Win!
Full disclosure: I’ve recently changed jobs and now work at Dynatrace. As a result, I’ve had to educate myself on how to get OpenTelemetry (OTel) data into Dynatrace (product name == company name). And now that I’ve figured out how to do it myself, I thought it would be useful to share my learnings, in case you’re interested in sending your OTel data to Dynatrace.
Come learn with me, y’all!
NOTE: This post assumes that you are familiar with OTel and the OTel Collector.
OTel is vendor neutral, y’all!
One of the things I love most about OTel is that it is vendor neutral, which means that you can send the same OpenTelemetry data to different vendors. In fact, most of the major Observability vendors out there not only support ingesting OpenTelemetry data, they also actively contribute to the project. Check out the 2023 OpenTelemetry Journey Report for more info.
Tell me why I should care, will ya?
Why does this matter? Well…I used to work at Lightstep (which was acquired by ServiceNow and is now known as ServiceNow Cloud Observability) and many of the OpenTelemetry examples that I played with and blogged about in the last 2 years or so featured sending OTel data to Lightstep. Like this one. And this one.
Now that I work at Dynatrace, which also ingests OTLP natively (yay!), it probably makes sense that I learn how to send OpenTelemetry data to Dynatrace. You know…because it’s kind of part of my job. 😁 And a great way to learn is to try to run my go-to examples using Dynatrace as the Observability backend. Luckily for me, since OTel is vendor neutral, all I had to do was reconfigure my OTel Collector to point to Dynatrace instead of Lightstep to get my examples to work.
NOTE: If you’re evaluating multiple vendors, you can send the same data to different vendors at the same time (à la “vendor bake-off”) to help you determine which vendor best suits your organization’s needs. In case you’re curious, I talk a bit about it here.
Now that we’ve gotten that out of the way, let’s get into the meaty bits.
Prerequisites for sending data to Dynatrace
In order to send OpenTelemetry data to the Dynatrace, you will need two pieces of information:
- Dynatrace instance: Each user (or, more likely, organization) has an instance, also known as a tenant. You need the instance in order to ensure that you are sending your OTel data to the right place.
- Access token: The access token allows you to send OTel data to your Dynatrace instance. It also specifies what kind of data you’re allowed to send to Dynatrace. You can find more on Dynatrace access tokens here.
BUT…before we get to any of that, you must first have a Dynatrace account. If you already have a Dynatrace account, feel free to skip the section below and fast-forward to me showing you how to get your tenant information and create an access token.
Creating a Dynatrace account
If you don’t have a Dynatrace account, the easiest way to get started is to create a free trial account. The free trial is valid for 15 days.
1) Go here, and click on the Free trial button
2) Enter your email, click on the Terms of Use checkbox, and hit Continue
3) Enter the rest of the info, and click Start free trial. Then, sit back, relax, and wait until you get an email confirming that your account is ready.
Once your account is ready, you’ll see a screen that looks like the one below. Click on the Launch Dynatrace button to get started.
This will take you to the Dynatrace login screen.
Your Dynatrace tenant
To find your Dynatrace tenant, log into Dynatrace by going to dynatrace.com, and clicking the Login button on the top right-hand side of the screen.
This will take you to the sign-in page. Once you sign in, take note of the URL. It should look something like this:
https://<your_tenant>.apps.dynatrace.comTake note of the value of <your_tenant>, because we’ll need that later.
Creating a Dynatrace access token
Making sure that you’re logged into Dynatrace, click ctrl+k, and then type access token. Next, select Access Tokens from the top of the search results.
On the Access tokens screen, click Generate new token.
On the Generate new token screen, enter:
- Token name: this can be whatever you want (hopefully something useful 😜)
- Expiration date: totally optional
- Template: select
Kubernetes Data Ingest
You might be wondering why we’re using a template called Kubernetes Data Ingest for our token configuration. The name is a bit misleading, but it happens to have all the token scopes (permissions) that we need to send OpenTelemetry data to Dynatrace:
- Ingest logs (
logs.ingest) - Ingest metrics (
metrics.ingest) - Ingest OpenTelemetry traces (
opentelemetryTrace.ingest)
For more information on these and other Dynatrace token scopes, check out this article.
Once you’re done, click the Generate token button on the bottom of the screen.
The next screen will show your access token. Be sure to copy it before clicking the Done button, because after that, it’s gone forever. If you lose that token information, you’ll have to delete the old one (not necessary, but highly recommended — you know…security), and create a new one.
Configuring the OTel Collector for Dynatrace
Great — we have our tenantand our access token. Now what? Now we can plug this information into our OpenTelemetry Collector.
NOTE: There are two ways to send OTel data to an Observability backend: direct from application, and via the OTel Collector. There’s a time and place for each, and you can check out my blog post on OTel Collector Anti-patterns for more info, if you’d like to learn more.
Your OTel Collector config YAML file would look something like this:
Dynatrace accepts telemetry in OTel’s native OTLP format via HTTP by way of the otlphttp exporter, using the following OTLP endpoint: https://${DT_TENANT}.live.dynatrace.com/api/v2/otlp. The value of ${DT_TENANT}, as you may have guessed, is Dynatrace tenant that you (hopefully) wrote down earlier after you logged in to Dynatrace.
And, as you may have guessed, ${DT_TOKEN} is the value of your Dynatrace access token that you hopefully jotted down and stored in a secrets manager for safekeeping.
Finally,“Dynatrace requires metrics data to be sent with delta temporality and not cumulative temporality”. This means that you’ll need to include the cumulativetodelta processor in:
- Your Collector configuration (line 8)
- Your metrics pipeline (line 27)
REMINDER: Never ever ever store your Dynatrace token and tenant name in plain text. You should instead store them in a secrets vault (e.g. HashiCorp Vault, Google Secret Manager) and pull them from that vault at runtime.
Dynatrace and the OTel Operator
If you’re using the OpenTelemetry Operator to send OpenTelemetry data to Dynatrace, you’ll need to configure your OpenTelemetryCollector resource as follows:
You’ll notice that the spec.config looks the same as what we defined in the otelcol-config.yaml that we saw earlier. The only added thing here is that DT_TOKEN and DT_TENANT are environment variables pulled from a Kubernetes Secret. The secret YAML definition looks like this:
apiVersion: v1
kind: Secret
metadata:
name: otel-collector-secret
namespace: <your_namespace>
data:
DT_TOKEN: <base64_encoded_dynatrace_token>
DT_TENANT: <base64_encoded_dynatrace_tenant>
type: "Opaque"Both DT_TOKEN and DT_TENANT values must be base64 encoded before being added to the secrets YAML. To base64 encode a value and copy the encoded value to your buffer (for easy copy/paste) action:
echo <value_to_encode> | base64Keep in mind that storing secrets in Kubernetes (or storing a secrets YAML in version control, for that matter), isn’t the greatest idea, because base64 does not encrypt your data. You should instead consider using sealed secrets or the Kubernetes Secrets Store CSI Driver + your favourite secrets provider. For more info on these better alternatives check out this article.
The Dynatrace OTel Collector Distribution
Many vendors have their own OTel Collector Distributions. These distributions are curated with Collector components that are specific to that vendor. They may be a combination of vendor-developed custom components and components from Collector Core and Contrib. Using vendor-specific distributions ensures that you are using just the Collector components that you need, reducing overall bloat. For more information, check out this article that I wrote.
All this to say that…you guessed it…Dynatrace also has its own Collector distribution! (Whaaaaat? 😁) This distribution features a set of Collector components for sending Observability data to Dynatrace from various sources. It stays up-to-date with components of the opentelemetry-collector and opentelemetry-collector-contrib repos, which means that you’re not getting some Collector that has deviated from OTel. 😅
Try it out!
Want to try to send data to Dynatrace yourself? Then check out my example repo. I created the repo for a talk on the Target Allocator that I gave at KubeCon in March 2024, and has been updated to include instructions on configuring the OpenTelemetryCollector resource to send OTel data to Dynatrace. It also still has instructions on how to send OTel data to Lightstep, so you can compare configurations for the two different backends.
OTel Data in Dynatrace
And if you’re curious to see what OTel data looks like in Dynatrace, here are some screenshots of what the UI looks like.
I’m not going in-depth on how to navigate the Dynatrace UI because a) you’re probably tired of reading and b) there are already some great videos on the Dynatrace YouTube channel that explain this stuff pretty well, so I encourage you to go there for a more in-depth look, if that tickles your fancy.
Final Thoughts
Overall, I found that getting OpenTelemetry data into Dynatrace was fairly straightforward. My only personal hiccup was in generating the application token, but I got that sorted out, and now I have passed on my knowledge and nauseatingly detailed screenshots on to you. 🤘
I have to say that it’s always fun to use product with fresh eyes, a fresh perspective, and a total newbie’s point of view. There’s nothing quite like it. And, having worked at another Observability vendor before, it’s always fun to see the similarities and differences. It’s like learning a new programming language and comparing it to another one that you already know. What a blast!
One final point that I want to make. I don’t want to trivialize things and give you the impression that moving from one Observability vendor to another is simply a matter of repointing your OTel Collector from one vendor backend to another. That is but one aspect of a vendor migration, no matter what vendor you’re moving to/from. You also have to consider the fact that you likely have a bunch of dashboards, alerts, and whatnot that you created with one vendor. When you migrate to another vendor, things things won’t be a 1:1 translation, so keep that in mind.
But that may be a sacrifice that you’re willing to make, because OTel’s vendor neutrality means that all vendors supporting OpenTelemetry are ingesting the same data. What sets them apart is what they do with the same data. And if one vendor does something with your data better than another one, well, damn, don’t you owe it to yourself to check that out? Food for thought.
And now, I will leave you with a picture the lovely Katie Jr, who seems a little unsure about the cuddles that she’s getting.
Until next time, peace, love, and code. ✌️💜👩💻
