Integrating Dynatrace using MuleSoft 4
Application logs are crucial component for monitoring a production-live environment. For DevOps team, it is important that they know what part of the application has failed — so they can act as soon as it appeared.
Not only application logs are for spotting logic errors, but it can be vital to identify possible slowdowns. With the help of tools, such as an Application Performance Monitoring or APM, we can log the important timings of processes in a single transaction.
In this article, I want to discuss how can we integrate Dynatrace, an APM, into logging application events from MuleSoft. Also, we will be using Distributed Tracing for logging events from MuleSoft. Having said, we will be needing an open-source Mule module called: Mule 4 OpenTelemetry Module from AVIO Consulting. This link leads you to more detailed explanation of Distributed Tracing, and the technical specifications of the Module.
Before you dig in in creating your own proof-of-concept, you must have a free trial account in Dynatrace. The account offers a free 15-day trial to most of the functionalities in Dynatrace — including Distributed Traces and Logs monitoring.
Also, we will be using Dynatrace API to ingest artifacts from MuleSoft to Dynatrace.
How to create API Token for authorizing Dynatrace API
Once you created your Dynatrace account, you will be allocated with your own trial environment. Once you have your own trial environment, you are automatically allocated with Dynatrace API.
It is important that before you use any Dynatrace API endpoint, make sure that you have generated an API Token for authorization.
- Go to Dynatrace > Manage > Access Token
2. Click “Generate new token”
3. Type preferred token name
4. Click the “Search scopes…” field.
5. Search for appropriate scopes
6. Tick the checkbox for appropriate scopes
7. Click “Generate token”
8. Copy the generated token by clicking the “Copy” and store it in a vault or password manager
Important note: You must select the scope based on what operations you need in Dynatrace API.
How to ingest Distributed Traces in Dynatrace
For this, you need to include the mule-opentelemetry-module as a maven dependency in your Mule project.
<dependency>
<groupId>com.avioconsulting</groupId>
<artifactId>mule-opentelemetry-module</artifactId>
<version>1.2.0</version>
<classifier>mule-plugin</classifier>
</dependency>Once installed, you should be able to see this module in the project structure:
Next, you should be able to create a global element in your main Mule file or the common configurations Mule file. Search for OpenTelemetry Config:
Once you have added the configuration, you will see different fields. Don’t be overwhelmed, as I will try to discuss the items you need to fill in:
<opentelemetry:config name="OpenTelemetry_Config" doc:name="OpenTelemetry Config" doc:id="87a73a15-2243-4ce5-a302-b201c752e9d2" serviceName="${domain.name}" spanAllProcessors="true" >
<opentelemetry:exporter >
<opentelemetry:otlp-exporter collectorEndpoint="${otlp.collector.endpoint}" protocol="HTTP_PROTOBUF" >
<opentelemetry:headers >
<opentelemetry:header key="Authorization" value="${otlp.collector.headers.authorization}" />
</opentelemetry:headers>
</opentelemetry:otlp-exporter>
</opentelemetry:exporter>
</opentelemetry:config>- serviceName: Usually, the service name is a string that distinguishes the application/domain in the Dynatrace monitor.
- collectorEndpoint: It is a link/endpoint that collects the OTLP traces. In Dynatrace, the endpoint is https://{environment_url}/otlp/v1/traces. Take note that, if we set this up in the configuration, make sure to omit “/traces”, if the protocol is “HTTP_PROTOBUF”.
- We also need to add opentelemetry:header to send the Authorization header in the HTTP Request, when the exporter is sending the request. When you generate an API key in Dynatrace, make sure that the openTelemetryTrace.ingest is included as one of the scopes.
Once you have properly configured the OpenTelemetry Config, when you try sending a request, traces should appear in the Distributed Traces screen, like so:
When you click the hyperlink, Dynatrace will direct you to a page that illustrates the traces and timings of every process in your Mule app.
If you have API-Led Architecture applied in your Mule apps, you can include those traces from the process and system Mule APIs. Currently, you have to pass additional headers when you call downstream APIs:
<http:default-headers >
<http:default-header key="traceparent" value="#[vars.OTEL_TRACE_CONTEXT.traceparent default '' as String]" />
<http:default-header key="tracestate" value="#[vars.OTEL_TRACE_CONTEXT.tracestate default '' as String]" />
</http:default-headers>You can include this snippet in the HTTP Request Configuration of every Mule API that has an OTLP configuration set. What this configuration does — Mulesoft has to include the traceparent and tracestate when we make HTTP requests to MuleSoft downstream APIs.
Mule variable OTEL_TRACE_CONTEXT is a map of the opentelemetry trace context that contains the traceparent and tracestate. Traceparent is important to track the processes and timings of a Mule transaction. It is automatically generated by the mule-opentelemetry-module, in every single transaction.
The headers are automatically recognized by the mule-opentelemetry-module, so we won’t have to do it programmatically. Just make sure that the module has been set up properly in every Mule app.
And with that, you have already created a proof-of-concept Mule project that ingests OpenTelemetry traces in Dynatrace.
In the next blog, I will discuss how ingesting application logs works in Dynatrace.
