Part 2: Integrating Dynatrace using MuleSoft 4

Ingesting Application Logs

Previously, I have discussed Distributed Traces Integration in Mule 4 into Dynatrace.

In this blog, I will discuss the step-by-step process of ingesting our application logs and how to monitor these in Dynatrace.

Pre-requisites

What we need for ingesting the application logs is the API token that grants the so-called logs.ingest scope through the Access Token screen in Dynatrace.

Also, we need to add the Tracing Module to our Mule application.

<dependency>
   <groupId>org.mule.modules</groupId>
   <artifactId>mule-tracing-module</artifactId>
   <version>1.0.0</version>
   <classifier>mule-plugin</classifier>
</dependency>

Using Tracing Module in Anypoint Studio

There are application contexts that are not yet supplied in the current Mapped Diagnostic Context (MDC), such as the trace ID from the mule-opentelemetry-module.

For log4j2 to recognize the trace ID, we need to set a logging variable called traceId:

It is recommended that we put it after the source component (i.e. HTTP Listener).

In the configuration, we need to extract the trace ID from the traceparent:

To illustrate, a traceparent string might looks like this:

00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

If we split this traceparent value by dash,

1. 00 - is the version;
2. 0af7651916cd43dd8448eb211c80319c - is the trace ID;
3. b7ad6b7169203331 — is the parent ID; and
4. 01 — is the trace flag.

Having said, the DataWeave expression that we need to set in the traceId context variable is:

#[((vars.OTEL_TRACE_CONTEXT.traceparent default '' as String) splitBy  "-")[1]]

💡 Remember vars.OTEL_TRACE_CONTEXT? That is an auto-generated variable of the mule-opentelemetry-module. It should contain the traceparent value.

Apache Log4j2 HTTP Appender

In ingesting application logs, we need to send it using the provided Dynatrace API v2 operation for ingesting logs: /api/v2/logs/ingest.

We also need the generated API key mentioned in the pre-requisite.

Since we need to send an HTTP request, we will be using Apache Log4j2’s HTTP Appender:

<Appenders>
  ...
  <Http name="dynatrace" url="${DYNATRACE_URL}/api/v2/logs/ingest">
    <Property name="Authorization" value="Api-Token ${TOKEN}" />
    <Property name="Content-Type" value="application/json; charset=utf-8" />
           
    <JSONLayout compact="true" eventEol="true" properties="true">
      <KeyValuePair key="service.name" value="${SERVICE_NAME}"/>
      <KeyValuePair key="trace_id" value="$${ctx:traceId}"/>
    </JSONLayout>
  </Http>
</Appenders>

We need to supply the following details:

1. The URL, the Dynatrace API endpoint for Ingest Log
2. Property elements for the HTTP Request headers, such as Authorization and Content-Type
3. We will be also using the JSONLayout, which will contain most of the context from logs. As you can see, we just need to include additional details/context, such as service.name and trace_id.

After setting up the appender, we need to include it in the AsyncRoot element, as an AppenderRef:

<AsyncRoot level="INFO">
  ...
  <AppenderRef ref="dynatrace"/>
</AsyncRoot>

Dyantrace Log Monitoring

So, we have set up everything. What remains is to execute a sample transaction from your proof-of-concept.

A caveat on this — if we want to deploy this proof-of-concept in CloudHub, the appender is considered a custom log appender.

For you to be able to see the logs in the Dynatrace, we need to disable the Cloudhub logs in Runtime Manager.

Having said, if you are using a free trial of Anypoint Platform, you won’t have this option to disable logs.

Checking Logs in Dynatrace

In Dynatrace, under Observe and Explore, click the Logs menu:

You can also dig for more details when we click a specific row in the table:

Sample log of fetching data from a mock user API.

Aside from that, since we have provided the trace ID in our log, we can easily redirect to the Distributed Trace view of this transaction. We need to click the View trace button:

Sample view from the Distributed traces, after clicking the View trace.

And that’s it! That ends the 2-part Dynatrace Integration using MuleSoft. Hopefully, you have learned something from these blogs.

Also, this blog can be beneficial to your customers or clients who plan to migrate and use Dynatrace for monitoring their Mule applications.