Integrating Datadog Log Alerts with MuleSoft Applications

Overview: Integrating Datadog log alerts with MuleSoft applications enhances monitoring and error detection capabilities, providing real-time insights into application health and performance.

1. Log in to Datadog

• Access your Datadog account.

2. Navigate to Log Explorer

• Go to Logs > Explorer in the Datadog dashboard.

3. Construct Your Search Query

• Use the OR operator to combine multiple conditions. For example:

status:error OR status:critical OR message:"OutOfMemoryError" OR message:"NullPointerException"

This query searches for logs with error or critical statuses, or specific error messages.

Steps to Implement in Datadog

Enter the Query

• Type your constructed query in the search bar of Log Explorer.

Example Query

• Example:

status:error OR status:critical OR message:"OutOfMemoryError" OR message:"NullPointerException"

Filter by Time

• Use the time filter to specify the timeframe you are interested in (e.g., last 15 minutes, last 1 hour, last 24 hours).

Creating a Monitor Based on This Query

1. Create Monitor

• Click on Create Monitor above the search results.

2. Configure Monitor

• Set monitor scope: Define search query and fiter time by 1 minute

• Set Alert Condition: If query is run more or equal to one time within the timeframe mentioned above, alert will be sent

• Set Notification Condition:

3. Save Monitor

• Review your settings and save the monitor.

Example Alert Condition

• Configure the alert condition to trigger when there are more than 10 logs matching your query within the last 5 minutes.

Example Alert Message

• Alert: High number of critical errors detected

{{#is_alert}}
There have been {{value}} critical or error logs in the last 5 minutes for service: {{service.name}}.
{{/is_alert}}

Configure values like env, host, correlationId, timestamp into the monitor for log alerts.

Alert Message Example

• Subject for alert:

{{log.service}}-{{log.tags.env}} - Error - {{log.attributes.correlationId}}

• Message Body:

{{#is_alert}}
{{log.service}} Error alert detected in {{log.tags.env}} environment on host {{log.host}}.

Error Count: {{value}}

Error Message: {{log.message}}
TimeStamp: {{log.timestamp}}
correlationId: {{log.attributes.correlationId}}

@user1.mail.com @user1.mail.com
{{/is_alert}}

Mention mail to notify

· Notify Members for Alert: mention mail id with @

Note: If you want recovery mail followed by alert mail after recovered then mention mail id outside alert tag. But if you don't want to receive recovered mail then mention mail id inside {{#is_alert}} {{#is_alert}}

Integrating with MuleSoft Application

Configure log4j.xml

• Add Datadog appender inside appenders tag:

<Http name="DataDog"
      url="https://http-in-take.logs.us5.datadoghq.com/api/v2/logs?host=${sys:hostName}&amp;ddsource=mulesoft&amp;service=datarobot-exp-api&amp;ddtags=env:${sys:mule.env}">
    <Property name="Content-Type" value="text/plain" />
    <Property name="DD-API-KEY" value="YOUR-API-KEY" />
    <JSONLayout compact="true" eventEol="true" properties="true" stacktraceAsString="true">
        <KeyValuePair key="host" value="${sys:hostName}" />
        <KeyValuePair key="ddsource" value="mulesoft" />
        <KeyValuePair key="correlationId" value="$${ctx:correlationId:-}" />
    </JSONLayout>
</Http>

Add appender reference to AsyncRoot

<AsyncRoot level="INFO">
    <AppenderRef ref="file"/>
    <AppenderRef ref="DataDog"/>
</AsyncRoot>

This guide provides a comprehensive approach to setting up log alerts in Datadog for monitoring MuleSoft applications, including integration steps and configuration examples. Adjust the specifics to match your environment and requirements.