Best Practices to create Highly Observable Applications in Mule 4

Elevating your Applications Visibility

Published in

Another Integration Blog

49 min readMay 10, 2024

In this blog, I would like to share some of the Best Practices in creating Highly Observable Applications in Mule 4 for all deployment options except CloudHub2.0.

The CloudHub deployment here refers to ONLY CloudHub1.0 (CH1.0).

Most of the configuration details (relevant to Security) shared here are taken from MuleSoft Documentation/Articles/Blogs.

Let’s start with the understanding on What is Observability:

Observability is a crucial attribute that allows us to comprehend a system’s internal states through its external outputs. It’s not the same as monitoring, but rather it provides an in-depth view of system behavior to ascertain its functioning and potential weaknesses.

Observability leverages three key types of data, often called the “Three Pillars of Observability”:

Logs: Timestamped events detailing what happened in a system.
Metrics: Numerical values captured over specific time frames, offering a data summary.
Traces: Records of a series of events, showing how requests move through a system.

These pillars enable faster issue identification and resolution, enhancing software reliability.

1. Logging

Logging is an essential aspect of application development and maintenance to ensure effective monitoring, debugging, and troubleshooting.

Mule runtime engine (Mule) 4 uses Log4j 2 asynchronous logging by default, so the logging operation happens in a separate thread from the message processing, and the latter can continue without having to wait for the log operation to complete.

1.1. Use Logger Component

Utilize the Logger Component to log messages at various points in your MuleSoft application flows for debugging & monitoring purposes, by including relevant important information such as error messages, status notifications, payload, message properties, flow variables, etc.

Steps to follow

Drag and Drop the Logger Component: In your Mule flow, drag and drop the Logger Component from the Mule Palette into the process area where you want the log message to appear.
Configure the Logger Component: Click on the Logger component to configure it, see below on how & what to configure:

3. Use DataWeave Expressions (if required): The Logger Component is particularly powerful because it allows you to use DataWeave expressions. This means you can log complex information from your Mule event. For example, #[vars.myVariable] would log the value of a variable named myVariable.

4. Use Log Categories (if required): Leverage log categories to categorize logs based on functionality or modules within your application. This allows you to filter and analyze logs more effectively, especially in large-scale applications.

Examples

The below logger is set to monitor message processing status, mixing Strings and expressions:

<logger category="monitoring" message="Message #[payload.id] processed successfully" level="INFO"/>

The below logger set to record the processing time of a flow, using a single expression and DataWeave’s String concatenation:

 <logger category="performance" message="#['Message ' ++ payload.id ++ ' took ' ++ vars.processingTime ++ ' milliseconds to process']" level="INFO"/>

5. Mask sensitive data (if any): Avoid logging sensitive data, such as passwords or Personally Identifiable Information (PII), to comply with security standards. Implement redaction or obfuscation for sensitive information, ensuring that logs don’t expose critical data.

1.2. Use Structured Logging

Use Structured Logging to organize log data in a consistent format. MuleSoft supports JSON logging (JSON Logger Connector), enabling better parsing and analysis. A structured log format enhances readability and facilitates log parsing tools for advanced analytics.

1.2.1 JSON Logger Connector

The JSON Logger Module/Connector (based on DevKit SDK) is a drop-in replacement of the existing Logger component. At the core of this implementation is the assumption that not every company will want the exact same data structure. Thus, the module was architected in a way that lets you customise the output JSON log by defining a matching JSON Schema.

Steps to follow

Install JSON Logger Module: You can download it from Exchange by using MuleSoft Premium or guest account in Anypoint Studio. Once installed, the JSON logger will be visible in the Studio Palette as any standard component.
Drag and Drop the JSON Logger: Drag and drop the connector from the Mule Palette into the process area.
Configure the JSON Logger: Configure various parameters as shown below:

Application name: Name of the application. This value should be based on pom.xml.

Application version: Version of the application. This value should be based on pom.xml.

Environment: Name of the environment where the application is running. This value should be based on external property.

Pretty print: Indicate if log entries should be formatted or single line. It can be one of these values — True (Default), False, Expression.

Log location info: Indicate if location information should be logged. Location object gives you everything you want to know about where you are in your application. This is what it looks like:

Critical troubleshooting information such as the name of the flow (rootContainer), name of the XML configuration file (fileName), and the exact line number where the logger is located (lineInFile) can now be part of the log metadata.

It can take one of these values — True (Default), False, Expression.

Parse content fields in json output: Indicate if Content fields should be parsed as part of the JSON Logger Output. It can take one of these values — True (Default), False, Expression.

With default value — True (Default):

If the content field output is a proper JSON, it will get parsed as part of the main logger:

If the content is anything else other than JSON, it will stringified as usual:

When ‘***Content***’ field is other than JSON

If you still want to keep the previous behavior where everything is stringified, then set as False.

Disabled fields: Indicate which fields (from JSON output) should be disabled from logging separated by comma. They should match the exact name given in loggerProcessor.json schema.

JSON Logger provides two very distinct fields out-of-the-box, in order to segregate/decouple how we log functional data (data that provides meaningful application state information without including sensitive data) vs. troubleshooting data (typically raw data coming in and out of different components):

message: Field meant for meaningful non-sensitive functional messages (non-sensitive data) such as “Request received from Customer with ID: 1234”
content: Field meant for raw data (raw troubleshooting data) typically useful during development and test phases (e.g. payload, attributes, vars, etc.)

We can leverage the “Disabled fields” feature to filter certain log fields from being printed (e.g. content).

Content fields data masking: Indicate which fields (inside content type with JSON output only) should be masked from logging separated by comma. They can be JSON keys or JSON paths. If empty, no masking will be applied.

To configure this feature, we need to provide CSVs (with no spaces in between) with the name of the JSON fields and/or JSON paths (relative to the root element inside the content field) that we want masked:

The JSON path should be relative to the root element of the content field and not to the payload object itself.

External destination: JSON Logger module facilitate pushing logs to external platforms, a feature called ‘External Destinations’:

AMQP (Advanced Message Queuing Protocol)
JMS
Anypoint MQ (AMQ)

With Destinations — JMS & AMQP

Steps to follow

a. Add either of those connectors to your Mule project (the first time you import JSON Logger to your project it will automatically add these dependencies for you so make sure you remove them if not needed).

b. Create a connector configuration (AMQP or JMS):

c. Configuration Ref/Exchange Destination: Pass the name of the connector config to the JSON Logger config along with the name of the destination (e.g., Queue for JMS or Exchange for AMQP):

d. Log Categories: Allow to specify the associated logger name (by default org.mule.extension.jsonlogger.JsonLogger) of a particular operation, we can filter which categories we want forwarded to the external destination:

You can provide a list of categories that should be published to External Destination. If none is specified, it will publish all logs (not recommended).

e. Max Batch Size: Indicate max. quantity of log entries to be sent to the external destination. The logger will internally try to buffer events (depending on how fast logs are generated) and send them in batches. Under high-load conditions, these batches can get large, so to define an upper bound to how big they can get, we set the max batch size (by default set to 25 log events).

With destination — AnypointMQ (AMQ)

There is no need for an additional configuration as the JSON Logger config will be self-sufficient by providing the additional AMQ specific parameters:

Queue or Exchange destination — Name of the target queue or exchange destination
URL — The region URL where the Queue resides.
Client App ID — The Client App ID to be used.
Client Secret — The Client App Secret for the given Client App ID.

**AMQ External Destination** Configuration

1.3. Use Log Levels Appropriately

MuleSoft supports various log levels such as ERROR, WARN, INFO, DEBUG, and TRACE. Use these levels judiciously to control the verbosity of your logs. As a good practice, in the configuration, choose:

— ERROR for critical issues.
— WARN for potential problems (Warnings that may not affect the application but should be noted).
— INFO for general operational messages.
— DEBUG for detailed diagnostic information.
— TRACE for highly detailed tracing messages.

1.4. Avoid Excessive Logging

While logging is crucial, excessive logging can lead to performance issues and make it difficult to find relevant information. Be mindful of what you log, and in production environments, consider adjusting log levels dynamically based on the severity of the issues or as part of a broader logging strategy. Also, avoid excessive or overly verbose logging that can impact system performance and consume storage space.

1.5. Implement Dynamic Log Levels

Consider making log levels dynamic based on properties or configurations (irrespective of whether using normal Logger or JSON Logger). This can be useful for adjusting logging levels in different environments or for specific components of your application at runtime. This can be beneficial for troubleshooting specific issues without restarting the application.

Sample code snippets:

<logger message="Payload: #[payload]" level="#[vars.logLevel]" doc:name="Logger"/>

<json-logger:logger config-ref="JSON_Logger_Config" priority="#[vars.logLevel]" message="This is just a test log message" />

1.6. Implement Log Rotation & Retention (Applicable for HYBRID Applications (OnPrem Runtime) ONLY)

Implement log rotation and retention policies to manage log files effectively. Regularly clean up or archive old log files to prevent storage issues.

NOTE: For info on how to implement, refer this article.

1.7. Include Version Information in Logs

Include version information in your logs (applicable only for normal Logger because in JSON Logger it has been taken care (see above Section 1.2 Use Structured Logging)) to quickly identify the deployed version of the application.

Sample code snippet:

<logger level="INFO" doc:name="Logger" message="Application version: #[app.version]" />

1.8. Build Custom Logging Components

Build custom logging components to encapsulate logging functionality. This promotes reusability and consistency across projects, making it easier to enforce logging standards.

Steps to follow (customizing existing JSON Logger)

Clone the repo: Clone the below repo (from mule-4.x branch) to your local directory:

GitHub - mulesoft-consulting/json-logger: Drop-in replacement for default Mule Logger that outputs…

Drop-in replacement for default Mule Logger that outputs a JSON structure based on a predefined JSON schema …

github.com

2. Change the icon (if required): If required, copy the icon image (with extension .svg) of your choice under icon folder & remove the existing one. Otherwise, you can use the existing one.

3. Modify your POM file: Add your Organization Id for groupId element tag by replacing ORG_ID_TOKEN.

4. Configure your Maven settings.xml: Changes are to be configured in the Maven’s settings.xml, you need to add exchange credentials under servers element:

<server>
  <id>exchange-server</id>
  <username>ANYPOINT_PLATFORM_USER</username>
  <password>ANYPOINT_PLATFORM_PASS</password>
 </server>

Where ANYPOINT_PLATFORM_USER and ANYPOINT_PLATFORM_PASS are the credentials normally used to login into our Anypoint Platform. Make sure you have ‘Exchange Contributor’ role granted.

5. Run the script: Run the deploy-to-exchange.sh script under the base folder by passing your Organization Id as an argument:

./deploy-to-exchange.sh <YOUR_ORG_ID>

This script will basically run the command mvn clean deploy to deploy the following projects into your own Exchange:

jsonschema2pojo-mule-annotations: this project generates all the SDK specific annotations once you build the JSON logger connector.
json-logger: is the actual customized JSON logger.

6. Add the connector to your project from Exchange: Now create a Mule application and import the custom connector from Exchange into your project:

**Custom Logger** imported from Exchange

1.9. Use Contextual Logging

Include contextual information in logs, such as Correlation IDs, Transaction IDs, or User IDs. This facilitates tracing a specific request or transaction across multiple systems, easing troubleshooting and monitoring.

1.9.1 MDC Logging

Mapped Diagnostic Context (MDC) enriches logging and improves tracking by providing more context or information in the logs for the current Mule event.

By default, Mule logs two MDC entries: processor, which shows the location of the current event, and event, which shows the correlation ID of the event.

Mule Tracing module enables you to enhance your logs by adding, removing, and clearing variables from the logging context for a given Mule event. The logging context exists for the entire execution of the corresponding event.

Available Operations

Steps to follow

Add the Module to Your Mule Project: Add Tracing module to your Mule project to automatically populate the XML code with the module’s namespace and schema location and to add the required dependencies to the project’s pom.xml file:

1a. In the Mule Palette view, click (X) Search in Exchange.

1b. In the Add Dependencies to Project window, type tracing in the search field.

1c. Click Tracing Module in Available modules.

1d. Click Add.

1e. Click Finish.

2. Change Pattern Layout to MDC in log4j2.xml: This change instructs Mule to automatically add the MDC context, which includes the correlation ID and the processor path. You can modify the log4j2.xml file of your Mule application or the file of your Mule instance, which is in /conf/log4j2.xml.

2a. Open the log4j2.xml file for editing.

2b. Replace [processor: %X{processorPath}; event: %X{correlationId}] with [%MDC].

3. Configure MDC Logging in Your Application: To configure MDC logging, add any of the available operations to your Mule application flow and specify the corresponding attributes:

3a. In the Mule Palette view, select Tracing and then select the desired operation or scope.

3b. Drag the operation or scope onto the Studio canvas and to the right of the input source.

**Tracing Module** & its Operations in Mule Palette

Flow with **Set logging variable** operations

NOTE: For more info on Tracing Module, refer here.

1.10. Configure Message Logging Policy

The Message Logging policy logs custom messages using information from incoming requests, responses from the backend, or information from other policies applied to the same API endpoint.

1.10.1 With Mule Gateway

We can apply the message logging policy at each individual API-level or for all APIs.

Steps to follow (for each individual API)

Log in to Anypoint Platform using your credentials.
Once logged in, go to API Manager from the main menu.
In the API Manager, locate the API to which you want to apply the policy.
Click on the API name to open its details.
Within the API dashboard, find and click on the Policies tab.
Click on + Add Policy. This opens a list of available policies.
Find the Message Logging policy (under Troubleshooting) in the list, select it & click Next.
Configure policy settings:

**Message Logging Policy** — Configuration Settings

You’ll see various options that you can configure. These typically include:

9. After configuring the settings, click Apply to save the policy and apply it to your API.

Steps to follow (for all APIs)

Log in to Anypoint Platform using your credentials.
Once logged in, go to API Manager from the main menu.
Select Automated Policies from the left navigation menu.
Click + Add automated policy.
Find the Message Logging policy (under Troubleshooting) in the list, select it & click Next.
Configure policy settings (see above).
After configuring the settings, click Apply to save the policy and apply it to your API.

1.10.2 With Flex Gateway

For info on how to configure, refer here.

1.11. Implement Centralized/External Logging

Consider using a centralized logging solution, such as ELK Stack (Elasticsearch, Logstash, Kibana), Splunk, or any other preferred logging platform. Centralized logging simplifies log aggregation, searching, and analysis across multiple instances and environments.

Also, CloudHub’s log persistence is limited in terms of size and log availability period (CloudHub stores logs of up to 100 MB per app & per worker or for up to 30 days, whichever limit is hit first), and organizations often need to access logs beyond these thresholds, or sometimes would want to publish the logs to external services such as Splunk, ELK, etc.,

Let’s consider Splunk as the external system here, so our steps for log forwarding will be based on the same.

Use HTTPS for log forwarding to protect log data in transit. Additionally, ensure your Splunk HEC token is securely stored.

1.11.1 For CloudHub Applications

These are the two main ways of publishing CloudHub application logs to external systems:

— Log Aggregator (No longer recommended due to performance issues)
— Log4j2 Appender

Log Aggregator

The idea is to create an independent mule application (or non-mule application) that retrieves the CloudHub logs using the CloudHub Enhanced Logging API and publishes them to Splunk using their TCP API or HTTP API at regular intervals. We will not get into more details as this approach is not recommended due to performance issues.

Log4j2 Appender

Prerequisite(s)

‘Disable CloudHub logs’ option enabled or visible in Runtime Manager. If the option is not enabled or visible, raise a support case.

Steps to follow

Change configurations in log4j.xml:

Update default log4j.xml as below:

1a. Add the package for com.splunk.logging in Configuration element

<Configuration status="INFO" 
               name="cloudhub" 
               packages="com.splunk.logging,org.apache.logging.log4j">

1b. Configure the log appender in one of the below ways:

Using Log4j2 Appender with HTTP and token — HTTP Appender

<Http name="Splunk" url="${sys:splunk.url}" index="${sys:splunk.index}" 
      disableCertificateValidation="true">
  <Property name="Authorization" value="Splunk ${sys:splunk.token}"/>
  <PatternLayout pattern="%-5p %d [%t] %X{correlationId}%c: %m%n"/>
</Http>

The Splunk URL above will be in this format: https://<<splunkcloud host name>>:<<port>>/services/collector/raw

If you want the logs to be formatted in JSON, then use this URL: https://<<splunkcloud host name>>:<<port>>/services/collector/

Using Splunk Appender — SplunkHttp Appender

<SplunkHttp name="Splunk" url="${sys:splunk.url}" token="${sys:splunk.token}" 
            index="${sys:splunk.index}" 
            disableCertificateValidation="true">
  <PatternLayout pattern="%-5p %d [%t] %X{correlationId}%c: %m%n"/>
</SplunkHttp>

The Splunk URL above will be in this format: https://<<splunkcloud host name>>:<<port>>

1c. Add the below entry if you want the logs to be written in the CloudHub as well:

<Log4J2CloudhubLogAppender name="CloudHub"
                addressProvider="com.mulesoft.ch.logging.DefaultAggregatorAddressProvider"
                applicationContext="com.mulesoft.ch.logging.DefaultApplicationContext"
                appendRetryIntervalMs="${sys:logging.appendRetryInterval}"
                appendMaxAttempts="${sys:logging.appendMaxAttempts}"
                batchSendIntervalMs="${sys:logging.batchSendInterval}"
                batchMaxRecords="${sys:logging.batchMaxRecords}"
                memBufferMaxSize="${sys:logging.memBufferMaxSize}"
                journalMaxWriteBatchSize="${sys:logging.journalMaxBatchSize}"
                journalMaxFileSize="${sys:logging.journalMaxFileSize}"
                clientMaxPacketSize="${sys:logging.clientMaxPacketSize}"
                clientConnectTimeoutMs="${sys:logging.clientConnectTimeout}"
                clientSocketTimeoutMs="${sys:logging.clientSocketTimeout}"
                serverAddressPollIntervalMs="${sys:logging.serverAddressPollInterval}"
                serverHeartbeatSendIntervalMs="${sys:logging.serverHeartbeatSendIntervalMs}"
                statisticsPrintIntervalMs="${sys:logging.statisticsPrintIntervalMs}">
    <PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n"/>
</Log4J2CloudhubLogAppender>

1d. Add the appender(s) reference to AsyncRoot configuration:

<AsyncRoot level="INFO">
    <AppenderRef ref="Splunk"/>
    <AppenderRef ref="CloudHub"/>
</AsyncRoot>

2. Enable external logging configuration in Runtime Manager:

By default, MuleSoft Anypoint Platform does not support routing of logs to an external log management tool like Splunk.

2a. From Anypoint Platform, select Runtime Manager.

2b. Click Applications in the left menu.

2c. Click the Type column to display the details pane for the application.

2d. Click Manage Application.

2e. In the Settings page, click Disable CloudHub logs:

2f. In the Disabling CloudHub logs confirmation window, select the checkboxes to verify that you want to disable CloudHub logs:

2g. Click Apply Changes.

2h. Review your application settings to ensure they are correct.

2i. Click Apply Changes to restart your application.

3. Add required Maven dependencies & repositories into POM file:

3a. Add below dependencies into POM, if not present:

<dependency>
     <groupId>com.splunk.logging</groupId>
    <artifactId>splunk-library-javalogging</artifactId>
    <version>x.x.x</version>
</dependency>

<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <version>x.x.x</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>x.x.x</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>x.x.x</version>
</dependency>

3b. Add below repository into POM, if not present:

<repository> 
        <id>splunk-artifactory</id> 
        <name>Splunk Releases</name> 
        <url>https://splunk.jfrog.io/splunk/ext-releases-local</url> 
</repository>

1.11.2 For Hybrid (OnPrem Runtime) Applications

Steps to follow

The steps are same as that for CloudHub Apps (see above) except Step# 2 — Enable external logging configuration in Runtime Manager, which is not applicable.

1.11.3 For RTF Applications

Steps to follow

Enable Log4J: Custom Log4j configurations are disabled by default. When you install Runtime Fabric, the parameter is disabled (false) in Values.yaml. You must set the value to true.

Custom Log4j is only supported by Mule applications running on Mule runtime engine instances greater than or equal to Mule version 4.3.0

1a. From inside your cluster, run this command:

kubectl -nrtf patch secret custom-properties -p '{"data":{"CUSTOM_LOG4J_ENABLED":"dHJ1ZQ=="}}'

1b. Restart the Runtime Fabric agent pod.

1c. Restart the application associated with the custom Log4j configuration.

2. Create a Log4j Configuration using Splunk: For logs to be viewable in Runtime Fabric and able to flow to Splunk, configure the SplunkHttp Log4j appender.

2a. Update the log4j2.xml configuration file with your logger settings and include the SplunkHttp Log4j appender:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO" name="cloudhub" packages="com.mulesoft.ch.logging.appender">
    <Appenders>
     <Console name="CONSOLE" target="SYSTEM_OUT">
            <PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n"/>
        </Console>
        <SplunkHttp name="SPLUNK"
         source="${env:APP_NAME}"
         host="${env:POD_NAME}"
         sourceType="mule-app"
            url="${sys:splunk.host}"
            token="${sys:splunk.token}"
            index="main">
            <PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" />
        </SplunkHttp>
    </Appenders>
    <Loggers>
        <AsyncRoot level="INFO">
            <AppenderRef ref="CONSOLE"/>
            <AppenderRef ref="SPLUNK"/>
        </AsyncRoot>
    </Loggers>
</Configuration>

2b. Ensure that the configuration includes the Splunk dependency:

<dependency>
   <groupId>com.splunk.logging</groupId>
   <artifactId>splunk-library-javalogging</artifactId>
   <version>x.x.x</version>
</dependency>
.
.
.
.
<repository>
   <id>splunk-artifactory</id>
   <name>Splunk Releases</name>
   <url>https://splunk.jfrog.io/splunk/ext-releases-local</url>;
</repository>

3. Enable Log Forwarding (on VMs or BareMetal):

3a. Go to Runtime Manager.

3b. Select Runtime Fabrics in the sidebar navigation.

3c. Select the applicable Runtime Fabric based on the name used during installation.

3d. Select the Logs tab.

3e. Select the Forward application logs to an external service checkbox to forward logs using a log forwarding output plugin.

3f. In the Connects to drop-down list, select the applicable third-party log forwarding solution.

3g. Enter configuration information for your third-party logging solution, as shown below:

3h. Select Apply changes.

4. Enable Log Forwarding (on Self-Managed Kubernetes): Anypoint Runtime Fabric on Self-Managed Kubernetes does not include external log forwarding. You are responsible for installing, configuring, and managing an external log forwarder. You can use any external log forwarding agent that is compatible with your Kubernetes environment running on Amazon EKS, AKS, or GKE. Common log forwarding agents include:

— Splunk Connect for Kubernetes
— Fluent Bit

1.11.3 With Flex Gateway

— For the steps in Connected mode, refer here.

— For the steps in Local mode, refer here.

1.12. Leverage Anypoint Monitoring Logs Feature

Anypoint Monitoring aggregates log files so you can manage, search for, filter, and analyse your logs. You can use aggregated logs in conjunction with monitoring to help identify issues and investigate failures quickly.

— Comparative analysis and identification of event and information patterns based on searches across selected Mule apps and environments.
— Searches within the logs of a specific worker (or set of workers) for a Mule app so that you can narrow scope of issues to check.
— Sharing a snapshot of important logs with developers who can work on the source of the issue.

Anypoint Monitoring Logs feature is available only if your organization has a Titanium subscription.

NOTE:

For on-premises servers with customized logging: Logs produced by apps with customized logging (by changing the pattern layout in the log4j2.xml file) might not get indexed correctly into Anypoint Monitoring. To ensure correct indexing, use the Mule default pattern layout, which is one of the following:

<PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n" />, or
<PatternLayout pattern="%d [%t] %-5p %c - %m%n" />

1.12.1 Search Logs

On the Log Search page, you can search for text in logs in a few different ways. You can type your search query into the search field, you can use the UI to add a filter, and you can create a query.

When you search for logs for APIs deployed through Flex Gateway, use the Application filter. Use the App ID field to identify the instance.

Some of the actions that you can perform here:

Log Search Query: To search the logs, type your search query into the log search field. For more info. refer here. Also, for info. on the query syntax, refer here.
Add a Filter: You can create a filter to search logs using predefined fields, operators, and values. For more info. refer here.
Search Query: You can convert a search query into a Query DSL. For more info. refer here.
View Log Data: By default, the log entries in your log search results are abbreviated. You can expand and view the log message and details as a table or in JSON format. For more info. refer here.
Use Predefined Filters: When the Quick Filters list is open, you can select and use predefined log filters. For more info. refer here.
Filter Based on the Entire Log Field Value: You can use the funnel-shaped icons on any log in your results to create and remove a filter for the entire log field value.
Share Logs: You can share the URL to your logs. For more info. refer here.
Save Searches: You can save your searches & view them later.

1.12.2 Enable Log Points

The Log Points feature allows you to generate logs for Apps and APIs in real-time and without writing code.

For Apps, you can specify the level of logs you want to generate.
For APIs, you can specify properties for the logs.

This allows you to extract logs on-demand from applications in real-time, without causing re-deployment or application downtime.

How to enable for Apps

The Application Log Points feature enables you to log information from connectors in any Mule app in real-time. If you are using one of the supported connectors, you can select a log level (debug, info, warn, or error) and generate logs.

Prerequisite(s)

The Runtime Manager > Manage Settings permission.

Steps to follow

Log in to Anypoint Platform.
In the navigation bar or the main Anypoint Platform screen, click Monitoring.
In the Anypoint Monitoring navigation menu, click Log Points.
Select the following:

Environment: The environment in which the app is running.
Resource name: The name of the app.

5. Click Select.

6. Select the connectors for which you want to generate logs.

7. Select a log level for each connector.

8. Click Save & Apply. A message that your configuration has been applied appears at the top of the page with a View API Logs link. The logs might take a moment to appear.

9. Click the View Logs link to view the logs for the selected log point.

NOTE: For info. on supported connectors, refer here.

How to enable for APIs

The Log Points feature for APIs enables you to interactively extract data from running managed APIs. You can specify properties of your managed API (for example, the host and method in the request header) for which to generate logs without having to write code or redeploy your application. This helps you troubleshoot without having to restart the application.

Prerequisite(s)

The API Manager > Manage Policies permission.

Supported APIs

Basic endpoint APIs (CloudHub and Hybrid)

— CloudHub
— Hybrid

Endpoint with Proxy APIs

— CloudHub proxy
— Hybrid proxy

Steps to follow

Log in to Anypoint Platform.
In the navigation bar or the main Anypoint Platform screen, click Monitoring.
In the Anypoint Monitoring navigation menu, click Log Points.
Select the following:

Environment: The environment the API is running in.
Resource name: The name of the API.
Version/Instance: The version and instance of the API.

5. Click Select.

6. Select the Request points to log and monitor. You can monitor the following request header log points:

User Agent
Path
Host
Method
Remote Access
Accept
Accept Encoding
Accept Language
Cookie
Referrer

You can click the blue box to the left of Header, to select all log points.
Select Body to monitor the Request Body.

7. Select the Response points to log and monitor. You can monitor the following request header log points:

Status Code
Connection
Date
Set-Cookie
Pragma

You can click the blue box to the left of Header to select all log points. Select Body to monitor the Response Body.

8. Click Save & Apply. A message that your configuration has been applied appears at the top of the page with a View API Logs link. The logs might take a moment to appear.

9. Click the View API Logs link to view the logs for the selected log point.

1.12.3 Download Raw Data

You can use Anypoint Monitoring to download archived log data (raw data) collected on Mule apps that are running in Hybrid, CloudHub, or Runtime Fabric deployments. This data is helpful for security, compliance, or other purposes where you need to examine or collect your warehoused data.

Anypoint Monitoring’s Raw Data feature is available only if your organization has a Titanium subscription.

Steps to follow (to download raw data)

Select the Environment and Resource.
The resource is a Mule app that runs in CloudHub, Hybrid, Runtime Fabric deployments.
Provide the date for the data you want to download, for example: 08/31/2018.
Locate and download log data that you want to examine.

If data is not available for a particular date, it might be that the app was not running that day, your company did not have a subscription to the logging or archiving service, or the app did not exist at that time.

2. Monitoring

2.1. Leverage Built-in Monitoring & Alerting Tools

2.1.1. Runtime Manager

Use Runtime Manager to get an overview of applications and servers, including status and resource usage.

2.1.1.1. Monitoring Dashboards

The Anypoint Runtime Manager monitoring dashboards give you in-depth visibility into usage metrics about both your deployed apps and the systems they’re deployed to. We are interested in ONLY applications.

Anypoint Runtime Manager provides these dashboards for applications:

Application Monitoring Dashboards: Displays information about applications, including deployment target and status. These dashboards help you answer questions about your integrated apps, such as:

— When did peaks in usage occur?
— What impact do high traffic peaks have on response time?
— How well is the processing power behind my app coping with these high traffic peaks?
— What impact do high traffic peaks have on the message fail rate?
— What impact do high traffic peaks of other apps running on the same server have on the performance of a given app?

We are referring to Built-in Application Dashboard or Built-in Anypoint Monitoring Dashboard here, which is part of Anypoint Monitoring.

The graphs and tables in built-in dashboards support a number of metrics, including application metrics. The Overview tab provides overall measures. You can view more granular application metrics through the Inbound, Outbound, and Failure tabs.

Various tabs of the Dashboard (in case of Gold or Platinum subscription)

Additional tabs — Flows (in case of Titanium subscription) & Connectors (in case of Titanium subscription or if the app is hybrid deployed on on-prem runtime)

Prerequisite(s)

You must have either the Runtime Manager Read Applications permission or the Runtime Manager Read Servers permission for the resource.

Steps to follow (to access the dashboard)

Log in to Anypoint Platform using an account that has the above-mentioned permission.
Go to Runtime Manager, click Applications.
Click the name of the application to monitor.

You can view all graphs for different time ranges by selecting the desired time interval in the top-right corner of the dashboard you are viewing.

If your application runs on multiple workers, the workers are charted as different curves on the same graphs, differentiated by different colors.

NOTE: For more detailed info. refer below section 2.1.2.1.1 Built-In Application Dashboards.

2.1.1.2. Alerts

Alerts use Condition-based triggers, or Event-based triggers in various components of the platform in order to notify interested parties or stimulate a response action.

Condition-based triggers recognize abnormal conditions which require investigation and/or action (Runtime Alerts, Monitoring Alerts, API Alerts). These alerts usually have a severity of Info, Warning, Critical. We are interested in ONLY these types of alerts.

Event-based triggers recognize a state change of a workflow that may be of interest to recipients (Contract Alerts). We are not interested in this type of alerts.

Runtime Manager includes several Standard Alert types that trigger notifications for common events such as:

— An application exceeds a specific number of events processed within a specified period of time.
— A deployment is complete, either successfully or not.
— The worker monitoring system notifies CloudHub about a problem with a worker on an application.
— A server is disconnected or deleted.

The major types of sources for Runtime Alerts are Application and Server. We are interested in ONLY Application as source here.

Runtime Fabric application alerts are supported only using Anypoint Monitoring.

2.1.1.2.1 Standard Alerts

Steps to follow (to create a standard alert)

In Runtime Manager, click Alerts in the left navigation pane.

If no alerts are set up, you see:

The arrow shows the **Create your first alert** option

If your account already has associated alerts, you see:

The arrow shows the blue plus (+) icon on the **Alerts** page

2. Click the blue plus (+) icon to create an alert.

Various fields of **Create an Alert** page

3. Complete the required fields:

Name: Unique and meaningful name for the alert. The Name is limited to 256 characters.
Severity level: Severity of the alert: Critical, Warning, or Info
Source: The source of the alert: Applications or Servers. This field appears only if your Anypoint Platform account supports local servers (On-Prem Runtime). We will select Applications as source here.
Application type: When the source is Applications, applications deployed to either CloudHub or local servers (On-Prem Runtime).

You can’t create an alert that applies to both applications deployed to CloudHub and local servers. You must create separate alerts for each deployment type.

Condition: The condition that triggers the alert. The available options depend on whether your alert is linked to an application running on CloudHub, or an application running locally (On-Prem Runtime).

NOTE: For more info. on Alert Conditions, refer below.

Subject: The subject of the alert email to send when the alert is triggered. The subject can include alert variables.
Message: The body text for the email to send when the alert is triggered. The message body can include alert variables.

NOTE: For more info. on Alert Variables, refer here.

Recipients: The recipients of the alert email.

You can select up to 20 platform user IDs and enter up to 20 email addresses.

4. Click Create Alert.

When an event that meets an alert condition occurs, Runtime Manager sends an email similar to the following:

Standard Alerts — Alert Conditions for CloudHub Applications

CPU usage — Cloudhub: The CPU usage of the CloudHub workers that run the application is above or below the percentage for the specified period of time. To enable this condition, specify the percentage and duration that triggers the alert.

Custom application notification: The custom notification contains the specified text. To enable this condition, specify the notification text from the CloudHub Connector in your Mule application. When the text in one of the notifications matches the specified string, an alert is sent.

***Custom application notification*** *Condition*

Exceeds event traffic threshold: The number of events of any type exceeds the threshold for the specified period of time. To enable this condition, specify the capacity threshold and duration that triggers the alert. This alert condition depends on the events you are tracking in your application.

***Exceeds event traffic threshold*** *Condition*

Memory usage — Cloudhub: Memory usage is a percentage that represents the JVM heap-used/JVM total memory. The alert triggers if the memory usage is above or below the limit for the specified period of time. To enable this condition, specify the percentage and duration that triggers the alert.

***Memory usage — Cloudhub*** *Condition*

These alerts do not require additional parameters:

— Deployment failed
— Deployment success
— Secure data gateway connected
— Secure data gateway disconnected
— Worker not responding

Standard Alerts — Alert Conditions for Hybrid Applications

Number of errors: The number of messaging errors reported by Mule runtime server in one minute reaches the specified limit. The number of errors reported depends on error handling configuration in the app. To enable this condition, specify the number of errors per minute that triggers the alert.

Number of Mule messages: The number of Mule messages since the application started reaches the specified limit. To enable this condition, specify the number of Mule messages that triggers the alert.

***Number of Mule messages*** *Condition*

Response time: The response time reaches the specified limit. To enable this condition, specify the maximum response time that triggers the alert.

These alerts do not require additional parameters:

— Application undeployed
— Deployment failure
— Deployment success

NOTE: For more info on how to manage standard alerts, refer here.

2.1.1.2.2 Custom Alerts

You can also create custom alerts for applications deployed to CloudHub. Custom alerts in CloudHub are triggered when your application sends notifications to the Runtime Manager console using Anypoint Connector for CloudHub (CloudHub Connector).

2.1.2. Anypoint Monitoring

As part of Anypoint Platform, Anypoint Monitoring provides visibility into integrations across your app network. The monitoring tools provide feedback from Mule flows and components in your app network.

2.1.2.1. Monitoring Dashboards

Dashboards in Anypoint Monitoring provide visibility into Mule apps and APIs that are deployed to your environments. Use built-in dashboards or customize your own to provide instant visibility into your applications’ performance and detailed analysis of your network.

Built-in dashboards contain a standard set of graphs, while custom dashboards can display additional data that is important to your organization.

Prerequisite(s)

You need Anypoint Monitoring User permission.
CloudHub: You must deploy the Mule apps to a supported version of Mule runtime engine and set the runtime version in your Mule app.
Hybrid: You need to install the Anypoint Monitoring agent on an on-premises server to monitor applications. For info. on how to install, refer here.

Monitoring for applications deployed to Runtime Fabric is enabled by default.

2.1.2.1.1 Built-In Application Dashboards

Built-in Application Dashboards contain graphs that plot current and historical data collected over a given time and date period. There is a built-in dashboard for each Mule application and API in each environment.

These are same Application Monitoring Dashboards, which are part of Anypoint Runtime Manager.

Steps to follow (to view the dashboard)

Log in to Anypoint Platform.
In the navigation bar or the main Anypoint Platform screen, click Monitoring.
Select the Environment and Resource name for the App or API to view the corresponding dashboard.
Click View. The built-in dashboard for the App or API is displayed.
To switch to a dashboard for a different resource, click the built-in dashboard selector at the top of the page.

Your data is prepopulated and displayed in the built-in dashboards and logs; no configuration is needed.

If your organization has a Titanium subscription, the Built-in Dashboard has additional capabilities.

Built-in Application Dashboards include various pages/dashboards to view different types of charts:

Overview

Mule Messages
Average Response Time
Errors
Inbound — Total Requests by Response Type
Inbound — Average Response Time
Outbound — Total Requests by Response Type
Outbound — Average Response Time
JVM — CPU % Utilization
JVM — Heap Used
JVM — Thread Count

NOTE: For more info. refer here.

Inbound

Inbound — Total Requests by Response Type
Inbound — Average Response Time
Inbound — Response Time by HTTP Endpoint
Inbound — Total Requests by Endpoint
Inbound — Response Time graphs (99, 90, 75, and 50 percentiles)
Inbound — Total Failed Requests
Inbound — Total Slow Requests by Endpoint

The following charts are available ONLY if you are using Unified Agent and your organization has a Titanium subscription:

Inbound — Protocols
Inbound — HTTP Endpoints
Inbound — Maximum Response Time

NOTE: For more info. refer here.

Outbound

Outbound — Total Requests by Response Type
Outbound — Average Response Time
Outbound — Average Response Time by HTTP Endpoint
Outbound — Total Requests by HTTP Endpoint
Outbound — Response Time graphs (99, 90, 75, and 50 percentiles)
Outbound — Total Failed Requests

The following charts are available ONLY if you are using Unified Agent and your organization has a Titanium subscription:

Outbound — HTTP Ports
Outbound — Operations
Outbound — HTTP Protocols
Outbound — Endpoints

NOTE: For more info. refer here.

Flows

The following charts are available ONLY if you are using Unified Agent and your organization has a Titanium subscription:

Inbound — Response Time by Flow
Outbound — Response Time by Flow
Inbound — Requests by Flow
Outbound — Requests by Flow
Failures by Flow
Inbound — Slow Requests by Flow
Outbound — Slow Requests by Flow
Inbound — Response Time by Flow
Outbound — Response Time by Flow
Inbound — Requests by Flow
Outbound — Requests by Flow
Failures by Flow
Inbound — Slow Requests by Flow
Outbound- Slow Requests by Flow

If you don’t see the Flows tab, verify that you are using Unified Agent and that you have a Titanium subscription.

NOTE: For more info. refer here.

Connectors

Connector Message Volume
Response Time by Connector
Salesforce Requests
Salesforce Response Time
Salesforce Failures
Database Requests
Database Response Time
Database Failures
HTTP Requests
HTTP Response Time
HTTP Failures
Web Services Requests
Web Services Response Time
Web Services Failures
FTP Requests
FTP Response Time
FTP Failures
SFTP Requests
SFTP Response Time
SFTP Failures
SMTP Requests
SMTP Response Time
SMTP Failures
Object Store Requests
Object Store Response Time
Object Store Failures
Anypoint MQ Operations — Inbound
Anypoint MQ Response Time — Inbound
Anypoint MQ Failures — Inbound
Anypoint MQ Operations — Outbound
Anypoint MQ Response Time — Outbound
Anypoint MQ Failures — Outbound
Other Connector Requests
Other Connector Response Time
Other Connector Failures

If you don’t see the Connectors tab, verify that:

— You have a Titanium subscription.
— The app you selected is a hybrid app deployed to a server, server group, or cluster.

NOTE: For more info. refer here.

Performance

Mule Messages
Average Response Time
Inbound — Average Response Time
Inbound — Response Time by HTTP Endpoint
Inbound — Response Time graphs (99, 90, 75, and 50 percentiles)
Outbound — Average Response Time
Outbound — Response Time by Endpoint
Outbound — Response Time graphs (99, 90, 75, and 50 percentiles)

NOTE: For more info. refer here.

Failures

Errors
Inbound — Total Failed Requests
Outbound Total Failed Requests
Inbound — Total Requests by Response Type
Outbound — Total Requests by Response Type
Inbound — Failure Response Time by Endpoint
Inbound — Total Failed Requests by Endpoint
Outbound — Failure Response Time by HTTP Endpoint
Outbound — Total Failed Requests by HTTP Endpoint

NOTE: For more info. refer here.

JVM

Old Gen GC Count
Old Gen GC Time
Young Gen GC Count
Young Gen GC Time
Classes (Loaded, Loaded Total, Unloaded)
Heap (Committed, Used)
Thread Count
JVM Uptime
Eden Space graphs (Used, Max, Init, and Committed)
Survivor Space graphs (Used, Max, Init, and Committed)
Metaspace graphs (Used, Max, Init, and Committed)
Code Cache graphs (Used, Max, Init, and Committed)
Compressed Class Space graphs (Used, Max, Init, and Committed)
Old Gen Mem graphs (Used, Max, Init, and Committed)

The following charts are available ONLY if you are using Unified Agent and your organization has a Titanium subscription:

Committed Virtual Memory

NOTE: For more info. refer here.

Infrastructure

JVM — CPU % Utilization
JVM — Heap Used
Total System Processors
System Memory
JVM Thread Count

The following charts are available only if you are using Unified Agent and your organization has a Titanium subscription:

System CPU % Utilization
JVM CPU Time
Operating System Swap Space

NOTE: For more info. refer here.

Custom Metrics

The Custom metrics tab appears only if:

— Your organization has a Titanium subscription.
— The app you selected is running on a supported version of Mule.

NOTE: For info. on how to configure Custom Metrics, refer here.

Application Network

The chart on the Application Network tab shows the Troubleshooting visualization for the selected app. The Application Network tab appears only if:

— Your organization has a Titanium subscription.
— You have set up Anypoint Visualizer.
— The app you selected is running on a supported version of Mule.

2.1.2.1.2 Built-in API Analytics Dashboards

The API Analytics Dashboards provide charts to monitor API usage, which gives you insight into how your APIs are used and how they are performing. If you have enabled API auto-discovery, your API metrics are available in the API analytics dashboards in Anypoint Monitoring.

The API analytics dashboard includes two views: Enhanced Metrics and Basic Metrics. The Enhanced Metrics view includes detailed information about your APIs, and charts are sorted into dashboards based on the type of data displayed. By default, the enhanced metrics view is selected.

Steps to follow (to view the dashboard)

Log in to Anypoint Platform.
In the navigation bar or the main Anypoint Platform screen, click Monitoring.
Select the Environment, Resource name, and Version/Instance for the API to view.
Click View.
On the API analytics page, you can view information about a running, managed API.
Select the date and time range you want to view statistics for and click Confirm.

6. If you want to view API analytics metrics in API Manager, click the View API in API Manager link.

You will come across 2 types of views:

Enhanced Metrics View

API analytics dashboards provide you with various pages from which you can view different types of charts.

Overview

Request Volume
Failed Requests
Total 4xx Errors
Total 5xx Errors
Policy Violations
Average Response Time

NOTE: For more info. refer here.

Requests

Requests
Requests by Method
Requests by Response Code
Requests by Request Size
Requests by Response Size

NOTE: For more info. refer here.

Failures

Failures
Failures by Method
Requests by Response Code
Requests by Request Size
Requests by Response Size

NOTE: For more info. refer here.

Performance

Requests by Performance (line chart)
Requests by Performance (table)
Slow Requests (Titanium subscriptions only)

NOTE: For more info. refer here.

Client Applications

Requests by Client ID
Requests by Client IP
Requests by Location

NOTE: For more info. refer here.

Basic Metrics View

For info. on Basic Metrics View refer here.

2.1.2.1.3 Custom Dashboards

Custom Dashboards in Anypoint Monitoring can bring together important metrics and data points that you need to see on one screen. You can specify the resources and metrics that you want to monitor, allowing you to:

— Correlate diverse metrics
— Perform comparative analysis
— Differentiate between regular trends and anomalies
— Isolate issues quickly

Steps to follow (to create a custom dashboard)

In Anypoint Monitoring, click Custom dashboards in the left navigation menu.
Click + New Dashboard.
Add and configure a chart for your dashboard.
Click Save Changes.
In the Save as dialog, enter a New name, and click Save.

After you create a custom dashboard,

You can configure dashboard-level properties, such as the name of the dashboard, any tags, its timezone, and metadata. For more info. refer here.
You can configure one or more charts (also called Widgets) for displaying your data in each row of a dashboard:

Graph: Can display a wide variety metrics as time-series data. Graphs support a query language capable of pinpointing the data you need to see. NOTE: For more info. on how to configure, refer here.

Singlestat: Provides a summary statistic (a single total or number) for a single grouping of time-series data, such as a response time or a heap usage statistic. NOTE: For more info. on how to configure, refer here.

Table: Supports time-series data in a tabular format. It allows for cell coloring and date and value formatting. NOTE: For more info. on how to configure, refer here.

Text: Provides an area for textual information (titles and descriptions) written in Markdown or HTML within rows or cells of a dashboard. NOTE: For more info. on how to configure, refer here.

NOTE: For more info. on Custom Dashboards refer here.

2.1.2.2. Alerts

Alerts indicate whether a resource (such as a Mule app) is behaving as expected or exceeding a defined threshold.

You can configure two different types of alerts:

— Basic alerts for servers, Mule apps, and APIs.
— Advanced alerts for graphs in custom dashboards in Anypoint Monitoring.

For basic and advanced alerts, thresholds are checked every five minutes.

2.1.2.2.1 Basic Alerts

Basic Alerts are applied to environment/application combinations and measure metrics to recognize an alert condition. You can set up Basic Alerts to trigger email notifications when a metric you are measuring passes a specified threshold.

You can create Basic Alerts for the following metrics:

For on-premises servers, server groups, server clusters, and apps deployed to Anypoint Runtime Fabric or CloudHub:

CPU utilization
Memory utilization
Thread count

For Hybrid, Anypoint Runtime Fabric, and CloudHub apps:

Message count
Message error count
Message response time

Steps to follow (to create a Basic Alert)

Go to Anypoint Monitoring.
Click the Alerts tab.
Click the + New Alerts button.
Click Basic alert.
In the Add basic alert window, enter the information required to create your basic alert:

Alert name: Unique and meaningful name for the alert. Name is limited to 256 characters.
Severity level: Severity of the alert: Critical, Warning, or Info. For more info, refer here.
Source type: Source of the alert: Select Application or API (We are interested in ONLY Applications & APIs).
Environment name: Name of the environment where the resource is deployed.
Resource name (apps), API (APIs): Application, or API to apply the alert to (We are interested in ONLY Applications & APIs).
Metric: Condition that triggers the alert. See below.
Subject: Subject of the alert email to send when the alert is triggered. The subject can include variables. For info. on variables, refer here.
Message: Body text for the email to send when the alert is triggered. The message body can include variables.
Recipients: Recipients of the alert email. You can select up to 20 platform user IDs and enter up to 20 email addresses. If you add or change the email address of an alert recipient, they do not receive an email until the alert is triggered.

6. Click Create.

Basic Alerts — Alert Conditions

For basic alerts, alert conditions consist of a metric, an operator, and a period of time that the condition is met.

Application Metrics

Message count: Total number of Mule messages that are processed successfully.

Message error count: Number of Mule messages that return an exception in a flow.

Message response time: Response time (in milliseconds) of a message.

CPU utilization: Percent of CPU used on the virtual machine. This metric includes all processes. This metric is available only for apps deployed to CloudHub or Anypoint Runtime Fabric.

Memory utilization: Percent of heap used. This metric is available only for apps deployed to CloudHub or Anypoint Runtime Fabric.

Thread count: Total number of threads used. This metric is available only for apps deployed to CloudHub or Anypoint Runtime Fabric.

API Metrics

Total request count: Sum of requests in the selected time period for the given API. Data is aggregated in one-minute increments.

Average response time: Average response time of requests in the selected time period for the given API. Data is aggregated in one-minute increments.

Response codes: Sum of HTTP response codes that occur in the selected time period.

Policy violation: Sum of requests that return policy violations that you select in the alert condition.

Operator

— Above
— Above or equal
— Equal
— Below or equal
— Below

Time Period

The time period specifies the duration for which the metric remains valid, based on the criteria defined using the operator and chosen value.

— 5 minutes
— 10 minutes
— 15 minutes
— 20 minutes

2.1.2.2.2 Advanced Alerts

Advanced Alerts are used to monitor a broader range of metrics and conditions.

This feature is available only if your organization has a Titanium subscription.

Alert States

No data: No data is available in the series to evaluate the alert threshold.

Disabled: The alert is disabled.

Alerting: The value of the metric passes the specified threshold.

OK: The value of the metric has not passed the threshold.

Pending: The alert is enabled and waiting for evaluation.

Steps to follow (to create an Advanced Alert)

Navigate to the configuration screen for a graph in a custom dashboard. For info. on how to configure, refer here.
Provide a general configuration for the graph that (at minimum) defines the metric, environment, and resource for which you want to set the advanced alert. For info. on how to configure, refer here.
Navigate to the Alert configuration tab for the graph.
Click Add Alert.
Use Alert Config to configure the alert. For info. on how to configure, refer here.

6. Configure email notifications when an advanced alert is triggered. For info. on how to configure, refer here.

7. Click Apply Changes.

8. Click X to exit the configuration and return to the dashboard.

2.1.2.3. API Functional Monitoring

API Functional Monitoring enables you to monitor responses to requests sent to API resources at their endpoints. You monitor APIs by accessing the Functional Monitoring section of Anypoint Monitoring.

The Functional Monitoring section contains tools for creating and editing monitors. A Monitor includes a test and one or more schedules for running tests.

A monitor test includes:

The endpoints to verify.
Optional assertions to verify at the endpoints (for example, a 200 status code).
Optional HTTP headers to include when sending requests to the endpoints.
Optional tools, such as SumoLogic and Slack, used to send test reports.

A monitor test schedule includes:

The location from which to run the test. Locations can be public or private.
The time interval between tests.

Steps to follow

Click Create Monitor in the Functional Monitoring section of Anypoint Monitoring.
Under Step 1 — Setup the monitor:

2a. Specify a monitor name.

2b. Select a testing location.

2c. Set the monitoring schedule.

3. Click the Step 2 — Select endpoints tab and set up the tests as follows for each endpoint:

3a. By default, the monitor validates self-signed certifications for each endpoint, so Validate SSL Certificate is toggled on. To allow unsafe self-signed certifications for an endpoint, toggle Validate SSL Certificate off.

3b. Select the method and enter the endpoint URL.

3c. Search for, select, or configure custom headers and insert a value for each header.

3d. For POST, PUT, or PATCH endpoints, optionally insert body content.

3e. Optionally select headers and insert their values.

3f. Select assertion information.

3g. Preview the test by clicking Run now under Preview the result of your monitor.

4. Click the Step 3 — Set notifications tab.

Optionally select notifiers from the list of integrated reporting tools and enter the required information in the value field for each selected notifier:

Various Reporting Tools & its required info.

5. Click Create Monitor.

For hybrid applications (deployed on OnPrem Runtime), make sure your Functional Monitors have proper access to your applications via VPN.

NOTE: For info. on how to manage monitors, refer to below links:

2.1.3. API Manager

2.1.3.1. Manage Alerts

API Alerts and Contracts Alerts are managed by Anypoint API Manager. We are ONLY interested in API Alerts.

2.1.3.1.1. API Alerts

You can add & configure API Alerts that flag unusual API behavior, such as:

An API response time over 60 seconds
More than 1000 API requests in 30 seconds

API Manager enables you to add the following API alerts to an API instance:

Policy Violation: An infraction of one or more policies governing the API occurs.
Request Count: Users request access to the API more times than allowed in a specified interval of time. Specify time interval values from 1 through 999999999. To prevent data bursts, intervals are implemented as sliding windows instead of as absolute minute boundaries.
Response Code: The API returns one of these HTTP codes upon receiving a request: 400, 401, 403, 404, 408, 500, 502, or 503.
Response Time: The response time of the API exceeds a specified timeout period.

Alerts cannot be added to API instances protected by automated policies.

When configuring an API alert, you select Business Group users to receive email notifications about the alert. Notifications include information about events that trigger alerts as specified by severity level. Users receive the following:

One email describes the alert:

Your API, CesarDemoAPI - Basic:151 (Environment: "NewOne"), has received greater than 2 requests within 1 consecutive periods of 1 minute.

Another email notifies you when the alert is resolved.

Your API Version, jsonplaceholderapi - 1.0.development, is no longer in an alert state. The number of policy violations was not greater than 1 in the last 1 consecutive periods of 1 minute.

After an alert is triggered, API Manager sends the first set of two notification emails and stops listening for alerts until the next alert period begins. This technique prevents repeated notification emails.

API Alerts are sent from no-reply@mulesoft.com. This cannot be customized.

Steps to follow (for Mule Gateway or Service Mesh)

Navigate to Anypoint Platform > API Manager.
In API Administration, click the API instance to which to add an alert.
Click Alerts > Add alert.
Specify the alert options:

Name: Type an alert name, for example trial-alert. If adding multiple alerts to the same environment, the alerts must have unique names.
Enabled: The default option is Enabled. You can delete an alert anytime if not required.
Severity: Select a severity for the alert, for example, Info.
Alert type: Select the type of alert to create (for example, Request Count) and provide appropriate alert configuration values.

5. Select Recipients. Specify what emails receive the email notification.

6. Click Create Alert.

Steps to follow (for Flex Gateway)

Navigate to API Manager > API Administration and click the API instance for which you want to add an alert.
From the left navigation pane, select Alerts > New alert.
Specify values for the displayed alert options:

Name: Type an alert name, for example, trial-alert. You can delete an alert anytime if not required.
Severity: Select a severity for the alert:

— Info: Assign to alerts that do not require immediate attention when triggered. This severity indicates that the metric should be passively monitored.
— Warning: Assign to alerts that require prompt attention when triggered. This severity indicates an alert should be closely monitored.
— Critical: Assign to alerts that require immediate attention when triggered. This severity indicates an alert should receive an immediate response.

4. Specify the alert conditions:

Metric: Condition that triggers the alert.
Logic: When the alert is triggered, based on the occurrence rate defined in the When number of occurrences are and For at least [number of] minutes fields. You can select a maximum of ten response codes for alert conditions.

5. Configure the email notification:

Subject: Subject of the alert email to send when the alert is triggered. The subject can include Alert Variables.
Message: Body text for the email to send when the alert is triggered. The message body can include Alert Variables.
Recipients: Recipients of the alert email. You can select up to 20 platform user IDs and enter up to 20 email addresses.

6. Click Create.

NOTE: For info. on alert variables refer here.

2.2. Integrate with External Third Party APM Tools

An Application Performance Monitoring (APM) tool is a software system designed specifically to help manage and monitor the performance and availability of software applications. APM tools play a critical role in ensuring that applications meet expected performance standards and provide a positive user experience. They are essential for developers, IT operations teams, and business stakeholders to understand the real-time health and efficiency of applications.

Integrating with external Application Performance Management (APM) tools can enhance monitoring and performance insights for your Mule applications. Common APM tools used with MuleSoft include Dynatrace, New Relic, AppDynamics, Datadog and Prometheus.

Let’s consider Dynatrace as the third-party APM tool here, so our steps for integration with APM tool will be based on the same.

2.2.1 For CloudHub Applications

CloudHub Apps integrated with Dynatrace for Monitoring

These are the metrics which can be monitored:

vCores Production — The number of vCores active on the organization for Production environments.
vCores Sandbox — The number of vCores active on the organization for Sandbox environments.
vCores Design — The number of vCores active on the organization for Design environments.
VPCs — The number of VPCs active on the organization.
VPNs — The number of VPNs active on the organization.
Gateways — The number of Gateways active on the organization.
Applications — The number of Applications on the organization.
Started Applications — The number of Applications that are started on the organization.
Workers — The number of Workers for an application.
Remaining Workers — The number of Workers remaining for an application.
Requests — The number of requests for an application.
Errors — The number of errors for an application.
Response Time — The average response time for an application.
JVM Heap Used — The amount of JVM Heap used for the worker.
JVM Heap Total — The total amount of JVM Heap for the worker.
Thread Count — The number of threads for the worker.
CPU — The CPU usage for the worker.
Load Average — The load average for the worker.
Available Processors — The number of available processors for the worker.
Requests — Per endpoint and response type.
Response time — Per endpoint and response type.
Outbound Response Time — Per endpoint and response type.
Outbound Requests — Per endpoint and response type.
API Requests — Per API, response_code.
API Request Size — Per API, response_code.
API Response Size — Per API, response_code.
API Response Time — Per API, response_code.
API Policy Violations — Per API, policy.
API Requests by Location — Per API and location.

Prerequisite(s)

Enable monitoring for your CloudHub deployed applications. For info on how to enable monitoring, refer here.
Make sure Dynatrace’s component — ActiveGate is installed on a separate host (Windows or Linux) & is running.

Steps to follow

MuleSoft Configuration: This is the configuration required to be done in Anypoint Platform.

1a. Login to Anypoint Platform as Org. Admin.

1b. Navigate to Access Management.

1c. Go to Connected Apps & click on Create app.

1d. Give the connected app these scopes:

— Monitoring Viewer
— View Organization
— View Environment
— Read Applications
— View APIs Configuration

2. Dynatrace Configuration: This is the configuration required on Dynatrace side.

2a. Under Extensions on the left menu, select Mulesoft Cloudhub .

2b. Add a monitoring configuration:

Mulesoft Domain URL: This is the URL to access your Mulesoft account.

For most cases, this is https://anypoint.mulesoft.com
For Europe, this is https://eu1.anypoint.mulesoft.com
For Government Cloud, this is https://gov.anypoint.mulesoft.com

Mulesoft Client ID: From the connected app you created, copy the Client ID

Mulesoft Client Secret: From the connected app you created, copy the Client Secret

Mulesoft Organization/Business Group ID: This is Optional, if empty the default Organization ID will be used.

Frequency: How often should we collect data (metrics in minutes).

Environments filter: Filter environments & applications to monitor.

Optionally you can filter to only monitor certain environments and applications. Otherwise, if empty all environments and applications will be monitored.
These filters behave as “contains”.
You can have multiple filters for a single monitoring configuration.

2.2.2 For Hybrid (OnPrem Runtime) Applications

OnPrem Runtime (Hybrid Apps) integrated with Dynatrace for Monitoring

In order to monitor Hybrid Applications (OnPrem Runtime),

A OneAgent must be installed on the host running that Runtime.
An ActiveGate is also required to route communication to the Dynatrace.

OneAgent requires a broader deployment footprint (i.e., installed on each host), whereas fewer ActiveGates are needed, and they serve as a central hub for multiple OneAgents.

Once the agent has been installed, it will monitor:

— The JVM running the MuleSoft Runtime (which hosts your Mule applications).
— The complete OS.
— All other applications.

3. Distributed Tracing

Distributed tracing is the practice of appending what’s referred to as a “trace,” i.e., tracking metadata stored in a request header — to a subset of requests as they move across your application ecosystem.

The benefit is quicker diagnosis and remediation of errors due to contextual data from your whole application stack, as opposed to the slower, segmented analysis provided by individual application logs.

A Trace aggregates the information from all the actions made from a single request in the correct order. These actions are known as Spans. A span that precedes another span is called a “parent span”, while a span that follows is called a “child span”.

***Traces*** *are made up of multiple* ***Spans***

Traces track requests across services, aggregate “parent” and “child” spans in the right order, and collect and monitor data along the journey, from start to finish.

3.1. Standardize Trace and Span IDs

Ensure that Trace IDs and possibly Span IDs are included in every log statement. This makes it easier to correlate logs from different components of your applications.

Steps to follow

Refer to Section 1.9.1 MDC Logging above.

3.2. Comply with OpenTracing Standard

One of the main challenges with Microservices is visibility of component interactions because of the distributed nature of the system.

OpenTracing (OT) is a free and open source framework that aims to help alleviate this problem. Basically, every application can be instrumented to automatically create or augment existing traces, and propagate them when interacting with external components.

Some of the popular frameworks (which are implementations of OT) are Zipkin, Jaeger, and Hawkular.

**Logical Diagram of OpenTracing Standard**

Client-side components

OpenTrace client

— A connector or a helper service that captures traces and/or spans and sends those events to an agent.
— User will use OpenTrace API methods like begin, end, inject, extract, tag, and log to create tracing events

OpenTrace Agent

— This is typically co-located along with your microservices to achieve optimal performance benefits.
— Collects trace events from OpenTrace clients and delivers those messages to a collector in near real-time.

Server-side components

Collector

— This is a high-throughput in-memory queue.
— As the name indicates, it collects the messages/events from the agent and delivers them to the Cassandra/ElasticSearch database.
— It throttles the incoming messages from multiple collectors and delivers them to the database.
— When the incoming message volume is higher than the processing capacity of DB, it reduces the sampling frequency.

ElasticSearch Database

— The NOSQL database stores the events for processing by OpenTrace query system.
— You can use Cassandra database instead of ElasticSearch.

Query system

— This queries the NOSQL database and provides the data to UI for rendering.

— UI displays the tracing data.

How to implement Distributed Tracing for Mule Microservices

**Distributed Tracing or OT Implementation using SideCar Pattern**

Use Sidecar pattern to implement tracing for Mule microservices, where we bundle two Docker containers in the same pod:

— One of those is the Mule microservice which creates the trace & span events.
— Other is an assistive container that actually intercepts and delegates those events into tracing collector.

3.3. Enable Trace Logging for Policies

Trace a request as it passes through the different policies enabled in a Mule Application in the following manner:

Each policy it goes through — In source and operation.
The outcome of each policy (allow or reject).
Final response.

Each request will have an event id allowing us to trace it unequivocally.

Steps to follow (for CloudHub Applications)

Log in to Anypoint Platform using your credentials.
From Anypoint Platform, select Runtime Manager.
Click Applications in the left menu.
Select the application for which you want to enable tracing.
Go to Settings of the selected application.
On the Logging section of your application add the following logging level and package:

DEBUG org.mule.runtime.core.internal.policy

Similarly for custom policies add the following:

DEBUG org.mule.runtime.core.internal.processor.LoggerMessageProcessor

Steps to follow (for Hybrid Applications)

Go to log4j.xml file of the application in Anypoint Studio.
Add the following snippet under the <Loggers> element:

<AsyncLogger name="org.mule.runtime.core.internal.policy" level="TRACE" />
<AsyncLogger name="com.mulesoft.extension.policies" level="DEBUG"/>

4. Exception Tracking

4.1. Use Structured Exception Handling

Structure your error handling to catch and process errors at different levels of your application, such as:

Global Exception Handlers: Define global error handling strategies that apply to multiple flows or the entire application, ensuring that any uncaught exceptions are handled uniformly. For steps/info on how to create a common/global error handler, refer this blog.
Flow-specific Handlers: Customize exception handling for specific flows to address flow-specific requirements and behaviors.
Processor-specific Handlers: Customize exception handling for specific processors in a flow to address processor-specific requirements and behaviors.

4.2. Log Detailed Error Information/Enrich Error Information with Custom Data

Ensure that your error logs are comprehensive and provide enough context to diagnose issues:

Error Metadata: Include error types, messages, stack traces, and any custom properties in your logs, which might help in troubleshooting.
Transaction Context: Include Correlation IDs or Transaction IDs to link errors to specific transactions or processes. For more info., refer to Section 1.9 Use Contextual Logging above.
Business Context: Include data such as the user ID, operation being performed, or business transaction details.
Performance Metrics: Record performance metrics at the time of the exception to analyze potential performance-related issues.

4.3. Implement Notifications for Critical Errors

Set up your real-time alerts for critical errors or anomalies detected in your logs or error handlers. You can achieve it by one of the ways:

— Create custom alerts. For more info. refer to section 2.1.1.2.2 Custom Alerts above.
— Use external third-party monitoring tools that can send notifications through email, SMS, or other communication channels based on error severity or type.
— Create your own Notification service/API which will take care of sending notifications via email, SMS, or other communication channels.

4.4. Monitor and Analyze Exceptions

Regularly review exception/error data to identify patterns or recurring issues. Use this analysis to:

Identify Common Errors: Determine if errors are caused by system issues, user errors, or external dependencies.
Refine Error Handling: Continuously improve error handling strategies based on the insights gained from exception analysis.

4.5. Implement Error Propagation Strategies Correctly

Decide on how to propagate your errors:

Error Responses: For API-based integrations, define clear error structures in your API specifications and ensure that errors are communicated effectively to consumers.
Graceful Failure: Design your application to fail gracefully, providing useful feedback to the consumer.

As a common good practice, make sure that any of your sensitive data (such as PII, etc) are not exposed directly. Instead use masking, or encryption, etc. techniques to avoid exposure.

Conclusion

This is an attempt to collate some of the good practices to build Highly Observable Mule applications. Developers who want to build Mule applications with high observability can refer this article.

Thank you for reading!! Hope you find this article helpful in whatever possible. Please do not forget to like, share & feel free to share your thoughts in the comments section.

If interested, please go through my previous blogs on best practices:

References

MuleSoft Documentation

MuleSoft Documentation Site

docs.mulesoft.com

What is Distributed Tracing? A Friendly Guide for Developers | Aspecto

You will learn what distributed tracing is, why logs are not enough to debug your microservices, and what OpenTelemetry…

www.aspecto.io

Optimizing Integration Performance: Implementing Custom Monitoring and Graphing in MuleSoft

In current dynamic business landscape, the ability to monitor and analyze integration performance is crucial for…

medium.com

Mulesoft Datadog Integration

A quick review of the steps involved in integrating Mulesoft with Datadog to collect the logs.

medium.com

Mulesoft Cloudhub (Extension v2) monitoring & observability | Dynatrace Hub

Gather in depth metrics and topology from your Anypoint Cloudhub Organizations, Environments, Applications, APIs and…

www.dynatrace.com

Dynatrace Monitoring Setup for MuleSoft

Why Dynatrace :

medium.com

MuleSoft and Application Performance Monitoring: Optimising Software Performance with Dynatrace …

Introduction Software applications are the backbone of many businesses and...

splatform.devoteam.com

Help And Training Community

Edit description

help.salesforce.com

Help And Training Community

Edit description

help.salesforce.com

Help And Training Community

Edit description

help.salesforce.com

Help And Training Community

Edit description

help.salesforce.com

https://blogs.mulesoft.com/dev-guides/microservices/distributed-transaction-tracing/

Best Practices to create Highly Observable Applications in Mule 4

Elevating your Applications Visibility

1. Logging

1.1. Use Logger Component

1.2. Use Structured Logging

1.3. Use Log Levels Appropriately

1.4. Avoid Excessive Logging

1.5. Implement Dynamic Log Levels

1.6. Implement Log Rotation & Retention (Applicable for HYBRID Applications (OnPrem Runtime) ONLY)

1.7. Include Version Information in Logs

1.8. Build Custom Logging Components

GitHub - mulesoft-consulting/json-logger: Drop-in replacement for default Mule Logger that outputs…

Drop-in replacement for default Mule Logger that outputs a JSON structure based on a predefined JSON schema …

1.9. Use Contextual Logging

1.10. Configure Message Logging Policy

1.11. Implement Centralized/External Logging

1.12. Leverage Anypoint Monitoring Logs Feature

2. Monitoring

2.1. Leverage Built-in Monitoring & Alerting Tools

2.1.1. Runtime Manager

2.1.1.1. Monitoring Dashboards

2.1.1.2. Alerts

2.1.2. Anypoint Monitoring

2.1.2.1. Monitoring Dashboards

2.1.2.2. Alerts

2.1.2.3. API Functional Monitoring

2.1.3. API Manager

2.1.3.1. Manage Alerts

2.2. Integrate with External Third Party APM Tools

3. Distributed Tracing

3.1. Standardize Trace and Span IDs

3.2. Comply with OpenTracing Standard

3.3. Enable Trace Logging for Policies

4. Exception Tracking

4.1. Use Structured Exception Handling

4.2. Log Detailed Error Information/Enrich Error Information with Custom Data

4.3. Implement Notifications for Critical Errors

4.4. Monitor and Analyze Exceptions

4.5. Implement Error Propagation Strategies Correctly

Conclusion

References

MuleSoft Documentation

MuleSoft Documentation Site

What is Distributed Tracing? A Friendly Guide for Developers | Aspecto

You will learn what distributed tracing is, why logs are not enough to debug your microservices, and what OpenTelemetry…

Optimizing Integration Performance: Implementing Custom Monitoring and Graphing in MuleSoft

In current dynamic business landscape, the ability to monitor and analyze integration performance is crucial for…

Mulesoft Datadog Integration

A quick review of the steps involved in integrating Mulesoft with Datadog to collect the logs.

Mulesoft Cloudhub (Extension v2) monitoring & observability | Dynatrace Hub

Gather in depth metrics and topology from your Anypoint Cloudhub Organizations, Environments, Applications, APIs and…

Dynatrace Monitoring Setup for MuleSoft

Why Dynatrace :

MuleSoft and Application Performance Monitoring: Optimising Software Performance with Dynatrace …

Introduction Software applications are the backbone of many businesses and...

Help And Training Community

Edit description

Help And Training Community

Edit description

Help And Training Community

Edit description

Help And Training Community

Edit description

Written by Praveen Sundar K. S.