MuleSoft Flex Gateway log forwarding to Datadog

In this article, we will explore the capabilities of MuleSoft Anypoint Flex Gateway to forward logs to an external third party logging system. Forwarding logs to external logging systems can be very helpful to improve the observability of distributed systems such as APIs, microservices, serverless functions, etc. By using tools like Datadog or Splunk, developers and system administrators can create dashboards that capture information that can be helpful to diagnose problems.

Flex Gateway uses Fluent Bit 1.8.X, so it can send logs to any output supported by Fluent Bit, such as Amazon CloudWatch, Amazon S3, Datadog, New Relic, Splunk, Prometheus, etc.

Configure Datadog

Datadog is an observability service for cloud-scale applications that provides monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform. It provides several pricing plans, including a Free tier, Pro and Enterprise. The free tier offers a 1-day metric retention and up to 5 hosts to be monitored.

There are the two steps required to start using Datadog to collect logs from Flex Gateway:

• Sign up to Datadog.
• Obtain an API key by navigating to Organization Settings > Access > API Keys.

This screenshot shows how to get an API Key:

Configure Flex Gateway

Flex Gateway can be installed as a Linux service, in a Docker container or as a Kubernetes Ingress controller. All those configurations support forwarding logs to a third party service. In this article, we are using Docker.

To configure Fluent Bit, we just need to create a folder named “app” exactly in the same directory where the file registration.yaml lies. In that folder “app” we are going to create a file named “logging.yml”, as shown in this screenshot:

You can find in this documentation page an example of a YAML file that can be used to configure log output. The most important sections of that file are:

• Metadata.Name: it has to be configured as “logging-config”
• Spec.Logging.Outputs.Type: In this example we are using the value “datadog” but it could be configured with other values such as “splunk”
• Spec.Logging.Outputs.Parameters.Host: This parameter is specific to Datadog and identifies the host that will receive the logs. In this example we are using the European Union host of Datadog: “http-intake.logs.datadoghq.eu”
• Spec.Logging.Outputs.Parameters.TLS: Specific to Datadog and configured to “on”
• Spec.Logging.Outputs.Parameters.Apikey: Also specific to Datadog. Contains the key obtained in the previous step.

This is an example of a YAML configured to send logs to Datadog:

apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
  name: logging-config
spec:
  logging:
    outputs:
    - name: default
      type: datadog
      parameters:
        Host: http-intake.logs.datadoghq.eu
        TLS: on
        apikey: b4286a9ceb29301da63e72465b49f***
    runtimeLogs:
      logLevel: info
      outputs:
      - default
    accessLogs:
      outputs:
      - default

Once that the YAML file is created, it is only required to restart Flex Gateway and automatically it will start sending logs to Datadog.

Configure Message Logging policy

We can take an additional extra step to increase the amount of information sent to Datadog, by enabling the Message Logging policy. This step is not mandatory, but it can be helpful to diagnose issues when invoking APIs in Flex Gateway. Please note, that — at least in version 1.3 of Flex Gateway — it only supports a subset of Dataweave in policy configuration.

By clicking in API Manager and selecting an API published in Flex Gateway, we can configure Message Logging to dump the headers or the body of the requests or responses of that API, either before or after calling the API. This screenshot shows how to write the request headers in the logs.

Testing

Once we have configured Flex Gateway with both the forward logging YAML configuration and the Message Logging policy, we can use Postman to trigger a call to the gateway. In this case, we are using the Star Wars API:

After few seconds, we will see in the Logs section of Datadog a new log entry, containing information about all the requests sent to the Star Wars API:

Summary

Configuring an external third party logging system can be very helpful to improve the observability of distributed systems, and Flex Gateway by using Fluent Bit can send logs to tools such as Splunk or New Relic. In this article, we have seen how to create a YAML to configure Flex Gateway to write logs to Datadog and how to use the Message Logging policy to enrich the amount of data captured per every request.