How the send logs from NodeJS to Datadog using Winston.

Introduction

Logging is a crucial aspect of application development and maintenance. It helps in monitoring, debugging, and understanding the behavior of our applications. In this tutorial, we will explore how to send logs from a Node.js application to Datadog, a popular cloud monitoring platform, using the Winston logging library. By integrating Winston with Datadog, we can gain valuable insights into our application’s performance and troubleshoot issues effectively. Let’s dive in and learn how to set up this logging system.

Prerequisites: Before proceeding, ensure that you have the following prerequisites in place:

Step 1: Set Up a Node.js Project Begin by creating a new Node.js project or using an existing one. Initialize a new project by running the following command in your project directory:

npm init -y

Step 2: Install Required Dependencies We need to install two packages: winston, winston-transport, axios, dotenv. These packages will enable us to log messages using Winston and send them to Datadog. Run the following command to install the dependencies:

npm install winston winston-transport axios dotenv --save

Don’t forget to install Typescript and other dependencies

npm install typescript nodemon ts-node --save-dev

Step 3: Configure Winston with Datadog, for that create a few files getEnvConfig.ts, index.ts, winston.ts and then add the following code to set up the Winston logger with Datadog integration:

//getEnvConfig.ts
import * as dotenv from 'dotenv';
dotenv.config();

const ENV = process.env.ENV || 'DEV';

export const getEnv = () => {
    return ENV;
};

export const getLogLevel = (): string => {
    return process.env.LOG_LEVEL || 'info';
};

export const getDataDogAppKey = (): string => {
    return process.env[`DATADOG_APP_KEY_${ENV}`] || '';
};

export const getDataDogApiKey = (): string => {
    return process.env[`DATADOG_API_KEY_${ENV}`] || '';
};

export const getApplicationName = (): string => {
    return `${process.env.APPLICATION_NAME || ''}_${ENV}`;
};

export const getLogHostName = (): string => {
    if (ENV === 'PROD' || ENV === 'QA') {
        return `cloud_${ENV}`;
    }

    if (ENV === 'DEV') {
        return `localhost_${ENV}`;
    }

    return 'localhost';
};

export const getEnableLogs = (): boolean => {
    if (process.env.DATADOG_ENABLED_LOGS=== 'true') {
        return true;
    }
    return false;
}

//index.ts
export { logger as LOGGER } from './winston';

//winston.ts
import { createLogger, format, transports } from 'winston';
import axios from 'axios';
import Transport from 'winston-transport';
import {
    getDataDogApiKey,
    getApplicationName,
    getLogHostName,
    getEnv,
    getEnableLogs,
} from './getEnvConfig';

const DATADOG_API_KEY = getDataDogApiKey();
const APPLICATION_NAME = getApplicationName();
const LOG_HOST_NAME = getLogHostName();
const ENV = getEnv();
const ENABLE_LOGS = getEnableLogs();
const PATH = `/api/v2/logs?dd-api-key=${DATADOG_API_KEY}&ddsource=nodejs&service=${APPLICATION_NAME}`;

const httpTransportOptions = {
    host: 'https://http-intake.logs.datadoghq.com',
    path: PATH,
    ssl: true,
    hostname: LOG_HOST_NAME,
    service: APPLICATION_NAME,
    ddsource: 'nodejs',
    ddtags: `env:${ENV}`,
};

const { combine, timestamp, json, errors } = format;
const errorsFormat = errors({ stack: true });

const datadogTransporter = async (payload: any) => {
    if (ENABLE_LOGS === false) {
        return;
    }

    const { level, message, timestamp, metadata, sendLog } = payload;
    const messageDate = `[${APPLICATION_NAME}]${message}[${new Date().toISOString()}]`;

    if (sendLog || level === 'error' || level === 'warn') {
        const data = [
            {
                level: level,
                message: messageDate,
                service: httpTransportOptions.service,
                metadata: metadata,
                ddsource: httpTransportOptions.ddsource,
                ddtags: httpTransportOptions.ddtags,
                timestamp: timestamp,
            },
        ];

        return axios
            .post(
                `${httpTransportOptions.host}${httpTransportOptions.path}`,
                data,
                {
                    headers: {
                        'Content-Type': 'application/json',
                    },
                }
            )
            .then((response) => {
                console.log('Response on transport success', response);
            })
            .catch((error) => {
                console.log('Error on transport', error);
            });
    }
};

class CustomTransport extends Transport {
    log(payload: any, cb: Function) {
        //Call datadog messages
        datadogTransporter(payload);
        cb(null);
    }
}

const logger = createLogger({
    level: 'info',
    exitOnError: false,
    format: json(),
    transports: [
        new transports.Console({
            format: combine(timestamp(), json(), errorsFormat),
        }),
        new CustomTransport({
            format: combine(timestamp(), json(), errorsFormat),
        }),
    ],
});

export { logger };

Step 4: Setup your Datadog api key

Create a .env file in the root of your project like following :

ENV=DEV

# Logger
DATADOG_ENABLED_LOGS=true
APPLICATION_NAME=DataDogAppTest
DATADOG_API_KEY_DEV=YOUR_DATADOG_API_KEY

Make sure to replace YOUR_DATADOG_API_KEYand APPLICATION_NAME

Start to logging

Note : https://app.datadoghq.com/ could change depend on your selected region.

Now we can create a simple index.ts to test our logging transporter.

import { LOGGER } from './logger';
const runApp = async () => {
    try {
        LOGGER.info(`[runApp][App is running]`, {
            metadata: '',
            sendLog: true,
        });
        LOGGER.warn(`[runApp][App WARN Test]`, {
            metadata: '',
            sendLog: true,
        });

        throw new Error('Error on run app');
    } catch (error: any) {
        LOGGER.error(`[runApp][Error on run app]`, {
            metadata: { error: error, stack: error.stack.toString() },
        });
    }
};

runApp();

If you setup your datadog agent properly you should see something like this: https://app.datadoghq.com/infrastructure/map

Datadog infra map host

You can check your agent status on : http://127.0.0.1:5002/

Datadog agent status

Now go to https://app.datadoghq.com/logs/onboarding/server and click on other then click on “Explore your Logs”.

Logs agent configuration

As you can see in the image bellow here we have our logs from our NodeJs application. You can try different filters and tags.

Datadog logs explorer

Datadog log explorer details

Conclusion

In this tutorial, we learned how to send logs from a Node.js application to Datadog using the Winston logging library. By integrating Winston with Datadog, you can centralize your logs and gain insights into your application’s performance. Logging is an essential tool for monitoring and troubleshooting, and with this setup, you can easily track and analyze your application’s behavior. Start implementing logging system in your Node.js projects and take control of your application’s logs.

You can find full code here GitHub.