Globant
Published in

Globant

How to deliver your logs/metrics to Datadog using Amazon Kinesis Data Firehose

AWS Data and Analytics services have evolved continuously; recently, I have seen that Amazon Kinesis Data Firehose allows third-party destinations such as Dynatrace, Datadog, and NewRelic, among others. This new integration will allow us to take our log and metric flows easily to these providers.

Some of the third party integrations

When I saw this new release, I began to work on an architecture that I had previously implemented, which you can see at this link. Since we had an on-premise ingest of many logs to AWS through a VPN. I decided then to test the new functionalities and implement the integration with datadog to see how it made the logs reach Datadog without taking advantage of what was previously implemented.

I made the integration with Datadog to test this new functionality and be able to take the advantages that this tool gives us in:

  • Log parsing.
  • Log enrichment.
  • Generation of metrics.
  • Filtering and prioritization.

How does the integration work?

The integrations with these providers are all done through an API KEY; you have to select the correct endpoint and use the API KEY generated from the service.

Datadog Integration

1- Go to the Datadog Configuration page.

2- Create an API KEY.

3- Create Amazon Kinesis Data Firehose and select the endpoint for Datadog.

4- Select the HTTP endpoint URL.

5- Put the API KEY from step 2.

Check the Integration

Check in the Amazon Kinesis Data Firose monitoring tab the Records read from Amazon Kinesis Data Stream (Sum).

Check on Datadog logs. The logs in the image are generating from the sample KPL.

Once you have all the logs in Datadog, you can take advantage of all the functionalities it provides.

Final Architecture and conclusions

In the architecture above, Amazon Kinesis Data Streams receive the information from an on-premise application running the KPL. This information is transmitted over the VPN and using a VPC endpoint; it reaches the Kinesis data streams. It feeds the entire Amazon Kinesis Data Firehose flow to reach the Datadog.

1- Integrations for managing logs and metrics are easier through Amazon Kinesis Data Firehose.

2- All integrations must use an API KEY; you must consult third-party provider documentation for this.

3- These integrations will allow you to take advantage of the functionalities provided by these third-party providers in the management of logs and monitoring.

4- Data that could not be sent to the third-party provider can be stored in a local bucket.

5- It will be possible to centralize N amount of log sources in an Amazon Kinesis Data Stream and use Amazon Kinesis Data Firehose to be sent to Datadog; this use case is widespread since many companies that use these tools from other providers as their monitoring and logging core.

6- In this post, I talk about Dynatrace, Datadog, and NewRelic. Still, you have to consider other third-party integrations such as LogicMonitor, MongoDB Cloud, Splunk, and Sumo Logic.

7- Many will wonder why not directly install the Datadog agent on the instance and avoid the use of Kinesis? In this scenario, using Amazon Kinesis Data Stream and Amazon Kinesis Data Firehose allows more flows within AWS that can use the logs, such as a flow that feeds S3 with the logs use Amazon Quicksight and visualize the information.

Carlos Zambrano

--

--

--

Our thoughts as a strategic disruptor in business and cognitive transformation. Visit us at www.globant.com

Recommended from Medium

Kubernetes Command Line Tools

Learning on Mistakes

𝐂𝐚𝐬𝐞 𝐒𝐭𝐮𝐝𝐲 :- Airbus using Openshift

Q&A on Spark Token distribution event.

peaq partners with Subsquid

peaq and subsquid partnership

Delivery Lead

Nigeria “tech” Twitter

BASIC TERMINAL COMMAND

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Carlos Andres Zambrano Barrera

Carlos Andres Zambrano Barrera

AWS x10, Tech Director en Globant con más de 7 años de experiencia en AWS.

More from Medium

Create a Microsoft SQL Server Instance on AWS’s RDS (Managed Relational Database Service)

How to Create an encrypted AWS RDS Database from an unencrypted Database

How to deploy a MySQL database using AWS RDS — a simple way!

Trigger AWS lambda function with s3 update !