How to deliver your logs/metrics to Datadog using Amazon Kinesis Data Firehose
AWS Data and Analytics services have evolved continuously; recently, I have seen that Amazon Kinesis Data Firehose allows third-party destinations such as Dynatrace, Datadog, and NewRelic, among others. This new integration will allow us to take our log and metric flows easily to these providers.
When I saw this new release, I began to work on an architecture that I had previously implemented, which you can see at this link. Since we had an on-premise ingest of many logs to AWS through a VPN. I decided then to test the new functionalities and implement the integration with datadog to see how it made the logs reach Datadog without taking advantage of what was previously implemented.
I made the integration with Datadog to test this new functionality and be able to take the advantages that this tool gives us in:
- Log parsing.
- Log enrichment.
- Generation of metrics.
- Filtering and prioritization.
How does the integration work?
The integrations with these providers are all done through an API KEY; you have to select the correct endpoint and use the API KEY generated from the service.
Datadog Integration
1- Go to the Datadog Configuration page.
2- Create an API KEY.
3- Create Amazon Kinesis Data Firehose and select the endpoint for Datadog.
4- Select the HTTP endpoint URL.
5- Put the API KEY from step 2.
Check the Integration
Check in the Amazon Kinesis Data Firose monitoring tab the Records read from Amazon Kinesis Data Stream (Sum).
Check on Datadog logs. The logs in the image are generating from the sample KPL.
Once you have all the logs in Datadog, you can take advantage of all the functionalities it provides.
Final Architecture and conclusions
In the architecture above, Amazon Kinesis Data Streams receive the information from an on-premise application running the KPL. This information is transmitted over the VPN and using a VPC endpoint; it reaches the Kinesis data streams. It feeds the entire Amazon Kinesis Data Firehose flow to reach the Datadog.
1- Integrations for managing logs and metrics are easier through Amazon Kinesis Data Firehose.
2- All integrations must use an API KEY; you must consult third-party provider documentation for this.
3- These integrations will allow you to take advantage of the functionalities provided by these third-party providers in the management of logs and monitoring.
4- Data that could not be sent to the third-party provider can be stored in a local bucket.
5- It will be possible to centralize N amount of log sources in an Amazon Kinesis Data Stream and use Amazon Kinesis Data Firehose to be sent to Datadog; this use case is widespread since many companies that use these tools from other providers as their monitoring and logging core.
6- In this post, I talk about Dynatrace, Datadog, and NewRelic. Still, you have to consider other third-party integrations such as LogicMonitor, MongoDB Cloud, Splunk, and Sumo Logic.
7- Many will wonder why not directly install the Datadog agent on the instance and avoid the use of Kinesis? In this scenario, using Amazon Kinesis Data Stream and Amazon Kinesis Data Firehose allows more flows within AWS that can use the logs, such as a flow that feeds S3 with the logs use Amazon Quicksight and visualize the information.
