Dynatrace vs Datadog: Metrics out of the box
In the previous blog, we looked at the various steps required to install Dynatrace and Datadog in two identical Ubuntu server. In this blog, lets see how the products provide metrics out of the box
What is a metric?
Metric is a numerical value represented over time. It could be resource utilisation, error rates, latency to user signups.
Infrastructure components refer to CPU, Memory, Disk and Network of the server. Traditionally they were physical components but they play critical roles in the overall performance of an application that is hosted on a server. Every application has to live somewhere physically and the physical attributes like the size of CPU, Memory, Disk and Network matters.
Lets take a look how infrastructure metrics are first seen in Dynatrace and Datadog and later how they respond to software installation. In this case, we will install Apache a web server on the host.
Dynatrace — Host metrics:
All the hosts that have an agent successfully installed would be listed in the hosts section in Dynatrace.
The section now has the host names and a summary of the utilisation. Clicking on the host opens additional information about the server:
The various infrastructure components CPU(1), Memory(2), NIC(3), Disks (4), Graphs related to any of the above 4 are shown in (6).
(5) shows the various network services. In this example, it showed the DNS errors and queries.
(7) is a button that showed the consuming processes as I had selected CPU in the previous example. Selecting the disks would show the contributing disks and their usage and other metrics like IOPS, Response times and more.
The recent Problems(8), Availability / Run time(9), the top processes running are shown.
(11) shows the recent events like software installation or configuration changes. Finally (12) lists all the logs that are available to use.
Datadog — Host Metrics:
To get host metrics on Datadog, click on Infrastructure(1) to list all the infrastructure. The front page shows CPU, IO wait in a summarised listing (2)
Clicking the host(1) opens a pop-up within the page with detailed information in a tabulated format.
The first tab provides the host metadata:
Clicking on the ‘Metrics’ tab shows the available metrics. There is an ‘Open in Host Dashboard’ tab that shows additional information:
There is a container tab as well as this server did not have any containers we are going to skip that.
Datadog — Processes not shown Issue:
Unlike Dynatrace, Datadog has the agent with Processes and Containers’ collection disabled by default. This caused the processes not to be shown.
Datadog — Processes not shown Fix:
To fix the above problem where processes are not reported, I had to login to the server, update the datadog.yaml file to enable processes.
File to update /etc/datadog-agent/datadog.yaml
Section:
process_config:
enabled: 'true'Source: https://docs.datadoghq.com/infrastructure/process/?tab=linuxwindows
After restarting the datadog agent, I could see the processes:
With this available in Datadog, lets find out how does this table compare in Dynatrace.
Unlike processes shown in Datadog, Dynatrace provides contextualised process. In Addition, there is a graph that shows the utilisation trend.
Is it possible to see all processes in Dynatrace?
Dynatrace can show all processes too. This can be set up in declarative process grouping.
Here is an example that shows snapd, polkit, ssh and rsyslogd processes shown.
Datadog — Network performance not shown issue:
In the next section, Network performance monitoring wasn’t available in Datadog. Similar to processes, they are disabled by default. To enable, we would need to follow instructions provided in the link.
Datadog — Network performance not shown fix:
I had to follow the instructions provided in the link. Unfortunately, it wasn’t seamless. I encountered few issues. The command provided in the documentation tries to run under the user dd-agent but gets a permission denied error
I initially thought there was a reason to use dd-agent but I later copied it with sudo which helped in copying the file but the file was still read-only which was a bit annoying. I double checked the documentation if I had missed a step or installed the agent without sudo in the first place was the cause but I couldn’t find any.
Nevertheless, I changed the mode of the file to writeable using the chmod command to go the next step
In the next step, I was now able to edit the system-probe.yaml file as required in the documentation:
I had to restart the agent after modifying the file:
It took lot time to see the graphs but in the end I could see Network graphs available:
Dynatrace — Network utilisation:
Network utilisation was available out of the box in Dynatrace. No configuration was needed on the Console or the agent.
Dynatrace — Logs:
Dynatrace had the Logs automatically available out of the box and all I had to do on the tool was to choose the logs I am interested. I had installed Apache on both the machines to see if logs can be seen and filtered.
Searching's in logs and creating custom metrics were seamless in Dynatrace. We will try the search functionality once we configure logs from Datadog.
Datadog — Logs
Logs are disabled in Datadog by default. To enable logs, I would need to enable it on the agent level by updating the datadog.yaml file.
## @param logs_enabled - boolean - optional - default: false
## Enable Datadog Agent log collection by setting logs_enabled to true.
#logs_enabled: true
For some reason the syslogs were not seen.
I tried to follow the instructions provided but still didn’t seem to work. After spending hours I finally figured out that the permissions of the file did not allow datadog agent to read and ship to datadog. I had to modify the permissions using the chmod command
sudo chmod 655 syslogWith the permissions sorted, I could finally see the syslogs.
Sending Apache Logs:
Similar to syslog issue, sending Apache logs were not readily available. The good news was that since we enabled Live processes in the previous step, Datadog could detect the new application (apache) that was installed.
However, the Apache logs required additional settings:
Clicking the APACHE icon still directed me to do additional configuration:
As instructed, I updated the conf.yaml in apache.d directory.
This next step is the most frustrating part. I followed the documentation and couldn’t find why my logs were not visible but again, similar to the syslog issue, datadog agent did not have permissions to read the log file and thus I had to update it manually:
With the permissions updated, I could see the Apache logs in the host field:
This was also available in the Log explorer
Dynatrace — Searching within the logs:
Both have similar capabilities to search and search in context to look for the specific filter in the logs:
Take a look at the search for a 404 error. I was able to get similar results in Both Dynatrace and Datadog
Datadog — Searching within the logs:
Application performance:
Dynatrace — Basic App performance:
Dynatrace automatically detected that there was a webserver installed and started to capture data. Out of the box Dynatrace detected the application installed, the page load times, the load actions, Multi-dimension performance and tracing again without any configuration changes or restart of the agents.
All in all, Dynatrace automatically detected the application, started to instrument it, benchmarked the usage and provided tracing with no configuration done.
Datadog — Basic App performance:
Datadog provides metrics of Apache logs after the conf.yaml files updated and after restarting the agent:
There were no other information in addition to the above available out of the box from Datadog.
Conclusion:
Dynatrace had all the infrastructure metrics, networking metrics and logs readily available. Logs were automatically detected and the option to choose was a simple click of a button in the console. All the metrics are available out of the box with no need to modify the hosts agent files, folders or permissions. The data out of it was very good and in fact contextualised. In addition, Dynatrace was able to detect an application(Apache) and automatically benchmarked it, provided tracing with absolutely no configuration changes done anywhere.
Datadog had very limited infrastructure metrics (CPU, Memory, ntp offset and load averages) available out of the box. Network, process and logs required additional configurations. The changes were quite low level and time consuming. Each new application that gets installed (apache in this case) required a separate configuration file (located in /etc/datadog-agent/conf.d/apache.d/conf.yaml) to be updated to monitor the software.
Overall, Datadog required lot of configurations to be done on the host level, updating the config file, modifying file permissions, restarting the application and the agent. Dynatrace had no configuration changes needed on the host end, application detection and tracing were available to be observed as soon as they were installed.
Continue comparing:
Dynatrace vs Datadog: Installing the agents
Dynatrace vs Datadog: Basics
Dynatrace vs Datadog: Metrics out of the Box
Dynatrace vs Datadog: SaaS and On-prem deployment offerings
Disclaimer: The author has been a Dynatrace and Datadog user in his previous job with variety of clients. Both products are constantly adding features and some features would change in the future. The author did this review during his interview process with both the companies. The author now works for Dynatrace.
