Boost ClickHouse Observability with OpenTelemetry: A Comprehensive Guide

Since ClickHouse DB has become increasingly popular in recent times, it has become crucial to monitor its performance. With OpenTelemetry emerging as the new standard for observability, it is essential to have a quick guide on how to set it up for ClickHouse DB.

I have set up this configuration on my local Ubuntu machine. Both ClickHouse and the collector are running on the same machine, but they can also be run on separate systems.

Configuring ClickHouse

To expose ClickHouse metrics so that they can be imported into the OpenTelemetry Collector, we need to modify the ClickHouse configuration file (/etc/clickhouse-server/config.xml) by adding the following code:

<prometheus>
     <endpoint>/metrics</endpoint>
     <port>9126</port>
     <metrics>true</metrics>
     <events>true</events>
     <asynchronous_metrics>true</asynchronous_metrics>
 </prometheus>

Select an unused TCP port and enable it in the Linux firewall before entering it in the configuration file. After adding the provided section to the ClickHouse server configuration, restart the service. ClickHouse will expose a new endpoint on the specified port in the format http://localhost:9126/metrics

ClickHouse Metrics

Configuring OpenTelemetry Collector

We’ll utilize Prometheus’s scraper to gather metrics from the endpoint exposed by ClickHouse DB. If the Prometheus receiver isn’t already included in the OTEL configuration, add it, or include this scraping process.

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: clickhouse
          scrape_interval: 10s # Adjust according to your need
          static_configs:
            - targets: ['localhost:9126']

Restart the collector and you should be able to monitor all ClickHouse DB metrics. The metrics list is extensive, so you can select the ones that are relevant to your monitoring needs.