Collecting Open Metrics In K8s And Shipping To Datadog

At Beam, we have a lot of metrics that we collect of which we utilize in pretty standard ways. For instance, we collect CPU and Memory metrics and utilize those for our horizontal pod auto scalers. We also collect custom metrics, like how many claims we’ve processed over a given amount of time so we can alert if the number of claims during a set period drops below an expected amount. Beam also utilizes Datadog for our metric and log aggregation so it made sense for us to ship custom metrics to Datadog.

Currently, we utilize the prometheus_exporter gem by Discourse to surface our metrics to Prometheus and wanted to continue to surface a subset of those metrics to datadog in the same way. The reasoning behind only surfacing a subset of the metrics to Datadog came down to one thing, cost. We have hundreds of metrics that are sent to Prometheus and we just wanted metrics in Datadog to facilitate alerting in our slack channels and bubbling those alerts into Victorops in cases where human intervention needed to happen.

To get started with this tutorial, you’ll need a few things:

• Working knowledge of Ruby
• A Kubernetes cluster and knowledge of how to set up Kubernetes manifests.
• A Datadog account and the Datadog k8s agent installed to your cluster.

Installing Prometheus Exporter

To make the tutorial as a whole more visible, I have set this up utilizing the Sinatra Ruby framework. The first step of setting this all up is getting the Prometheus exporter gem installed. With bundler this is as simple as running:

bundle add prometheus_exporter and then bundle install

Once installed, you can then create a client registry in your application and bind the metrics server to 0.0.0.0:9394 . In app.rb:

require 'sinatra'
require 'prometheus_exporter'
require 'prometheus_exporter/server'
# client allows instrumentation to send info to server
require 'prometheus_exporter/client'
require 'prometheus_exporter/instrumentation'
# bind is the address, on which the webserver will listen
# port is the port that will provide the /metrics route
server = PrometheusExporter::Server::WebServer.new bind: '0.0.0.0', port: 9394
server.start
# wire up a default local client
PrometheusExporter::Client.default = PrometheusExporter::LocalClient.new(collector: server.collector)

Once this is all in place, you can then view empty metrics are being served up by running: bundle exec prometheus_exporter -b 0.0.0.0 and then opening your browser to localhost:9394 (we will call this the metrics tab). This should show you an empty metrics page because no metrics have been exported yet.

Setting Up Your First Metric

Now that Prometheus exporter is configured, let’s go ahead and add a metric that will count HTTP requests. Appending to app.rb above, add the following:

http_requests = PrometheusExporter::Metric::Counter.new("http_requests", "A counter of HTTP requests made")
server.collector.register_metric(http_requests)
get '/' do
  http_requests.observe(1)   
  content_type 'text/html'
  "hello world"
end

Now you can start up your Sinatra app with the following command: ruby app.rb -p 3000 -o 0.0.0.0

Once the application is started, in a new tab you can open your browser to localhost:3000 (we will call this the app tab) which should print out hello world . If you switch back to your metrics tab and refresh you will then see

# TYPE http_requests counter
# HELP http_requests A counter of HTTP requests made
http_requests 1.0

The more you refresh your app tab, the higher the http_requests counter will go.

Sending To Datadog

Now that we are collecting HTTP request metrics, we will want to ship these metrics to datadog.

To do this, set up a pod with two containers. One that will accept your production traffic and one that will handle metrics:

apiVersion: v1
kind: Pod
metadata:
  name: sinatra
  labels:
    app.kubernetes.io/name: sinatra
  annotations:
    ad.datadoghq.com/sinatra.check_names: |
      ["openmetrics"]
    ad.datadoghq.com/sinatra.init_configs: |
      [{}]
    ad.datadoghq.com/sinatra.instances: |
      [
        {
          "prometheus_url": "http://sinatra-metrics.sinatra.svc.cluster.local/metrics",
          "namespace": "sinatra",
          "metrics": ["http_requests"],
          "tags": [
            {"namespace": "sinatra"},
            {"app": "sinatra"},
            {"environment": "production"}
          ],
          "ssl_verify": "false",
          "ssl_ignore_warning": "true"
        }
      ]
spec:
  containers:
    - name: sinatra-metrics
      image: myawesomesinatrarepo/sinatratutorial
      command:
        - bundle
        - exec
        - prometheus_exporter
        - -b
        - 0.0.0.0
    - name: sinatra-app
      image: myawesomesinatrarepo/sinatratutorial
      command:
        - ruby
        - app.rb
        - -p
        - 3000
        - -o
        - 0.0.0.0

The real magic happens with the pod’s annotations. The Datadog Kubernetes agent is looking for pods that have the ad.datadoghq.com/<podname>.<config_option> annotations and utilizes them for autodiscovery of metrics. One thing that I found hard to find when setting this up at Beam was finding all available <config_option>'s which can be found here.

The most important part of this piece was the "metrics": ["http_requests"] bit, since this is datadogs whitelist of metrics. It will only pull metrics that are in this whitelist and it allows you to use * as a wildcard. So you can pull all ["http_requests", "ruby*"] metrics or even ["*"] all metrics.

The final part of this entire thing is that you will need a service configured so that datadog can hit your metrics endpoint from anywhere within your cluster by utilizing the in-cluster DNS of your service:

apiVersion: v1
kind: Service
metadata:
  name: sinatra-metrics
spec:
  clusterIP: 172.20.20.20
  ports:
  - name: metrics
    port: 80
    protocol: TCP
    targetPort: 9394
  selector:
    app.kubernetes.io/name: sinatra
  sessionAffinity: None
  type: ClusterIP

Once the service is in place, you should now see your custom whitelisted OpenMetrics flowing to Datadog within seconds!