Tomcat JVM metrics monitoring using Prometheus in Kubernetes

Monitoring metrics and runtime characteristics of an application server is essential to ensure the adequate functioning of the applications running on that server, as well as to prevent or resolve potential issues in a timely manner. As far as Java applications go, one of the most commonly used web application servers is Apache Tomcat, which will be the focus of this article.

Tomcat performance monitoring in Kubernetes cluster can be done either by relying on JMX beans or other tools available in the market. We have listed the prerequisite with which this monitoring can be achieved.

1. Prometheus Agent
2. Kubernetes YAML for application
3. Annotations
4. Scraping Policy
5. Prometheus Server

Prometheus Agent

Prometheus provides a collector that can scrape and expose the mBeans of a JMX target. It runs as a Java Agent, exposing an HTTP server and serving metrics of the local JVM. It can also be run as an independent HTTP server and scrape remote JMX targets, but this comes with various disadvantages, such as it being harder to configure and it being unable to expose process metrics (e.g., memory and CPU usage). Running the exporter as a Java Agent is thus strongly encouraged.

The following Dockerfile adds support for the options we’ve listed.

FROM tomcat
RUN mkdir /data
ADD jmx_prometheus_javaagent-0.7-SNAPSHOT.jar /data/jmx_prometheus_javaagent-0.7-SNAPSHOT.jar
ADD prometheus-jmx-config.yaml /data/prometheus-jmx-config.yaml
CMD java -javaagent:/data/jmx_prometheus_javaagent-0.7.jar=8088:/data/prometheus-jmx-config.yaml -jar $TOMCAT_HOME/start.jar

• Connect to an exposed JMX port of the JVM (not recommended)
• Java agent version (recommended)

If you use the agent version, you’ll have to modify three configuration options:

• the Jar file location
• the port for the http(s) interface, where the metrics will be available for scraping, already in Prometheus friendly format
• additional configuration options

An example looks like this:

-javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent-0.7.jar=8088

The latest jar can be downloaded from the following link:

wget http://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.7/jmx_prometheus_javaagent-<latest-version>.jar

Create prometheus-jmx-config.yaml with following config in it

---
hostPort: 127.0.0.1:1234
lowercaseOutputName: false
lowercaseOutputLabelNames: false

The complete options for the prometheus-jmx-config.yaml is available here.

Kubernetes YAML for application

When you run the tomcat image in Kubernetes infrastructure, the JMX port needs to be exposed to scrape the data from the container. Following sample tomcat YAML file shows how to deploy the basic tomcat service in K8s infrastructure with prometheus JMX port exposed and annotated.

apiVersion: v1
kind: Service
metadata:
  name: <service name>
  labels:
    app: <service name>
spec:
  ports:
  - port: 8080
    targetPort: 8080
    name: http
  - port: 8088
    targetPort: 8088
    name: prom
  selector:
    app: <service name>
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: <deployment name>
spec:
  # this replicas value is default
  # modify it according to your case
  replicas: 1
  template:
    metadata:
      labels:
        app: <deployment name>
        type: java
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '8088'
    spec:
      containers:
      - name: <container name>
        image: <docker image>
        ports:
        - containerPort: 8080
        env:
        - name: JAVA_AGENT_PORT
          value: "8088"
        resources:
          requests:
            memory: "1024Mi"
            cpu: "1"
          limits:
            memory: "1024Mi"
            cpu: "1"

Annotations

Annotations on pods allow a fine grained control of the scraping process:

• prometheus.io/scrape: The default configuration will scrape all pods and, if set to false, this annotation will exclude the pod from the scraping process.
• prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
• prometheus.io/port: Scrape the pod on the indicated port instead of the pod’s declared ports (default is a port-free target if none are declared).

These annotations need to be part of the pod metadata. They will have no effect if set on other objects such as Services or DaemonSets. The above Deployment manifest will instruct Prometheus to scrape all of its pods on port 8088.

Scraping Policy

To minimise the amount of configuration required from the user, the default setup will try to scrape all pods present in the system. This may have some undesirable side-effects such as:

• Prometheus issuing HTTP GET requests to TCP ports not even exposing an HTTP server.
• Prometheus issuing HTTP GET requests to fetch /metrics on services that may not export any metric, polluting logs.

While the annotations described in the previous section allow the user to opt-out of the scraping process, we also offer an alternative Prometheus configuration that only scrapes a minimal set of system metrics and requires the user to opt-in on a per-pod basis. In other words, in this mode the prometheus.io/scrape: true annotation is required for Prometheus to scrape the pod.

Now that we have all the required configs in place to scrape the JVM data from the tomcat container. Let see how it can be scraped via prometheus server.

Prometheus Server

Prometheus is a monitoring platform that collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. Please use the following docker-compose.yaml for downloading the Prometheus Image

---
version: '2'
services:
  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    restart: always
    ports:
      - "9090:9090"
    volumes:
    - /data/prometheus/prometheus.yml:/prometheus/prometheus.yml:ro
    command: [ '--config.file=/prometheus/prometheus.yml', '--web.enable-lifecycle', '--web.enable-admin-api', '--storage.tsdb.retention=1y' ]

Configuring Prometheus

The Prometheus download comes with a sample configuration in a file called prometheus.yml

global:
  scrape_interval:     15s
  evaluation_interval: 15s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: 'prometheus
    scrape_interval: 30s
    kubernetes_sd_configs:
    - role: pod
      namespaces:
        names:
        - staging
    scheme: http
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name

There are three blocks of configuration in the example configuration file: global, rule_files, and scrape_configs.

• The global block controls the Prometheus server's global configuration. We have two options present. The first, scrape_interval, controls how often Prometheus will scrape targets. The evaluation_interval option controls how often Prometheus will evaluate rules.
• The rule_files block specifies the location of any rules we want the Prometheus server to load. For now we've got no rules.
• The last block, scrape_configs, controls what resources Prometheus monitors. Since Prometheus also exposes data about itself as an HTTP endpoint it can scrape and monitor its own health. Prometheus expects metrics to be available on targets on a path of /metrics. So this default job is scraping via the URL: http://<hostname>:9090/metrics.

When you scrape data from the containers running inside the Kubernetes cluster, don’t forget to mentions the namespace from where the data should be scraped as mentioned in the above config file.

Once you have all prerequisites configured and deployed, the JVM metrics will be collected and stored in the prometheus. We use Grafana to create a graphical dashboard and measure the performance as shown below.

Kube JVM Metrics

References

JVM overview - Prometheus dashboard for Grafana

prometheus/jmx_exporter

Collect Docker metrics with Prometheus