Using Elastic APM in Kubernetes

“If it isn’t monitored, it isn’t production!”

In a microservice environment handling huge workloads, a performance dip of a single microservice will mean the performance degradation of an entire application. A single weak link in the chain can endanger our SLAs. The SCM product at Logistimo handles mission-critical public-health supply chains for host of countries, via an array of microservices. Since the vitality of these systems is important at many levels, we take monitoring performance very seriously.

Our microservices are managed in a Kubernetes (K8s) cluster. The K8s ecosystem provides an array of options to monitor health and performance of the services. However, we wanted a solution which could monitor the performance of the application at the micro-service level. This brought us to the use of Elastic APM for our Kubernetes cluster. APM stands for Application Performance Monitoring. Deploying Elastic APM in Kubernetes ecosystem is a topic on which not much literature is found on the web. Hence, We decided to share our experience.

Elastic APM system:

If you are using the ELK stack( Elasticsearch, Logstash and Kibana ) for log aggregation in your infrastructure, you might be recording all the application logs in the Elasticsearch. While this in itself is useful, It is necessary to have APM monitoring for microservices. Elastic APM streams all events to Elastic search much like the Logstash.

Since data is stored as raw documents in Elasticsearch, your service metrics can coexist with your infrastructure logs making it easy to explore all of your data in one place. These metrics can be visualised in Kibana with pre-configured dashboards.

To do this APM metrics collection, you need two components:

1. APM server
2. APM agent

APM server

APM Server is a central component in the Elastic APM system. It receives data from APM agents and transforms them into Elastic search documents

A single APM server can handle data from multiple microservices. APM Server works by exposing an HTTP server to which agents post the APM data they collect.

To deploy the basic APM Server in K8s infrastructure, you can use following Kube-apm-server.yaml file:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: apm-server-config
  labels:
    k8s-app: apm-server
data:
  apm-server.yml: |-
    apm-server:
      host: "0.0.0.0:8200"
      frontend:
        enabled: false
    setup.template.settings:
      index:
        number_of_shards: 1
        codec: best_compression
    setup.dashboards.enabled: true
    setup.kibana:
      host: "< kibana endpoint >"
    output.elasticsearch:
      hosts: ["< elasticsearch endpoint >"]
      indices:
        - index: "apm-%{[beat.version]}-sourcemap"
          when.contains:
            processor.event: "sourcemap"
- index: "apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"
          when.contains:
            processor.event: "error"
- index: "apm-%{[beat.version]}-transaction-%{+yyyy.MM.dd}"
          when.contains:
            processor.event: "transaction"
- index: "apm-%{[beat.version]}-span-%{+yyyy.MM.dd}"
          when.contains:
            processor.event: "span"
---
apiVersion: v1
kind: Service
metadata:
  name: apm-server
  labels:
    app: apm-server
spec:
  ports:
  - port: 8200
    targetPort: 8200
    name: http
    nodePort: 31000
  selector:
    app: apm-server
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: apm-server
spec:
  # this replicas value is default
  # modify it according to your case
  replicas: 1
  template:
    metadata:
      labels:
        app: apm-server
    spec:
      containers:
      - name: apm-server
        image: docker.elastic.co/apm/apm-server:6.3.1
        ports:
        - containerPort: 8200
          name: apm-port
        volumeMounts:
        - name: apm-server-config
          mountPath: /usr/share/apm-server/apm-server.yml
          readOnly: true
          subPath: apm-server.yml
      volumes:
      - name: apm-server-config
        configMap:
          name: apm-server-config

Replace the host: “< Kibana endpoint >” and hosts: [“< Elasticsearch endpoint >”] with end points.

APM Server comes with predefined Kibana dashboards and index templates for APM data.

Following is the list of default dashboards available. You can also configure custom dashboards according to your needs.

1. Transactions
2. Span Details
3. Services
4. Errors
5. Error Details

APM predefined Kibana dashboards

These dashboards give you an instant overview of application response times, requests per minutes, error occurrences and more.

Response time and Request per minute metric

APM Agent

The agents are libraries that run inside your application process. Agents automatically instrument your application in order to measure the duration of requests to your service. It also automatically measures things like database queries, cache calls, external http requests and errors. Ensure you add APM agent to all of your micro services to get the metrics.

APM agent for Dockerized Java application:

The first step in getting started with the Elastic APM Java agent is to download the latest release of the agent jar file from maven central.

Add the following code in your Dockerfile to download and set the javaagent environment variable at the build time of the microservice.

RUN wget -P < $TOMCAT_HOME/ > http://repo1.maven.org/maven2/co/elastic/apm/elastic-apm-agent/<latest-version>/elastic-apm-agent-<latest-version>.jar
ENV JAVA_OPTS=-javaagent:$TOMCAT_HOME/elastic-apm-agent-< elastic-apm.version >.jar

Elastic APM agent has lot of configuration options.The most important ones are service_name (required), server_url, and application_packages.

System properties:
-Delastic.apm.service_name=my-service
-Delastic.apm.application_packages=org.example
-Delastic.apm.server_url=http://localhost:8200

The complete Dockerfile entry for apm-agent will look like this

RUN cd $TOMCAT_HOME && wget http://repo1.maven.org/maven2/co/elastic/apm/elastic-apm-agent/<latest-version>/elastic-apm-agent-<latest-version>.jar
 
ENV JAVA_OPTS=-javaagent:$TOMCAT_HOME/elastic-apm-agent-<latest-version>.jar \
              -Delastic.apm.service_name=< Service name > \
              -Delastic.apm.application_packages=< application pakage name > \
              -Delastic.apm.server_url=< APM server url/endpoint >

The service name is used by Elastic APM to differentiate data coming from different services and to group data coming from the same services. When configuring an agent, you need to supply a service name.

The service name can only contain alphanumeric characters, spaces, underscores, and dashes (must match ^[a-zA-Z0–9 _-]+$).

You can set the Service Name and APM server url as the docker environmental variable as the value for these properties changes per environment you deploy.

APM agent for Dockerized Node.js application

In Dockerfile add the follwoing line

RUN npm install elastic-apm-node --save

Agent should be started before you mark other modules as dependency using require in your Node.js application . You will include the following code to enable agent in your application’s main file (usually index.js, server.js or app.js).

// Add this to the VERY top of the first file loaded in your app
  var apm = require('elastic-apm-node').start({
    
    // Set required service name (allowed characters: a-z, A-Z, 0-9, -, _, and space)
    serviceName: '< Service Name>',
// Use if APM Server requires a token
    secretToken: '< Secret token >',
// Set custom APM Server URL (default: http://localhost:8200 )
    serverUrl: '< APM Server URL >',
  })
var app = require('express')()

Provided the correct value for the serviceName and serverUrl for the metrics to be collected.

End Note:

Elastic APM allowed us to avoid developing additional metering and instrumentation inside the micro services. APM agent takes care of monitoring all the key metrics. The data has been immensely useful for us to detect problems early and understand the state of the systems in production.

Happy Monitoring !!