Monitoring JVM Metrics using Grafana, Elasticsearch, Telegraf

What is Jvm metrics?

The JVM Runtime Metrics feature allows you to view heap and non-heap memory, as well as garbage collection activity, for any app that runs inside of the Java Virtual Machine (JVM).

In this journey you need to know what are the products that we are going to use.

Grafana - Grafana is an open-source software. Grafana is best known as a visualization / dashboarding tool focused on graphing metrics from various data sources, such as Elasticsearch, InfluxDB, prometheus.

Elasticsearch - Elasticsearch is a distributed, open source search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. Elasticsearch is built on Apache Lucene.

Telegraf - Telegraf is the open source server agent to help you collect metrics from your stacks, sensors and systems. It can collect metrics from a wide array of inputs and write them into a wide array of outputs. It is plugin-driven for both collection and output of data so it is easily extendable.

I’m using Red Hat Enterprise Linux Server version 7.5. To install these products follow as below.

Install Grafana

Follow the below steps to install Grafana in a single instance.

1. Add new Grafana to the YUM repository by executing following command.

$ cd /etc/yum.repos.d/
$ nano grafana.rep

2. Add the following configurations to the above created “grafana.rep“ repository and save to complete the adding Grafana repository.

[grafana]
name=grafana
baseurl=https://packagecloud.io/grafana/stable/el/7/$basearch
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

3. Once it is done, install Grafana using the following yum command.

$ yum -y install grafana

4. After completion the above reload the systemd manager configuration and start the Grafana service.

$ systemctl daemon-reload
$ systemctl start grafana-server
$ systemctl enable grafana-server

Install Elasticsearch

PREREQUISITES

Following software should be installed in the instance.

• Java 8

INSTALLATION GUIDE

1. Create the repository file

$ sudo curl https://d3g5vo6xdbdb9a.cloudfront.net/yum/opendistroforelasticsearch-artifacts.repo -o /etc/yum.repos.d/opendistroforelasticsearch-artifacts.repo

2. List all available Open Distro for Elasticsearch versions:

$ sudo yum list opendistroforelasticsearch — showduplicates

3. Choose the version opendistroforelasticsearch-1.3.0 and install

$ sudo yum install opendistroforelasticsearch-1.3.0

4. Disable security plugin for http configuration -/etc/elasticsearch/elasticsearch.yml

$ opendistro_security.disabled: true

5. Start Open Distro for Elasticsearch

$ sudo systemctl start elasticsearch.service

6. Send requests to server to verify that Elasticsearch is up and running:

$ curl -XGET https://localhost:9200 -u admin:admin — insecure curl -XGET https://localhost:9200/_cat/nodes?v -u admin:admin — insecure curl -XGET https://localhost:9200/_cat/plugins?v -u admin:admin — insecure

If you encounter following errors you can safely ignore if the elasticsearch service is running, according to the official documentation

elasticsearch[3969]: java.security.policy: error adding Entry:
elasticsearch[3969]: java.net.MalformedURLException: unknown protocol: jrt
elasticsearch[3969]: java.security.policy: error adding Entry:
elasticsearch[3969]: java.net.MalformedURLException: unknown protocol: jrt

7. Start Open Distro for Elasticsearch

$ sudo /bin/systemctl daemon-reload
$ sudo /bin/systemctl enable elasticsearch.service

Where are the files?

• The RPM package installs files to the following locations:

REFERENCES : Elastic search open distro installation guide

Install Telegraf agent

Go to the VM that need to be monitored

1. First you have to add the Influxdata repository to the system. Influxdata provides the repository for installing InfluxDB and telegraf

$ vi /etc/yum.repos.d/influxdb.repo

2. Add following to the file

[influxdb]
name = InfluxDB Repository — RHEL $releasever
baseurl = https://repos.influxdata.com/rhel/$releasever/$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key

3. Update cache to confirm that the repository is working fine(optional). Then install telegraf.

$ sudo yum makecache fast
$ sudo yum -y install telegraf

4. Let’s start the service and enable it to start on boot.

$ sudo systemctl start telegraf
$ sudo systemctl enable telegraf

Configure telegraf for JVM metrics.

1. Config Jolokia in WSO2 EI/ APIM. To download Jolokia osgi agent CLICK HERE.

Download jolokia osgi agent and,

- In WSO2 EI, add it to the <EI-HOME>/dropins/

- In APIM, add it to the <PRODUCT-HOME>/repository/components/dropins/

2. Restart EI/ APIM

$ sh integrator.sh restart

Reference : Monitoring a WSO2 product with Jolokia

3. Jolokia 2 plugin is using the post request of jolokia to extract stats

Example code :

$ curl -X POST \
http://localhost:9763/jolokia/read \
-H ‘authorization: Basic YWRtaW46YWRtaW4=’ \
-H ‘cache-control: no-cache’ \
-H ‘content-type: application/json’ \
-H ‘postman-token: 8222ef18–1e22-a944–05e8–6b8d68ddf225’ \
-d ‘{ “type” : “read”, “mbean” : “java.lang:type=Memory”, “attribute” : “HeapMemoryUsage”, “path” : “used” }’

4. If you are using APIM node this request will give a 403 response due to following error(CSRF attack)

TID: [-1234] [] [2019–09–25 04:24:26,258] WARN {org.owasp.csrfguard.log.JavaLogger} — potential cross-site request forgery (CSRF) attack thwarted (user:<anonymous>, ip:192.168.56.95, method:POST, uri:/jolokia/read/read, error:required token is missing from the request) {org.owasp.csrfguard.log.JavaLogger}

Inorder to allow this request within WSO2 apim

To exclude this url pattern from CSRF protection

Reference : Securing web applications

4.1. Open the catalina-server.xml file from the <PRODUCT_HOME>/repository/conf/tomcat/ directory.

4.2. Set the compression parameter (under each of the connector configurations) to false as shown below.

compression=”off”

4.3. Add jolokia property as follow

$ vi wso2/wso2am 2.6.0/repository/conf/security/Owasp.CsrfGuard.Carbon.properties
org.owasp.csrfguard.unprotected.jolokia=/jolokia/*

5. Config Jolokia in telegraf agent

All configs can be found in Jolokia2 Input Plugins repository

$ vi /etc/telegraf/telegraf.conf

6. Add following

# Global Agent Configuration
[agent]
hostname = “apim-tagent”
flush_interval = “15s”
interval = “15s”
# Input plugin for JVM metrics
[[inputs.jolokia2_agent]]
 urls = [“http://localhost:9763/jolokia"]
 [[inputs.jolokia2_agent.metric]]
 name = “java_runtime”
 mbean = “java.lang:type=Runtime”
 paths = [“Uptime”]
[[inputs.jolokia2_agent.metric]]
 name = “java_memory”
 mbean = “java.lang:type=Memory”
 paths = [“HeapMemoryUsage”, “NonHeapMemoryUsage”, “ObjectPendingFinalizationCount”]
[[inputs.jolokia2_agent.metric]]
 name = “java_garbage_collector”
 mbean = “java.lang:name=*,type=GarbageCollector”
 paths = [“CollectionTime”, “CollectionCount”]
 tag_keys = [“name”]
[[inputs.jolokia2_agent.metric]]
 name = “java_last_garbage_collection”
 mbean = “java.lang:name=*,type=GarbageCollector”
 paths = [“LastGcInfo”]
 tag_keys = [“name”]
[[inputs.jolokia2_agent.metric]]
 name = “java_threading”
 mbean = “java.lang:type=Threading”
 paths = [“TotalStartedThreadCount”, “ThreadCount”, “DaemonThreadCount”, “PeakThreadCount”]
[[inputs.jolokia2_agent.metric]]
 name = “java_class_loading”
 mbean = “java.lang:type=ClassLoading”
 paths = [“LoadedClassCount”, “UnloadedClassCount”, “TotalLoadedClassCount”]
[[inputs.jolokia2_agent.metric]]
 name = “java_memory_pool”
 mbean = “java.lang:name=*,type=MemoryPool”
 paths = [“Usage”, “PeakUsage”, “CollectionUsage”]
 tag_keys = [“name”]
# output for elasticsearch
[[outputs.elasticsearch]]
urls = [ “http://localhost:9200/" ]
index_name = “custom_index_name_jvm-%Y.%m.%d” # required.

7. Restart telegraf agent

$ sudo systemctl restart telegraf

8. Test your configuration file

$ telegraf test — config /etc/telegraf/telegraf.conf

Sample output :

$ telegraf test — config /etc/telegraf/telegraf.conf
2019–09–26T08:31:31Z I! Starting Telegraf 1.12.1
2019–09–26T08:31:31Z I! Using config file: /etc/telegraf/telegraf.conf
2019–09–26T08:31:31Z I! Loaded inputs: jolokia2_agent
2019–09–26T08:31:31Z I! Loaded aggregators:
2019–09–26T08:31:31Z I! Loaded processors:
2019–09–26T08:31:31Z I! Loaded outputs: elasticsearch
2019–09–26T08:31:31Z I! Tags enabled: host=apim-tagent
2019–09–26T08:31:31Z I! [agent] Config: Interval:15s, Quiet:false, Hostname:”apim-tagent”, Flush Interval:15s

9. Config Jolokia dashboard in grafana

9.1 Get the JVM Dashboard for jolokia2 plugin for java

9.2 Import the dashboard to grafana

Create grafana datasource

To visualize metrics, grafana needs to create a datasource to collect logs that are stored in elasticsearch index.

What is an Elasticsearch index?

An Elasticsearch index is a collection of documents that are related to each other. Elasticsearch stores data as JSON documents. Each document correlates a set of keys (names of fields or properties) with their corresponding values (strings, numbers, Booleans, dates, arrays of values, geolocations, or other types of data).

Elasticsearch uses a data structure called an inverted index, which is designed to allow very fast full-text searches. An inverted index lists every unique word that appears in any document and identifies all of the documents each word occurs in.

During the indexing process, Elasticsearch stores documents and builds an inverted index to make the document data searchable in near real-time. Indexing is initiated with the index API, through which you can add or update a JSON document in a specific index.

Create datasource

To create a datasource you can follow my other story for Create a Datasource in Grafana

To limit the datasource only for one index, enter its name or it’s pattern. As below I have entered my index name and selected the “Daily” pattern.

Index name : “[custom_index_name_jvm-]YYYY.MM.DD” and use pattern “daily”

Thank you for reading! If you enjoyed please leave a clap and follow my page.

Kishara Buddika