Geek Culture
Published in

Geek Culture

Log and Server Metrics Monitoring - WSO2 Micro Integrator 7.1.0

Monitoring is an important part of a deployment, which enables support and development teams to easily find the bug or an issue during the run time. An efficient Logging and Monitoring System will not only should support to find the issues and also must not impact the Performance of the Systems.

WSO2 has introduced earlier days with WSO2 EI Analytics for this use case and the problem that had was the Log Monitoring, even though we had the facility to monitor the API request counts, Proxy Service Request Counts and etc. But we do not have the capability to do the Log Message Monitoring and Tracing. Even though there were some addons implemented around EI Analytics, but they had some performance impact to the WSO2 Micro Integrator.

Due to these the recommended way to handle this will be using the combination of Prometheus, Fluent Bit, Loki, Jaeger and Grafana. Each of these use cases are described as below:

Prometheus: This is for Server Metrics Monitoring

Fluent Bit: Agent which listens to wso2 logs and publish them to Loki.

Loki: Store the logs and Process it whenever needed by the Grafana.

Jaeger: This is used to enable message tracing. Helps to trace the message in the Micro Integrator flow between mediators.

Grafana: Connects with Loki, Prometheus and Jaeger and show stats in Visualize Mode.

Note: In this article we are not going to Integrate the Jaeger will be considering that in my next blog on this concept. And also the setup is done on Ubuntu 20.04.2 LTS with Minimum Setup Configurations.

What we are going to Setup is Highlighted in the below Diagram and there interconnection.

1. Setup Prometheus

Go to https://prometheus.io/download/ and download the prometheus distribution. Here I have downloaded prometheus-2.28.1.linux-amd64.

Open prometheus.yml and update the below:

here: localhost:9090 — Prometheus running Host and localhost:9201 is the Micro Integrator host and port.

Execute the below command to start the Prometheus Server.

./prometheus

2. Setup Loki

Once that done we need to setup the configuration file. Create a file called loki-local-config.yaml and Insert the below.

To start the Loki Server execute the below:

After a successful start you can see the logs as below:

3. Setup Fluent Bit

There are two options to set this up.

Once extracted go to home directory and execute the below commands. Some dependencies also needed as below.

4. Setup Grafana

Go to https://grafana.com/grafana/download/7.1.1 and download the product.

Extract it to a location.

Go to bin directory and start the Server using ./grafana-server. Once Successfully started you can see log as below:

5. Setup Data sources and Dashboards to Visualize

Go to http://localhost:3000/

Now we need to setup the data sources, our main sources are Loki and Prometheus.

Go to Configuration -> Data Sources and Configure Loki and Prometheus.

Now we are done with data sources and we need to upload the WSO2 dashboards.

Go to https://grafana.com/orgs/wso2/dashboards, here I’m only going to load the highlighted three.

After selecting specific download the JSON file.

Like wise download the others as well.

Now Go to Grafana and Load the JSON files.

Above is one of the Dashboards and you can see the Logs section is still empty.

Also note that we need to start the Micro Integrator as below to experience the above dashboard.

Add the below to deployment.toml

sh micro-integrator.sh -DenablePrometheusApi=true

6. Enabling Log Monitoring

To Enable Log Monitoring need to send Log data to Loki, so the Fluent Bit Client will do this part.

To setup Fluent Bit client for Loki first we need to build the plugin. Go to

https://github.com/grafana/loki.git and clone the repository. Here I’m using the v1.6.1, if you are using the latest versions there can be mismatch in the folder structure as it has been updated in latest versions.

Go to loki home folder and execute:

make fluent-bit-plugin

The generated plugin will be under the name out_loki.so.

Now we need to create a config file to pull the logs and push it to Loki using Fluent Bit.

Create a file called parsers.conf

Note: Here the Time_Offset is important otherwise there will be a error in post data to API to Loki.

Create a file called fluent-bit.conf

Now run the Fluent Bit Agent using the below command from the location where we build the Fluent Bit at Setup Fluent Bit step.

After a successful start:

7. Verify the Log Monitoring

Send an API request and view the Dashboard Logs Section.

In Loki you can see a log as below:

In Dashboard:

8. References

[1] https://ei.docs.wso2.com/en/latest/micro-integrator/setup/observability/setting-up-minimum-basic-observability-deployment/#setting-up-prometheus

[2] https://ei.docs.wso2.com/en/latest/micro-integrator/administer-and-observe/cloud-native-observability-dashboards/

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ajanthan Eliyathamby 🇱🇰

Ajanthan Eliyathamby 🇱🇰

Associate Architect — Enterprise Integration | WSO2 Certified Solution Architect | https://ajanthane.blogspot.com/