ELK — Elastic, Logstash & Kibana
If you’re looking to setup the ELK stack opting Elastic’s cloud service, and visualise your logs in Kibana, you’re at the right place! In addition to that, you can also monitor a lot of API metrics from a performance standpoint using Elastic’s solution - APM (Application Performance Monitoring).
Table of Contents
This article will explain how we will ship NodeJS service (run using pm2) logs to Logstash, see API metrics in APM, and visualize logs in Kibana dashboards.
Prerequisites:
A logger that outputs logs in JSON format!!
- A logger for Node that has capability of writing every single LOG line as a JSON. We have gone with winston to achieve the same.
- We will include timestamps to denote at what time each line was printed, along with the JSON. Other meta attributes like file, requestId can help us trace logs even better.c
Sample log JSON
{
"service":"phable.api.backend.js",
"file":"ReportsService.js",
"level":"debug",
"message": {
"reportsCount": 6723
},
"timestamp":"2020–11–13T08:22:35.279Z"
}Crontab commands for pm2 logs
If you run your services on server instances that reboot every day, or have a scheduled downtime to minimise cost, here are the crontabs for you!!
#@reboot cd path_to_service/ && sudo pm2 start index.js --log-date-format '' --name "ServiceName"Note: Common pm2 logging mistakes
- If you had pm2 services running before this change, make sure you delete and start them again (not just restart). Do a
pm2 delete "ServiceName"without fail, for thelog-date-formatchange to get reflected the next time you start! - We need to make sure — log-date-format attribute is empty or removed. As that makes the log output a string with timestamp prefix; instead of a JSON, which is not what we need.
- Also,
colorize: trueoption in winston logger adds some prefixes and suffixes like^[[32m { JSON log data } [[36mto indicate to the terminal that this particular line has to be green/red. Choosecolorize: falsein production winston config.
Now that pm2 logs are set up in the desired format, we will just have to work on shipping them to Logstash. Let’s ship it!
Creating ELK Deployment
If logging, monitoring and observing API metrics are your only objectives, I’d suggest to choose Elastic Observability. Choose the number of instances for elastic (2+ if you’re keen on not losing data).
Once you’re done creating the deployment, elastic gives you username and password, keep it saved. When you want to add data to Kibana, you will need to use these credentials.
After your deployment is created. Copy the cloud Id and cloud auth key. You will need this in the Filebeat setup explained later below.
Now that the deployment is created, let’s see how the architecture looks like and works… As shown in the picture below, log files is the use case we’ll be focusing on and we’ll see how we can have your logs shipped using FILEBEAT and visualize it in dashboards in Kibana!
Tools/Solutions used
APM (Application Performance Monitoring)
Elastic APM is an application performance monitoring system built on the Elastic Stack. It allows you to monitor software services and applications in real-time, by collecting detailed performance information on response time for incoming requests, database queries, calls to caches, external HTTP requests, and more. This makes it easy to pinpoint and fix performance problems quickly.
Elastic APM also automatically collects unhandled errors and exceptions. Errors are grouped based primarily on the stack trace, so you can identify new errors as they appear and keep an eye on how many times specific errors happen.
Can give us an overview of which call is taking how much time. Can think of it as a node middleware that’s been added to monitor api calls and push metrics to this solution from Elastic.
Snippet added in Node service to plug this tool in
Reference: https://www.elastic.co/guide/en/apm/agent/nodejs/master/configuring-the-agent.html
// app.js OR index.js -- entry point
// add config values for APM to .env variablesconst {
apmEnv, apmServiceName,
apmSecretToken, apmServerUrl
} = process.envif (apmEnv && apmEnv !== 'dev') {
var apm = require('elastic-apm-node').start({
serviceName: apmServiceName,
secretToken: apmSecretToken,
serverUrl: apmServerUrl
})
}
Action items that can be created from this solution
- We can sort the average time taken for every call in the backend. And pick each one up, and optimize wherever there is scope to improve!
Eg: The following data clearly shows that /notify endpoint requires a revisit. We can find more problems once we have this in prod, these is performance metrics from staging
- Find time taken in worst case scenarios. Some APIs you write may seem performant for the sample data you’re requesting. But APM provides the 95th percentile time taken value for every API. That way we can find which calls are slow in production
- Find which method or function block inside the APIs implementation is taking the longest to execute. That level of clarity is provided for each and every API call by APM.
Eg: The following data shows Firebase dynamic link generation took the longest amount of processing time in one of our api calls!
FileBeat
It can be looked at like a log watcher. It’s a service/executable developed by elastic, which comes with it’s own config file. You need to mention the path of the directory that it needs to monitor for logs. \home\ubuntu\.pm2\logs\*.login our case.
Ship to Elasticsearch or Logstash. Visualize in Kibana.
Filebeat is part of the Elastic Stack, meaning it works seamlessly with Logstash, Elasticsearch, and Kibana. Whether you want to transform or enrich your logs and files with Logstash, fiddle with some analytics in Elasticsearch, or build and share dashboards in Kibana, Filebeat makes it easy to ship your data to where it matters most.
Filebeat.yml
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched.
- /home/ubuntu/.pm2/logs/*.log#elastic cloud configcloud.id: "phable_staging:xxxxxxxxxxxxxxxxx"
cloud.auth: "elastic:xxxxxxxxxxxxx"#cloud id can be found in the your deployment's landing page
#use the pwd that was generated on creating a deployment here "elastic:xxxxxxxxx"
Installation guide:
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html
Logstash
Logstash is an open source data collection engine with real-time pipelining capabilities. Logstash can dynamically unify data from disparate sources and normalize the data into destinations of your choice. Cleanse and democratize all your data for diverse advanced downstream analytics and visualization use cases.
source: www.elastic.co
I don’t think I could have described it any better, you can think of it as the port where your logs reach after you ship ’em! ;)
Now that filebeat is set up, we need it to ship logs to logstash — as the name suggests — a stash for logs. All we need to do is enable the logstash module in filebeat. It automatically ships every single line written to pm2 log files to logstash.
Kibana
Whatever logs go into Logstash can be visualized in Kibana, with kibana you can create customized dashboards. With pm2 logs coming in as jsons in this format. You can add filters to sub fields in the JSON too, eg: search by message.body contains ‘exception’
Sample JSONs coming to Kibana — A Morgan log
{
"service":"phable.api.backend.js",
"file":"app.js",
"level":"info",
"message": "fy7aihcu3vk00 POST /api/apiName 200 100–971.558 ms",
"timestamp":"2020–11–12T12:52:53.029Z"
}Kibana sample view…
Once we load the data in the dashboard, and after we get hold of KQL {Kibana Querying Language : https://www.elastic.co/guide/en/kibana/7.9/kuery-query.html }
- we can filter the logs with queries like message contains “exception” and filter out occurrences of errors/exceptions
- Build reports and visualize them, say you logged the JSON here..
{
"reportTypeA" : 20,
"reportTypeB" : 72,
"reportTypeC" : 19
}once every day in the logs’ JSON output. Then you can use Kibana and visualize it by building reports like this too…
Hope this article helped you understand ELK setup using Elastic’s cloud solution. We at Phable will be running some cost-benefit analysis to see if this is a sustainable solution, or switch to in-house elastic server deployments after monitoring how the solution works for a couple months. Hopefully the cost benefit analysis becomes another fruitful article in the future.
Until then, happy logging!
