Why and how of centralized logging with ELK stack for Rasa - Part 3.

Image Source — Logz.io

In the last one, we spoke on writing multiple log files also adding a component field to still be able to segregate them, and finally have different indexes for each component.

In this one, we talk more about:

• Logging file pattern
• Using grok filter in Logstash
• Using log timestamp for filtering in Kibana

Logging File Pattern

• It’s always important to have a unified pattern when writing logs, we need it for regex matching so that we can extract important information like the timestamp, sender details, messages received, etc.
• I have written the logs in the files as,

2020–10–15 04:55:12 INFO sender id:xxx~request:Hello

2020–10–15 04:55:12 INFO sender id:xxx~response:Hello!How can I help you?

The above log includes :

1. The timestamp of the log
2. Log Type
3. Sender Id/Session-Id for the logs
4. Request
5. Response

I used a ~ as a delimiter to segregate them with the help of grok and have all these as fields.

Using grok filter in Logstash

Grok is a great way to parse unstructured log data into something structured and queryable.

• Grok pattern matching will be used under the filters section in the /etc/logstash/conf.d/30-elasticsearch-output.conf file

%{TIMESTAMP_ISO8601:time} %{LOGLEVEL:log-level} sender id:%{GREEDYDATA:sender_id}~%{GREEDYDATA:log_type}:%{GREEDYDATA:log_content}

2020–10–15 04:55:12 INFO sender id:xxx~request:Hello

• It’s like a regex match. In addition to matching the logs, it will also create additional fields as below:

1. time - 2020–10–15 04:55:12
2. log-level - INFO
3. sender_id - xxx
4. log_type - request
5. log_content - Hello

There is a prebuild library of grok regular expressions that can be used.

Using log timestamp for filtering in Kibana

• Next up is logging time, Elasticsearch by default adds a timestamp field with the value of when the logs were stored in it.
• This same timestamp field is available as a global filter for us, which we select when creating an index pattern.
• We can still get the time of our logs from the above time field that we just created, but we cannot sort the logs using that time in Kibana.
• So in order for that to work, we need to replace the default timestamp value with the value of our actual log time.
• This same thing can be done in Logstash filter as,

• We add a new field called log_time that is set to the time value from the grok match
• Next, we add a date match, and replace the timestamp field with log_time field and also specify the locale and timezone
• This does the trick and now we have our actual log timings and we can also sort our logs basis on the timestamp.
• The next steps are the same, we create an index pattern in Kibana and we can check our logs as,

• This is how we added new fields, made sure the timestamp had our log timing and not the default one.

So, that's it, there are a lot many things that can be done with ELK, especially with the Logstash pipeline and structuring the logs the way we want for ease of visualization.

Please get in touch in case of any queries.

Thanks.