Formatted Logs to ELK from K8S/Openshift

You might be in a situation when you run the microservice in Kubernetes, the logs are not formatted in ELK, as fields. This is especially a case when the flow is Fluentd->ELK instead of Filebeat->Logstash->ELK

Simple Plain architecture we are discussing today

Springboot Logs

Folks use Logback for logging as its fastest and smallest library. The appenders normally configured to use patterned so that developers can see logs easily in terminal their MDC settings from the code.

logback.xml Entry

Logs in console ( Developer IDE )

Logs are not parsed as Kibana fields in the K8s environment.

when you turn on the above architecture, you will see the logs are not formatted. The message field, in Kibana, will show as one, an unparsed line like below a makes debugging harder.

How do we fix the issue without major rework?

Required changes in logback.xml

The above settings will dump the logs as JSON

Fluentd reads this JSON and pumps to ELK. ELK has an inbuilt mechanism of understanding the JSON and parse as the fields (ex: Level, Thread Id, Message) in an Index.

Outcome as fields

The last thing to remember

you might feel, Logstash is meant for this to parse as the fields before importing into Index. There is no adapter between Fluentd & Logstash and in fact, there might be none because both are log aggregators.