Distributed Logging with ElasticSearch, Kibana, and SeriLog
This article is part of my Microservices and Cloud-Native Applications series. You can find the other parts of the series below.
- Saga Orchestration using MassTransit in .NET
- API Gateway with Ocelot
- Authorization and Authentications with IdentityServer
- Eventual Consistency with Integration Events using RabbitMq
- Distributed Logging with ElasticSearch, Kibana, and SeriLog
- Resiliency and Fault Tolerance with Polly
- Health Check with WatchDogs in a Microservices Architecture
- Distributed Tracing with Jaeger and OpenTelemetry in a Microservices Architecture
- Metrics to Monitor Microservices with OpenTelemetry and Prometheus
If you want to take a look at the GitHub code, you can access it here: https://github.com/ebubekirdinc/SuuCat
In order to track the flow of events among various microservices, distributed logging is a crucial component of microservices design. Applications are divided into smaller, independent services in a microservices environment, and these services interact with one another over a network. To obtain insights into the behavior and performance of the system, log data from each service must be gathered and examined.
Distributed tracking requires collecting log information from various services and consolidating it in one place. A logging framework that allows distributed tracing, like ElasticSearch, and Kibana, is typically used to accomplish this. These frameworks offer APIs that enable programmers to instrument their applications to record trace data, which contains details about the system’s requests and responses.
Distributed logging captures monitoring data from multiple services, allowing us to identify the root cause of problems and optimize the performance of the system as a whole. It also enables to watch the system in real-time and address problems as they come up.
Distributed logging, though, can also present some difficulties. For instance, the sheer amount of log data that microservices produce can be overwhelming, making it challenging to handle and analyze. Additionally, logging infrastructure must be dependable and scalable, which can be difficult to execute, in order to collect log data from numerous services.
In the SuuCat project, distributed logging is implemented with ElasticSearch, Kibana, and SeriLog. SeriLog is a logging framework that allows distributed tracing in .NET. ElasticSearch is a distributed, open-source search and analytics engine for all types of data. Kibana is an open-source data visualization plugin for ElasticSearch.
SeriLog can be installed with the following Nuget packages.
After that, we can configure SeriLog in the Logging project like below.
Here you can find descriptions of some of the settings properties.
Enrich.FromLogContext(): Enriches the log events with property values from the log context. This is useful for adding context-specific information to the logs.
Enrich.WithMachineName(): Enriches the log events by adding the name of the machine generating the logs.
WriteTo.Debug(): Log messages are sent to the Debug output. This can be useful during development.
WriteTo.Console(): Log messages are also sent to the Console.
WriteTo.Elasticsearch(): Log messages are sent to the specified Elasticsearch endpoint defined by elasticUri.
Inside the ElasticsearchSinkOptions:
IndexFormat: Defines the format of the index name in the Elasticsearch server.
AutoRegisterTemplate: If set true, the sink will make sure a template is registered in the Elasticsearch server which matches the logs. This is used to optimize the Elasticsearch storage and search capabilities for logs.
NumberOfShards and NumberOfReplicas: These options control how many shards and replicas the logs should be stored in on the Elasticsearch server.
ElasticSearch and Kibana can be installed using the following Docker files. More information about the installation is here: https://github.com/ebubekirdinc/SuuCat/wiki/GettingStarted
docker-compose.yml
docker-compose.override.yml
Then proceed to http://localhost:9200/ to get to the ElasticSearch page.
If you see the following screen, it means that ElasticSearch is up and running.
Use the following URL to see the logs of each microservice on Kibana.
You can use default credentials to log in to Kibana.
user: elastic
password: changeme
Once you log in, you can create your index like below. Define it like “suucat*”. You can get all logs starting with “suucat” by putting * at the end.
After creating an index pattern you can view it here:
By clicking the index you can display all log fields and refresh them.
Then click on the Discover button to see the logs of each microservice.
A process can from start to end can be tracked with the help of CorrelationId. A CorrelationId is a unique identifier that is added to a transaction as it enters a microservice-based system. This identifier is then passed along to each service that participates in the transaction. By including this identifier in the logs generated by each service, it becomes possible to trace the entire transaction across the entire system.
In distributed logging, the use of CorrelationId aids in locating and resolving problems in a microservices design. It enables tracking of the transaction’s path through various services, assisting in determining which system or service is responsible for the problem. This is particularly helpful in highly distributed environments where various services and
systems may produce logs. Below you can see the CorrelationId field logged in Kibana.
Last but not least, it would be good to have a look here to be able to trace in Jaeger with TraceId.
More info can be found in the Elastic docs, and SuuCat GitHub.
References:
