Central API Logger

A log pipeline called Central API logger was recently developed by Venture Garden Group (VGG) SMART Team to centralize logs from different APIs which helps to make the logs accessible from ElasticSearch and then visualize them from Kibana.

Logging generally fulfill two major requirements: monitoring application performance, and providing adequate information. In terms of performance, logging gives you an insight into what’s happening inside your application and, in some cases, what’s happening in the world outside as well.

The Central API Logger is very useful when attempting to identify problems with your APIs, as it allows you to aggregate and search all your logs in a centralized place. It also allows you to identify and analyze issues that span multiple APIs by correlating their logs during a specific time frame. It can simply provide status information on API Applications that are working correctly incorporating those that talk to the outside world. Let’s face it, there’s nothing more rewarding for a developer than seeing a log full of insightful information on what is happening with his/her application.

The visualizations in Kibana allow you to see failure trends, response time, recovery time, down time, earliest time to recovery just in a single dashboard so as to be able to know what to do to optimize your API Applications.

Components

The Central API Logger has a library called Autonise.Logger that combines the power of NLog, Filebeat and the ELK Stack to provide a seamless logging pipeline. Logs pulled by this framework are saved in CSV format, centralized in a folder, enriched and cleaned up by logstash filters and automatically pushed to ElasticSearch. Here are the components and their major functions in the pipeline:

1. Autonise Logger Core — Used to standardize the format of the API logs, centralize the log files to a folder, zip and archive the logs after a certain period of time usually a month.
2. Filebeat — Picks up the centralized logs real time from the central folder and pushes it to Logstash. It also helps to ensure there is no loss in data during transfer.
3. Logstash — Enriches the logs; added geolocation using the request IPs of the caller, cleaned up the urls and IP addresses and then did some other cool manipulations to the logs before indexing it straight to ElasticSearch.
4. ElasticSearch- Real-time search server based on Lucene
5. Kibanna — Visualization of all the logs that has been indexed to elasticsearch

Architecture

The Figure below shows the current architecture of the API Logging Pipeline using Autonise API Central Logger (AACL).

API Central Logging Pipeline Architecture[/caption]

Filebeat is used on each server (APIs with AACL hosted) where API logs are to be harvested. These nodes will send data to the indexer: logstash which will enrich, cleanup and process them using filters and then send the formatted data to the elasticsearch cluster. One instance of Kibana to visualize and dashboard the compiled log data.

This architecture is very scalable; more logstash instances can quickly be added to augment the indexing to elastic search. Same for elasticsearch, which works as a cluster of one node by default. The architecture also ensures that there is no data loss in the sense that even if elasticsearch is down, the logs are on physical files and can always be re-indexed anytime.

Conclusion

This log pipeline has brought a lot of simplicity and efficiency into API logging as all the traffic going in and out of an ASP.Net Web API will be seen on your Kibana dashboard in real-time. All you need to do is to import the ACCL library into your ASP.Net Web API and then do some configurations on your Application Servers as regards Fibeat, Logstash, Elasticsearch and Kibana and you are good to go.

The code base and all the configuration scripts are open sourced. Feel free to check out the code base on https://github.com/VentureGardenGroup/CentralAPILogger.git .