Parsing and Sending Laravel Logs to ElasticSearch using Logstash
When your laravel.log file size is way too big for you to analyze, it might help to load it to ElasticSearch and analyze it there. This article will go over the basics of how you can do that using LogStash.
Laravel’s error and logging features allows us to log application-specific events that might be proved useful in analyzing behaviour of our application. But problem arise when lets say you have 1 GB size of log file, or 10–15 different laravel application each one producing enormous amount of log data, suddenly answering questions like following is a very difficult task:
- How many error that is related to PDOException had occurred last weekend?
- Compare the amount of Log::warning generated compared to last month
- Sort the list of laravel applications by number of Log::critical recorded in descending order between March 1 to March 15, 2016
Answering to this kind of inquiries might be impossible w/o any sort of tools at hand. Some people actually tries to explicitly record this events in the business logic of Laravel app itself, but that practice is really bad since that weighs down the performance of the application.
It is more ideal for you to accumulate those data in a simple log file and forward it to a background processing server that can then further process the information into a useful form. This way, your application entirely focuses in what it is supposedly doing — serve http request in the most fastest and efficient manner.
Logstash allows us to process those gigantic log files and break them down into manageable parts. It can also monitor the log files for any new entry and automatically process it. The processed data can then be forwarded to ElasticSearch.
ElasticSearch is a Java-based search engine with analytics capabilities, it allows you to store enormously huge amount of data, then analyze it in a way that is more than anyone could imagine. It also has a built-in REST API, on which you can query the data that you had stored on it, or use Kibana that can allow you to visually build queries instead of hitting the REST API directly. To summarize:
- ElasticSearch — for storing, analysis and API of data
- LogStash — for parsing and pre-processing of raw log files
- Kibana — serves as the user-interface of ElasticSearch so you can visually see the output of ElasticSearch results in different format (tabular, Pie Charts, Historgram etc).
The combination of this three great products is what usually called as ELK Stack.
Once you had forwarded your Laravel log files into ELK stack, searching, analysis and any other form of inquiries is very easy to answer no matter how huge the error log files you have.
Setting up the ELK stack in your local machine
The easiest way to have an ELK stack running in your machine is to setup a vagrant box that has those 3 softwares pre-installed. That way, you don’t have to worry about the details of setting those up. Follow these instructions to install it in your machine:
Once you have those installed, clone the vagrant-elk-box
git clone https://github.com/comperiosearch/vagrant-elk-box.git cd vagrant-elk-box vagrant up
It might take a while for it to complete, since it will download first the basebox and install everything for you. After it had completed, you should have the following available in your local machine:
In production, you should not process logs in your local machine! ELK stack is usually installed in some seperate cluster of servers not facing your end-users (i.e. not in web servers).
Processing an Example laravel.log file
In able to illustrate the process, I created a sample project with sample laravel.log file in GitHub. You first need to clone this inside the vagrant-elk-box folder.
Take a look at the contents of logstash-laravel-logs/logs/laravel.log
This is basically a kind of file that you will usually see inside storage/logs/ folder of a Laravel 5.2 application. This log file basically simulates different kind of log output you will find in a typical Laravel log file:
Usually, you would like to pre-process the logs generated by those into following:
- Group and Filter logs by environment (local, production, testing etc)
- Group and Filter logs by type (error, emergency, alert, critical,warning, notice, info, debug)
- Group by date, month, year. So you can query all logs given a particular date-range
- Search by particular keyword. For example, If you want to view all logs that mentions the keyword “QueryException”, you should be able to retrieve it.
Parse log files using LogStash
Originally published at www.laravelfeed.com.