Build your own Log Management Tool
Always logs are the critical part of any application, they provide enough information about systems/applications at a given point in time. Here in this title, we’ll try to import log files to MySQL DB.

There are many candidates to serve the better way in the market.
This title helps us to build an own Log management tool with few simple steps.
Here are major steps to build own log management tool.
- Identify the logs we need to manage/monitor.
- Build a small MySQL Database ( let say: log_server).
- Create a cron to send logs from all nodes/servers to ‘log_server’.
- Create a multiple databases if needed w.r.t environments ( let say prod_logs, stage_logs, qa_logs).
- Create tables in the databases with appropriate type w.r.t logs.
- Import the logs to tables.
- Search the logs through MySQL client or any other tools you use to connect to MySQL DB.
- We can create a fronted app if required to retrieve logs from backed (log-server) DB server.
Benefits :
- Easy to retrieve the logs from any server/node.
- No need to login to servers, this helps us to speed up the search and more secure.
- No need to grant access to the production server for Dev/QA team for search logs.
- Search multiple servers logs in one place with simple queries.
- These logs help to troubleshoot.
- Logs help the top level management team to take a decision.
- Any many more…
Pros :
- Free
- Small
- Simple
Cons :
- New setup may take some time.
- Latency ( 15min) least recent data is available without any issue.
- If we want to retrieve within 5sec need more compute power and logic to do the retrieve.
- Lack of dashboard view.
Steps in depth to build it:
1. Identify the logs we need to manage/monitor.
Answer: It’s our choice, in general production, demo, stage, logs are enough.
2. Build a small MySQL Database ( let say: log_server).
Choose any small Linux server ( RHEL/Ubuntu) and install MySQL and do a secure installation.
3. Create a cron to send logs from all nodes/servers to ‘log_server’.

rsync can help to sync logs from multiple nodes to “log_server”
Setup a cron on all nodes ( production server, stage servers etc..) to send logs.
4. Create a multiple databases if needed w.r.t environments ( let say prod_logs, stage_logs, qa_logs).
Connect to MySQL on “log_server” and create few databases. e.g;
>create database prod_logs;
5. Create tables in the databases with appropriate type w.r.t logs.
This is the most important step in our build.
Create a table ‘n’ number of columns based on the log files ‘n’ number of fields. And choose a proper type for each field, let say date, time, text etc…
e.g: If the log file having 8 fields in it then create a table with 8 fields.
6. Import the logs to tables.
Split the ’n’ number of fields in the log file with a delimiter and import to the required table.
We can use ‘LOAD DATA LOCAL INFILE’ option in MySQL.
7. Search the logs through MySQL client.
THE END
“Indmax is an IT Consulting and Services firm providing End-to-End IT Support unified support across Web, App, Data, Network and Security Layers. We have demonstrated experience in managing the complete tech stack deployment in DC, Co-Lo, Private, Public and Hybrid cloud for reputable clients over the last 11 years of our existence with a proven record of managing 7K+ Virtual and Physical devices. Notably, we have pioneered and practiced Infrastructure automation, Continuous Integration and Deployment for businesses across multiple domains”