It’s been almost 4 years now since I began working and playing with centralized log management systems based-off open source projects like ELK, Grafana Loki, Fluentbit & FluentD, Splunk etc
One very useful step common to all the projects I worked on was the ability to load test such systems with the aim to find the breaking points in whatsoever system design was implemented. This step is critical to find performance and scalability issues that may arise once in production.
For this, it was important to find/build a utility capable of randomly generating log events.
Here are the two factors that really mattered;
- Ability to generate random or fixed log lines
- Ability to generate logs at a given rate (for example 1k events per second)
If you did some minimal research, you most have come across the “chentex/random-logger” docker image from the DevOps 2.5 handbook.
Note: This utility works well for producing steady event streams, but it is not sweetable if we consider alternating the event burst rates.
A better option to consider today is the ocp_logtest utility from openshift.
Here is quick documentation containing everything you need to get up and running.
Sneak peek of what others have done already with ocp_logtest: https://www.outcoldsolutions.com/blog/2018-10-22-collector-performance/
Happy reading :) Just a roman seeking & sharing my findings while gaining knowledge!
