Load Testing ELK Stack with 1M Hits.
This is a continuation of my previous post on What and Why EKL Stack.
In this article, we will cover
1. Why load testing is important
2. Load testing results for ELK stack for 0.1M HTTP request or 1M hits
3. My experience setting it up on AWS
Why is load testing important?
Whenever I build any new server, I plan to load test it. Load testing allows me to simulate real-life conditions for the application and determine how it behaves when it is working under a massive load. Server load testing gives me answers to the following typical questions:
- Does my server support N concurrent users?
- How many users can simultaneously work with the webserver without a perceptible slowdown?
- How does the server response time change if the number of users increases or decreases?
Here’s an example. For this load test, I used Jmeter with 100 active threads in a loop of 1000 generating 0.1M HTTP requests in 2 minutes.
This HTTP request hits my local NodeJS Server which generates 10 logs in each request. So we will see 10*0.1M =1M logs after the load test.
Here is the new system architecture. NodeJS server is on my local system and Elastic search and Kibana are installed in AWS c5.xlarge (4 CPU, 8GB).
Nginx is used to reroute requests from port 80 to elastic search.
Setup on AWS
The configuration required to setup ELK stack on AWS is mentioned in the file.
I used Nginx to make elastic search publicly available.
Elastic search can be configured to add authentication in elasticsearch.yml.
The initial configuration consumed 50% of server memory or 4GB of memory and on load test hardly consumed any memory. However, the CPU utilization jumped above 40%.
How does it work?
My local system has a 10 Nodejs server running in cluster mode on port 3000. Jmeter hit my local servers at 0.1M request in 2 min. Each request generates 10 logs, meaning that 1M logs (96 MB) were generated in 2 min.
Logstash collects data from the log file in real-time, transforms it on the fly, and sends it to the elastic search.
Logstash took by 10 more minutes to generate them, as compared to Nodejs. This is because 8 Logstash workers pull files at 125 logs/worker in parallel on my local machine.
The Logstash configuration can be found in this file.
Load testing results for ELK stack for 0.1M HTTP request.
100 threads were used in parallel, in a loop of 1000 with a ramp-up time of 0.005 sec, to generate 0.1M request in 2 min. (Jmeter File)
As each request generates 10 logs, 0.1M should generate 1M logs or hits.
We can ignore the 475 other logs, which are different system logs like file rotation, etc.
I then checked if there was any load on the ELK server during this load test, with the ELK APM server installed on the ELK server.
There was a spike in CPU to 42.6% when the load test was running, then it went to ~0% after that.
Surprisingly, the load testing didn’t have any effect on memory utilization.
Conclusion
ELK stack is a great tool that comes with many functionalities like log analysis, APM, and business analytics, and it can be considered as a replacement for expensive logging tools.
ELK can be configured to make it highly available by running an elastic search in a multinode environment.
Dashboards can be prepared in Kibana for business analytics.
Note: log0, log1 were added just to create charts, and do not represent any business parameters.
Follow for more like on LinkedIn: https://www.linkedin.com/in/deepak6446r/
