JMeter vs k6 — who makes a stronger case?
Introduction
While JMeter has been in the market and used since the 1990s, k6 is the newer kid in the performance testing realm. Nevertheless, k6 has made quite an impact and is gaining popularity every day.
Head-to-head
While judging a tool or a framework for its efficiency and effectiveness, we would generally like to keep few parameters in mind. Based on my experience and the time I have spent with these two, I shall try to compile it into a few points which might help others make the decision much quicker. In reality, both these toolboxes have their own specific use cases and it is just about figuring out which is the right fit for your application.
1.Installation & set up
JMeter has a prerequisite of java. Compatibility with the right version of jdk.
k6 does not require nodejs or any package manager other than the k6 bundle itself.
2.Availability of an UI
Jmeter comes with a built in UI where all the tests are configured and the action takes place. Therefore, it is apparently easier for a novice to pick up.
No UI in the k6 free version. Easier to pick up if one has basic javascript coding familiarity.
3.Test Design
JMeter, in most cases is used via the UI and the options to design a custom load/performance test by default are limited. custom thread groups, sampler plugins are almost a necessity
k6 is meant to be highly configurable by design since the test would be entirely based on a script(js) that we write. This probably has been the USP for k6.
4.Reporting
JMeter has inbuilt html reports, plus few more with plugins. Together they seem to be enough for any user but again the lack of customisation could be a deal breaker.
k6 does not have any built in reports apart from a vanilla command line report. Furthermore, there are counters, thresholds, rates etc. that can be configured and baked into the report. So highly configurable reports but few options out of the box. Has third party integrations with grafana to curate data into dashboards.
5.Resource economy
JMeter uses the JVM to generate new threads. 1 single Jmeter instance can simulate up to few thousand users. Also the duration and design of the test is limited by the java heap memory. Therefore, resource heavy.
k6 internally uses goroutine to simulate virtual users. Go is a compiled language and hence performance is an inherent feature. 1 single instance of k6 can generate hundreds of thousands of VUs. for producing the same traffic/load, k6 would need significantly less resources.
6.Distributed load tests
Jmeter natively supports distributed load tests using the Jmeter server.
k6 supports distributed load generation but with the help of k6 operator, it can do the same over a kubernetes cluster across multiple pods.
7.Community
Jmeter has been around for more than 20 years now, therefore has a wider and diverse user base with strong community support.
A strong community is being built up everyday with a growing user base.
8. Network Protocols
JMeter supports SOAP, HTTP/1.1, FTP, JDBC, LDAP, SMTP, POP3, IMAP, TCP — most of the ones available today.
k6 supports the modern ones — gRPC, HTTP/1.1, HTTP/2. legacy might be an issue.
9.Maintainability/Collaboration
JMeter scripts are saved in the form of .jmx files, re-using and loading them from one machine to another can prove to be painful as version mismatch might cause it to break.
k6 scripts are written from the scratch, they can be version controlled using git and hence can be collaborated over with other teammates.
These were the determining factors for me when I was in a fix between Jmeter and k6. In my case, we chose k6 since we wanted to be prudent on the cost we incur and were able to generate close to 5x production traffic using minimal infrastructure.
Bottomline
As you can see it all depends on the context and use case. For instance, if you want a UI tool with a strong community support, Jmeter could be your thing. Else, if you are decent with js and need a highly configurable test design, k6 should be your go-to.
