A Perfect Match
We Tried Out Three Load-Testing Tools So You Don’t Have To!
“We’re building a game with a boatload of socket connections.”
That’s how Chakrit Nimmanant, one of our developers here at Myplanet, describes his most recent project to me.
We’re having a conversation about load testing — something he’s done a lot of, recently — and I ask what the biggest challenge was this time around. As is often the case, there are a few different things he lists as trouble spots that had to be navigated around but really, this time, it was those socket connections.
Load testing tries to find out a system’s maximum capacity before any noticeable degradation of the user experience occurs. When done correctly, it allows developers to optimize performance and guarantee usability (think back to the launch of the US healthcare site — HealthCare.gov — if you wonder why this might be necessary).
As a company that works with enterprise clients, the work we do routinely faces heavy user loads. Whether we’re building commerce experiences for hundreds of thousands of users, or employee tools that are in use every day in offices around the world, we’ve gotten quite good at load testing and optimization.
But for Chakrit’s most recent project, the tried-and-true tactics he’d used before weren’t going to cut it. In fact, this project required not only a more thorough testing protocol than might normally be necessary, but also a new way of executing those tests.
So, as any dev would do, Chakrit went searching for a load-testing tool that fit the specific needs of this project (in essence, one that would support all those darn WebSockets).
Option A: JMeter
Developers familiar with load-testing procedures and tools are most likely familiar with JMeter. It is, according to Chakrit, a “very complete and very good tool”. As such, it seemed a very obvious and very good choice.
But things aren’t always as they seem and in spite of its completeness, there were still some considerations to take into account.
On top of the steeper learning curve, there were also code adjustments to be made so the tool could do what was needed for testing. Those changes plus the Java concerns and in the end, there were simply too many hiccups to overcome in using JMeter for this project.
No, this time around JMeter wasn’t the solution Chakrit needed, so it was on to the next.
Option B: Tsung
Once he’d decided against JMeter, Chakrit undertook a fairly exhaustive search for a load testing tool that would support WebSockets and still provide the flexibility he needed. That’s when he came across Tsung.
Tsung supports WebSockets (essential!). Tsung is open-source (also essential!). Unfortunately, Tsung uses Erlang for development. And if he had concerns about the additional work involved with using Java as the main language, Erlang was definitely not going to make on-boarding and sharing knowledge with co-workers easier.
Chakrit said sorry to Tsung and cut that option, too—but he did so with a heavy heart.
“The key thing we were looking for when searching for load-testing tools was support of WebSockets, because as I’ve mentioned, we had a boatload of sockets to test,” Chakrit says. “There are very few tools on the market that meet that specification, so our options were pretty limited.”
Basically, the search wasn’t getting any easier.
Option C: Artillery
By now, the search for a load-testing tool that could do everything he needed was starting to feel like his own personal Everest. But Chakrit persevered and eventually came across artillery.io. Could this be the solution he’d been searching for?
He went through his checklist:
- Does it support WebSockets? Yes (with some modifications)
- Is it open-source? Yes
Chakrit started to get excited.
Artillery is still a relatively new tool on the load-testing scene. Even since this summer when Chakrit came across it there have been upgrades and improvements, but it was clear that the bones of Artillery were good and he saw the potential for its use right away.
“With modifications and code tweaks, we knew we could use it,” says Chakrit. “Plus, we could make it modifiable for our needs for this and other projects going forward. That isn’t ever the main reason for choosing a tool, but it’s certainly something we hope for.”
Artillery offered most of what Chakrit needed and what it didn’t offer, he knew he’d be able to adjust with relative ease. At last, his load-testing tool was selected! Now for the fun part…
Making It Work!
Once he’d decided on artillery, Chakrit got down to business and started making the modifications that would make it work for his *SUPREME WEBSOCKET UNDERTAKING*.
Step 1: Authentication
To begin with, he quickly realized he’d need to get the socket authentication handshake in place, or none of his other necessary modifications would be possible.
Artillery does not allow the configuration of some properties of the socket connections, nor does it invoke the socket authentication, since it is not implemented. So, step 1 was to define some configuration syntax to be used in the scenario file.
This key setup, plus some minor code changes to Artillery, allowed Chakrit to direct it to perform the handshake to the destination his server listens on. To complete the process, he added the authentication step, which basically emits an ‘authenticate’ message along with a token to be used to authenticate the socket channel.
Step 2: Support JSON payload in the Channel
Once he had the handshake authorization in place, Chakrit was ready to make the other mods he needed to complete his load testing. Because Artillery did not allow him to easily send messages through the channel as a JSON object and Chakrit needed to have control over this aspect, his next step was to implement new code that would allow him greater control.
“We needed a way to control artillery.io, to be able to send a greater variety of message types to fit our needs.”
To do this, Chakrit defined his new syntax, jsonData, on top of ‘data’ syntax in the scenario side.
Finally, he built a function called “buildJsonData” which allowed him to get the information given from the scenario file and modify appropriate values before producing a proper JSON object.
Step 3: Create Feedback Testing Options
Scenarios in Artillery typically flow in one of the two following ways:
1 — Sequence
- Testing follows a sequence (e.g. Steps 1–10) in direct order, with no modifications or alterations to the direct order of operations.
2 — Time pause
- Testing follows a sequence that has been slightly adjusted for timing changes (e.g. Step 1, wait 10 seconds, then Step 2, etc). Time pause helps with things like simulating network delays and real-world conditions, which allows us to test for things like server response to delays.
For this project, however, Chakrit needed to be able to a) stay in a step until a response arrived, b) validate response values, and c) store the values for later steps.
So instead of Step 1 followed by Step 2 (either immediately or after a pre-set delay), he wanted to test Step 1 and either mark the test as a failure or proceed to the next step, depending on the response Step 1 received — or whether a response arrived from the server at all. To do this he had to set up some other modifications that would allow him to have feedback testing abilities.
He introduced these syntax modifications on the scenario file:
Chakrit then added the support of this step in the code, which allows a step to wait and validate a response before proceeding to the next step.
And with those modifications complete, he was all set. Load testing could be done, including support for all those many, many, many WebSockets.
Challenges are par for the course in development work—it’s often a big chunk of what motivates most developers.
The chance to innovate, test, explore, and experiment is usually viewed as an exciting opportunity, which is exactly how Chakrit saw his hunt for the perfect load-testing tool.
“It’s a good challenge to read the code and try to understand the technology that it uses and to come up with something that, if it works for our company, we can then contribute back,” says Chakrit.
Artillery.io is fairly new — like, infancy new — and parts of it have already evolved beyond what was possible even this summer when Chakrit was using it. But the opportunity to poke around the code and make modifications to it was an exciting challenge in and of itself.
Innovation is a core value at Myplanet. We relish the opportunity to try new things and expand our understanding of different fields by testing and experimenting. For Chakrit, this summer’s innovation playground was load testing, but it could have been anything. Bring on the next boatload of sockets—or whatever other complicating features you can think up — Chakrit’s ready for it.
Written by Leigh Bryant