Paying per Test or Virtual User is Broken
Selling performance testing services by the number of tests you can run or virtual users you can simulate is a thinly veiled process of extracting the most money in the shortest amount of time from performance testers globally.
Artificially limiting the concurrency (virtual users), time or quantity of these tests is inherently broken!
If you have ever used these sorts of services, then you know exactly what I am talking about.
A real world example
In our experience at Flood IO, most tests run like this:
- Create a test plan
- Shakeout the test with some virtual users
- Realise 10 minutes in that the test plan is wrong
- Stop the test and mutter to yourself, fixing the test plan as you go
- Get it running with some more virtual users
- Realise 10 minutes in that the test plan is less wrong
- Stop the test, face palm and fix the test
- Get it running with another bunch of virtual users
- Realise 10 minutes in that you might have it right and stop to really ramp it up
- Get it running with lots of virtual users and a smug grin on your face
- Realise 10 minutes in that you forgot to to turn off instrumentation on your site and nothing makes sense at load
- Stop the test, swearing, fix the test plan
- Get it running with lots of virtual users again
- It runs well and you extract some usable data, get the data from the system and send it off with a “Yeah, this was no sweat” note to whomever wants it.
The addiction of billing
In the world of per test or per virtual user billing, you would have been charged (penalised) for every one of those “stop and fix the test plan” steps. More often than not rounded up to the nearest unused quantity of your quota.
“Yes, that’s right.”
Just used your monthly quota of tests? Need to run 1100 virtual users instead? “Sorry but you can’t proceed until you upgrade to the next plan”.
The worse thing is that the load testing industry is so addicted to this idea of selling per test or per virtual user that it’s become the norm and everyone just accepts it.
Wean yourself off the addiction
At Flood IO, one of our most common questions is “How many virtual users does your XYZ plan include?” to which we always answer with totally true but seemingly confusing statement of “As many as you want”.
All the planning in the world can’t predict actual concurrent loads and response times. You might have ramped up too fast, you might have forgotten to take into account a user flow, you might have left some debugging in place which skews your results, your test plan may be unnecessarily complex or your site may already have serious performance issues. Any number of things that make getting concurrency right the first time, every time, impossible.
If that is the way load testing goes, then why is it that most performance testing platforms charge per user or per test?
This frustration was one of the primary factors in why we created Flood IO and is what sets us apart.
Do something different
At Flood IO, we do something different.
You boot some load generation server nodes, collectively called a grid. Then you can run as many tests as you like on that grid. Instead of paying per test, or per virtual user, or per report or per anything, you just simply pay for the minutes your grid is running.
So instead of the real world example blowing out your test or virtual user quota with our competitors, on Flood IO, it would cost you an hour or two of grid time.
This also means you can start up a small grid for testing at a very cheap rate (or use the free grid hours you get every month as part of your plan) and then when you are ready to scale out, boot up multiple grids to generate as many virtual users as you like.
Because of our distributed shared nothing architecture the performance you get from a single node is representative of many nodes. We let you start up to 30 nodes per grid with as many grids in as many regions as you like attached to your flood test. Run as many tests as you like. Throw as many virtual users in as you like. This makes it easy for you to baseline performance and plan to scale out from there.
As one of our happy customers once said:
Flood IO, stupid simple, stupid cheap!
We believe strongly that the tools made for our profession should be designed for our profession to be as efficient as possible. We believe penalising quick exploratory load experiments is counter productive. We don’t want to get in the way of your testing, just help you get your testing done.
Flood IO is an insanely easy to use load testing service that saves you time and money, allowing you to focus on what’s important — Your product.
Visit flood.io and sign up for a free account.