Flood IO is an insanely easy to use load testing service that saves you time and money, allowing you to focus on what’s important — Your product.

Visit flood.io and sign up for a free account.

Paying per Test or Virtual User is Broken

Selling performance testing services by the number of tests you can run or virtual users you can simulate is a thinly veiled process of extracting the most money in the shortest amount of time from performance testers globally.

Artificially limiting the concurrency (virtual users), time or quantity of these tests is inherently broken!

If you have ever used these sorts of services, then you know exactly what I am talking about.

A real world example

In our experience at Flood IO, most tests run like this:

1. Create a test plan
2. Shakeout the test with some virtual users
3. Realise 10 minutes in that the test plan is wrong
4. Stop the test and mutter to yourself, fixing the test plan as you go
5. Get it running with some more virtual users
6. Realise 10 minutes in that the test plan is less wrong
7. Stop the test, face palm and fix the test
8. Get it running with another bunch of virtual users
9. Realise 10 minutes in that you might have it right, stop it and ramp it up
10. Get it running with lots of virtual users, with a smug grin on your face
11. Realise 10 minutes in that you forgot to to turn off instrumentation on your site and nothing makes sense at load
12. Stop the test, swear, fix the test plan
13. Get it running with lots of virtual users again
14. It runs well and you extract some usable data, get the data from the system and send it off with a “Yeah, this was no sweat” note to whomever wants it.

The addiction of billing

In the world of per test or per virtual user billing, you would have been charged (penalised) for every one of those “stop and fix the test plan” steps. More often than not rounded up to the nearest unused quantity of your quota.

Seriously?

“Yup!”

Just used your monthly quota of tests? Need to run 1100 virtual users instead?

“Sorry but you can’t proceed until you upgrade to the next plan”.

The worse thing is that the load testing industry is so addicted to this idea of selling per test or per virtual user that it’s become the norm and everyone just accepts it.

Wean yourself off the addiction

There is a better way to do this. Instead of paying per test, or per virtual user, just pay for the load generation server time.

Actually, this is how you would pay for it if you were putting the infrastructure together to run a load test yourself. You would fire up a number of servers, then install your load testing software on them and finally run the test, collecting the results after.

If your test plan didn’t work, you would simply stop the tests, reconfigure them, and run the test plan again. You wouldn’t be charged for this mistake beyond a bit of your time and a bit of the wasted server time.

All the planning in the world can’t predict actual concurrent loads and response times. You might have ramped up too fast, you might have forgotten to take into account a user flow, you might have left some debugging in place which skews your results, your test plan may be unnecessarily complex or your site may already have serious performance issues. Any number of things that make getting concurrency right the first time, every time, impossible.

If that is the way load testing goes, then why is it that most performance testing platforms charge per user or per test?

This frustration was one of the primary factors in why we created Flood IO and is what sets us apart.

Doing the Usual… Unusually.

At Flood IO, we do the usual… unusually.

The company was created by load testers, people that were sick of paying per test and just wanted a system that would boot up, control, run the tests and gather the results of what they were running.

So, as you would do by yourself, you boot some load generation server nodes, collectively called a grid. Then you can run as many tests as you like on that grid. Instead of paying per test, or per virtual user, or per report or per anything, you just simply pay for the minutes your grid is running.

So instead of the real world example blowing out your test or virtual user quota, it would cost you an hour or two of grid time.

This also means you can start up a small grid for testing at a very cheap rate (or use the free grid hours you get every month as part of your plan) and then when you are ready to scale out, boot up multiple grids to generate as many virtual users as you like.

Because of our distributed shared nothing architecture the performance you get from a single node is representative of many nodes. We let you start up to 30 nodes per grid with as many grids in as many regions as you like attached to your flood test. Run as many tests as you like. Throw as many virtual users in as you like. This makes it easy for you to baseline performance and plan to scale out from there.

As one of our happy customers once said:

Flood IO, stupid simple, stupid cheap!

We believe strongly that the tools made for our profession should be designed for our profession to be as efficient as possible. We believe penalising quick exploratory load experiments is counter productive. We don’t want to get in the way of your testing, just help you get your testing done.