Why you need a custom device lab?

A lot of mobile teams i have worked with often have this dilemma, i.e. if they need a custom device lab or can they live with a third party device cloud. From our experience at TestVagrant, it is a fairly straight forward choice to make based on the kind of product, the market and the tech team you are dealing with.

We work with clients big and small. On one hand we have smaller teams that have just conceived the idea and trying to gauge the market response with a basic app and on the other hand organisations that are established startups on a mission to disrupt the market with diversified and distributed teams. What is common between them is their penchant for cutting edge technology, development practices and a zeal to release very often that bring the difference. We consolidated the kind of problems we faced and solutions that worked for us with respect to test infrastructure.

Here are some questions that helped us set direction and you should think about these too:

1. How many times in a day you need to engage devices for test run? Consider the kind of CI integration, the number of automated tests and the number of check-ins per day.
2. Is your product in B2C space? And if so, can you really live with a fix number of hours per month given the frequent releases you deal with being in B2C space.
3. What is max number of parallel sessions you would want to have to be able to run tests more frequently? Does any cloud provider allows the same?
4. Are you comfortable sharing your test code or the app with a third party. I am not saying that means any compromise but still makes things a tad bit complex.

In my opinion, public clouds do make sense for teams that have a more determined pace(read relatively slow) and are making controlled changed to their product and hence the duration that they need devices for testing is fairly predictable.

In addition, they give you access to a wide variety of devices and you can get almost any make/version available on demand. This is almost impossible if you are setting up your own lab.

But is that something you really need often? What if you are a fast moving B2C startup, releasing very often, wanting to run a large number of automated functional tests every now and then. You can’t rely on a third party owned device farm for your day to day needs. It just becomes cumbersome and costly.

Let’s not undermine the immense value that these third party device cloud organisations bring to the table but at the same time lets not shy away from asking if we really need to follow the fad.

Also, find below a notional comparison of cost between some of the prominent players in third party device cloud space and that of a custom device lab setup.

For the sake of this comparison, i have chosen a use case where the team has an unlimited access to 4–5 parallel sessions for a project duration of a year. Also, for the custom device lab setup i have assumed a one time setup cost of USD 2000 (USD 1000 for a mac mini that can host 4 connected android devices valued at the remaining USD 1000).

Data sourced from respective company’s website

Have a custom device lab if you value faster feedback. You can always use a third party device farm on demand, if and when you need a specific make/version to test a certain scenario. But for your daily needs in an agile environment, third party device farms/cloud would slow you down.

What you need is a mechanism to setup a local device lab effectively. A custom device lab that is easy to setup, can be smartly configured and shared within your company and can be plugged in to your existing test framework out of the box.

Look forward to this space for an exciting announcement on the same sooner!