How to test on smartphones without smartphones?
Mobile phones are everywhere and almost every service in the world has its own application. If not, the website is responsive and can be used on a small screen with bad network, and not only on your desktop with a fiber network. As a tester of a website or an Android/iPhone/WinPhone application, you need to test it as a real user, meaning on one or more smartphones or at least on the most used by your users/customers. It includes manual and automatic testing though. What if you don’t have one billion €uros/$ollars/£ounds to buy all the smartphones needed to be tested? Will you rely on Chrome dev tools for browsers application, knowing that developers already “quickly” test their work with it?
Browsers tools
Browsers can be helpful to help testing web applications that are available in a browser. Chromium/Chrome dev tools are the most advanced tools for this.
A few smartphones screen resolutions are available so that you can use them quickly (Galaxy S5, iPhone 6, etc.), you can also configure some more or directly chose the screen resolution you want to test.
You can also alter the network by using the throttling feature. A few ones are presets but custom ones can also be configured. It’s always a good idea to test your product with a bad network.
Firefox also have a responsive design tool but not yet a network throttling tool in stable version.
Breaking news: it’s now available in nightly build starting with Firefox 52
For Safari, you can use the “Network Link Conditioner” which is available on Apple’s developer portal.
Dedicated tools
This is not enough to use “developer tools”, maybe one day browsers will provide “testers tools”. Let’s talk about emulators, simulators and cloud services that allows to use remote emulators and real-devices.
Emulators versus Simulators
Sometimes we talk about simulators, sometimes about emulators. Is it the same or not? In fact, terms are often used to talk about the same thing, i.e. a way to run an application or a mobile device browser on your own desktop environment (in the browser or with a dedicated application).
But if you want to be more precise, the emulator simulates not only the software but also the hardware of the original device. It is closer to reality than the simulator.
Indeed, the simulator simply simulates the environment of the originating device without attempting to simulate the hardware, which can have an impact on the observed result.
All platforms make available emulators/simulators. We have Apple iOS simulator that runs iOS apps on Mac, Windows Phone Emulator by Microsoft and BlackBerry Simulators by RIM. Also, Google’s Android SDK includes a mobile emulator. But these solutions are incomplete, and there’s no way at the moment to install Chrome in a Android emulator.
Keynote
Mobile testing by keynote allows manual and automated testing of mobile applications and websites on hundreds of the latest smartphones, feature phones, and tablets. It’s some real devices made available for testing as a commercial service. As a tester, I wanted to try with a trial account but both of my emails were rejected, probably because it was not a real company email. That’s weird and made me run away!
Testdroid
Bitbar Testdroid service is a cloud-based mobile device farm equipped with real Android and iOS devices for automated and manual mobile app testing. 381 browser devices are available and allows you to test on real devices, not emulators or simulators. Very few smartphones are available for free (I tried with a free for open-source project account) and I experienced some failures when I tested it. To access a private network, a VPN connection can be implemented. I didn’t test it.
TestObject
TestObject has a big list of available real devices for testing. You can easily select the one you want with smart filters in this page. Because TestObject has been acquired quite recently by Saucelabs, we may hope it will help Saucelabs to be even more interesting.
Saucelabs
I’ve been using Saucelabs since a long time for some manual testing on browsers I don’t have, or that I just don’t want to install for a quick test. See the list of Platforms. Saucelabs has so many combinations available.
Lots of Operating Systems like iOS (8.1->10), Android (4.0->5.1), Windows (XP->10), MacOS (10.8->10.11) and Linux, and lots of browsers including Opera, Chrome, Firefox, Safari, IE and Edge. That’s more than 800 combinations available with a click, for manual testing but it can also be used with Selenium and Appium frameworks.
For private network access, a VPN tunnel called Sauce Connect can easily be used.
All these are emulated browsers. Saucelabs also provides a short list of Real devices but I couldn’t use them because they are not available for free accounts.
Saucelabs also allows you to have your automated checks/tests to use their browsers and you can have several tests running in parallel, depending on the Sauce plan you subscribed to. That’s 5 concurrent executions with a free for open-source project account.
BrowserStack
BrowserStack is quite similar to Saucelabs regarding access to emulators. For local access you don’t need any VPN manually executed, all you need is a plugin only available in Chrome store.
You can have access to dev tools very easily just like you would use them in your own browser. Quite impressive, contrary to Saucelabs that only allows to open the dev tools in the emulated remote browser.
Real devices access are supposed to be freely available for open-source projects if you add BrowserStack icon and link in your open-source project website and github. I couldn’t test it yet and will upgrade this article when I will have some more insights.
One cool stuff that BrowserStack provides is the ability to quickly test responsive design in a predefined list of iOS, Android, OS X & Windows devices.
You can also have screenshots taken with a selection of smartphones and tablets.
Perfectomobile
Perfectomobile allows access to real devices and allow manual and automated testing (on iOS, BlackBerry, Android, Windows Phone). They provide an interesting tool called The optimizer which gives you a list of devices you should test first depending on the location your clients are, OS mostly used and type of devices (smartphones or Tablet). I couldn’t test it.
Genymotion
Genymotion is a fast and easy Android emulator, so with it you will only be able to test with Android. See the cloud platform or Android on AWS EC2, it might be a great help for testers that only need Android testing. And it’s built in Lyon.
Xamarin
Xamarin is an IDE that helps build C# apps on Android, iOS, Windows, and Mac. But the company also provides a Test cloud. It adds a test recorder to quickly create UITest scripts by capturing interactions. Xamarin only provides real devices and no emulators. It allows to run automated tests and don’t give access to devices for manual tests. I didn’t test it.
Conclusion
Cloud services that allows you to access real devices or emulators for testing are numerous. You must test them before making your own choice, but don’t forget that maybe the best choice for you is to have a few real devices.
Originally published at www.lyontesting.fr.
