Automation strategy for mobile first testing
Now that we have understanding of what is mobile first, how is different, what to test for and some tools, now it is time to talk about Automation.
The purpose of automation is to eliminate repeated rote tasks. With that in mind lets see how we can tackle the chief troublemakers for mobile website testing.
Functionality
This is the most obvious one. Just like any other site(desktop), you want your mobile first sites to be functional. If you already have automation infrastructure, tests it should not be too difficult to adapt those tests to this world.
- You use selenium: Then you can use appium to add additional capabilities into your desired capabilities hash and some other setup and you are ready to go. This testdroid article goes a bit deep to this
- You are using chrome driver: Just like you can use chrome emulator to do you manual testing for various device modes, it ridiculously simple to do the same from your automation.
Which to choose — use chrome emulation if you are starting out and do not want to get into appium setup hassle but if you want more fidelity and even actual device automation then choose the appium approach.
Layout
More often than not this is a very tricky thing to verify. You change style on a button and verify it on desktop and mobile, but tablet gets screwed up. It is not reasonable to expect to keep on checking layouts for each of style changes. This is where power of automation comes to rescue with visual comparison tools. The idea is that you establish baseline images and for any future changes you compare your stuff against these baselines.
Github gives you a very rudimentary image comparison tool that you can integrate with your test infrastructure. But image comparison is a very manual task, even a gradient change could trigger a test failure, or if you add a link to footer triggers cascading failures. Some of these problems are solved uniquely by applitools, give them a shot and they integrate well with saucelabs. Some best practices here
- Assert only part of the screen
- Exclude very dynamic parts of the page (ads, banners)
- Ignore gradients and colors
Ghostinspector is my current favorite, though sometimes brittle it lives up to my requirements. And it has other goodies — runscope integration, API endpoints, mobile mode that I like.
There is an alternative approach though: Galen framework offers a very programmatic way of validation and it doesn’t involve pixel-pixel comparison. I haven’t had enough experience to recommend one way over the other.
Performance
For mobilesites performance is more crucial than desktop because usually users are on worser network, worser device and will have latency. But they expect content to be instant nevertheless. To get started with performance testing pagespeed insights is a very good tool, it is optimized for mobile sites. Webpage test is another popular tool that has some capabilities to make it work with your local websites and on real phones. It is also possible to get a local instance of webpage test to make it work with your own devices and local networks.
Device compatibility
For mobile web testing, you have to come to this aspect last. Modern browsers are pretty good at abstracting away any discrepancies of the devices sizes, types etc. Emulators are software programs that emulate what a device browser would look like. Chrome emulators are pretty accurate and would likely get 90% of accuracy but for more accuracy you can try android emulators (more accurate) and for iOS they are simulators (less accurate) these are little bit hard to get setup. More about this in a previous post. But running automation on emulators is slow, you have to do it parallely. So it will become hard to do on local machines. Cloud runners like sauce labs are very good candidates for this.
Automation on actual devices is impractical and prohibitively expensive, though many vendors would want to make you believe otherwise, current industry practice indicates the cost outweighs the benefits. But if you are still inclined to try real cloud device providers checkout smart bear, perfecto mobile and testdroid(my vote).
Conclusion
These are not easy aspects to get right but once you get there, your tests will make your life a lot easier. And will let you concentrate on more focussed testing.
