Mobile Testing: The Uphill Struggle
by Chris Bower
The Early Days
When the Financial Times launched its first Mobile Web App in 2009, little was known about the potential impact mobile platforms would have on the way we consume news and information.
The existing, but recently launched, m.ft.com site was already a successful platform in its own right, primarily aimed at the business user on Blackberry Devices which were unable to effectively render www.ft.com. In 2007 the launch of the Apple iPhone device and subsequent success of the App Store, provided an opportunity to offer a portable digital FT that FT.com Desktop users could access from their pockets at any time. The iPad was launched a few years later along with a small number of Android alternatives.
The FT met this new challenge with native apps that would work on each of these individual platforms. The initial presence in the App Store and the subsequent retreat in 2011 are well documented. What has been maintained throughout has been an opportunity for FT readers to access the FT on whichever platform they choose.
In 2010, the FT Mobile QA team was in its infancy. One part-time test resource was assigned a handful of devices based on what was currently being actively sold and marketed. This was a fairly simple task given there wasn’t much choice on offer at the time. The devices included handful of iPads, iPhone 4’s, HTC Desire’s, Blackberry and a Samsung Galaxy 7”.
Current Status
Fast forward four years and the landscape has changed dramatically. You’d be hard pressed to open a paper, watch tv, or view any website without seeing an Apple, Google, Samsung product based article or offer. Such is the success of the mobile platform, all these companies are all suing each other for various copyright infringements in multi billion dollar lawsuits.
Internally, the success of the Mobile platform with FT subscribers has created an appetite for more mobile products from the FT portfolio and more functionality within each. The FT Web App alone now features richer content, more sections, inline videos, high-res photos and graphics, swiping, clippings, recommended reads, International, morning and live editions, full page Ads, improved analytics and tracking.
We now find ourselves supporting several products across mobile and desktop including:
- www.ft.com on all size mobile devices
- m.ft.com on all small screen mobile devices
- app.ft.com on iOS devices
- The FT Web App on Android and Windows 8
- The ePaper reader (an interactive PDF of the physical newspaper)
- Investors Chronicle and FTAdviser apps
To support this, the FT Mobile Test team are now a team of ten people in two countries, specialising in multiple platforms, and running two independent test automation frameworks.
The FT mobile device pool now comprises an impressive collection of:
- 23 x iPads
- 4 x iPad Minis
- 22 x iPhones
- 6 x Blackberries
- 10 x large screen Android tablets (Google, Sony and Samsung)
- 7 x Medium screen Android tabs (Google and Samsung)
- 25 Android phones (HTC, Google, Sony, Motorola and Samsung)
- 7 x Windows 8 tablets and laptops
The reason we have so many of each is down to the fragmentation within each platform. We support iOS 5, 6 and 7 as well as Android 4 and above (ICS, Jellybean and KitKat) and the variants thereof.
Mobile Testing Fragmentation
Android
Although they might appear to look and operate on the same basic platforms, all Android devices run their own OEM builds of ICS, Jellybean with different screen sizes, resolution and memory etc. Looking at the July 2013 Android Device and OS Fragmentation graphics from an article in OpenSignal, the scale of the problem is clear:
As of August 2013, there were almost 12’000 Android devices in use around the world. No company in the world could manage that sort of fragmentation. So it is up to us to use a combination of data analytics and specialist judgement to adequately manage the distribution of devices. Fortunately, most FT readers meet a certain profile and are likely to own a mainstream device.
Source: opensignal.com/reports/fragmentation-2013/
iOS
Generally the iOS devices fall into fairly similar categories. In terms of display size and resolution, the iPhone 4 and 4S are almost identical (as are the current 5 generation) and the team have never found an issue specific to one and not the other. The same applies to the iPad 1 and iPad 2 which have the same screen resolution, as well as the iPad 3rd,4th and 5th generations with retina display. In these instances you would really only require one device of each type. If investing in new devices, we would always recommend going for the newer generation as these will be supported by Apple for much longer.
iOS Betas are another area we have to be on top of; Apple users are extremely loyal and almost evangelical when it comes to new device launches and software releases — so any bugs are quickly spotted. In the early days of the FT Web App development, a runner was dispatched to the Apple Store in the US to retrieve a very small number of the very first iPads a couple of weeks ahead of its UK launch.
Device Management
Unfortunately, the availability of so many rapidly-evolving platforms creates further problems for the development and QA functions. With the increasing number of new devices on the market and the maintenance and support of existing platforms, physical device management alone can be a full time job. This is simply down to wanting to support the largest possible user base. Mobile contracts in the UK typically run for a two year period and a device can stay on the market for two years so the support lifecycle of any one device can run for four years.
The Mobile platform usage information we receive from our Data Analytics team allows us to prioritise platforms and keep our own set of devices upgraded and maintained accordingly.
Given most devices come in a limited range of black, white and silver colours, within the team we colour code our phones and tablets with garishly bright rubber skins. This helps quickly identify a specific device/OS combination. They also provide a layer of protection for when they are hastily shoved into their respective draws before being locked away.
Devices are often loaned to other areas within IT, Product and Editorial for verification and sign off. Keeping tabs on these loans is an unenviable task and we run a check in/check out system with a black list of particularly poor returners!
New devices are greeted with keen curiosity by our work colleagues, especially those looking for an upgrade. A selection of the cheeky requests I’ve had from people include:
- “Can I borrow the new iPad mini 2 for a few weeks — I’ve heard it’s much lighter than my iPad 1”
- “My husband/boyfriend/Wife was thinking of buying an iPad — may I show her one of these this weekend?”
- “This Samsung Note 3 looks nice, can I see if it’s better than my 2007 HTC brick?”
- “I’ve dropped my iPhone down the toilet — can I borrow a new iPhone 5S until I get a replacement?”
- “I’ve smashed my iPad — can I swap it with one of yours?”
- “Got any spare iPads I can give to my Program — you know, the one with 30 contractors and a £5M budget”
A team branded ’FT Mobile’ are also an easy target for forgetful colleagues who have lost their mobile chargers and connectors. As a team, we have to ensure their swift return as if we were a jealous, spoilt only-child.
Emulation
For IT departments with tight budgets, mobile device emulators or third-party device hosting are always being pushed as a solution. Emulators give a good feel for how an app or a website will render and function, but will only get you so far. On an Emulator and you can’t adequately test for issues with memory usage, app crashing, flickering, switching between apps, refreshing, battery, performance and connection issues, that we often come across when testing mobile products.
WiFi, 3G, 4G and Airplane mode
The way an application behaves can vary considerably depending on the connection type. As Offline-mode is a key feature of our Web App on every platform, this is something that we test for continuously. Behaviour can vary depending on an individual users connection speed, availability of signal, WiFi restrictions etc. We sometimes find it extremely challenging to recreate a use case involving a specific device and coverage scenario.
The Future
Several years ago it would have been impossible to predict the meteoric impact Smartphones and Tablets have had on our lives. Habits have changed and a selection of recent amusing stories from the media highlights these trends:
Tourist walks off Australia pier while checking Facebook
Smartphone obsession makes ignored partners cheat: study
and a slightly more dramatic…
Australia’s obsession with smartphones and tablets could be killing us
The FT subscriber is now furnished with a device that contains a portable newspaper, accessible at any time that will automatically update and provide a huge archive of news and resources. As our usage statistics show us, the FT is no longer considered a morning-only or daytime read.
As a team, we are responsible for ensuring that each and every user is able to access our content reliably, at any time, and on whichever device or medium they choose. With the way the FT product, content, devices, OS’s and networks are evolving, the task becomes more challenging each day.
In the next few months there are plans to improve the FT experience cross-platform and merge the existing products into single Digital platform that would scale and work for everyone. This may simplify some of the problems we face in trying to manage quality, but the release of new platforms is likely to continue to grow. Rumours are always flying around about the Next Big Thing. Whether it’s Smart TV apps, in-car Apps, Google glasses or the iWatch, the team have to stay ahead or at least up to date with the technology and more importantly, our users’ consumption habits.
Hopefully this article will have helped you think about what you need to consider when planning your own QA framework and coverage.
