Test Strategy in Mobile and Big Screen Testing

This story is co-authored together with Sivamoorthy Bose

At the moment the software applications become an integral part of our daily lives. QAs are the most familiar with web based application testing, however there are much more other types of applications. In this article we would share our experience of testing mobile, STB, big screen and casting applications.

Mobile

Usage of Smartphones have been increasing these years. Businesses are now focusing on integrating and delivering mobile applications to provide ease of using business features to engage all sorts of users. Quality assurance of mobile applications ensures the efficiency, user experience, performance and robustness are meeting the user expectations.

Mobile testing is always complex when compared to the web UI testing. Mobile is used across different regions with more number of users accessing it widely. So it needs through testing especially on the applications that are downloaded from both IOS and Android stores. The Mobile application function has different types:

• Native Application (It is specific to Android or IOS).
• Hybrid Application (It can be accessed by both Android and IOS).
• Web Application (These are accessed through the native mobile browsers).

Both functional and non-functional testing has to be done for all the mobile applications. The functional testing involves, Critical business flows, UI enhancements, backend verifications and cross browser testing. The non — functional testing involves endurance or performance testing to test and validate the Stability of the application. A complete security and penetration testing should be certified before launching.

Mobile automation testing can be done with Appium framework and the web application can be done with Selenium or Cypress frameworks. It is always better to do testing on real devices than using simulators for certification.

Usually functional testing covers the application launch, verify key business flows and the usability testing covers the design and layout verification. The other important testing is compatibility testing. This verifies the application on different mobile devices with different memory and processor. Also test both on Android and IOS platforms. For the web versions, it has to be verified on different browsers like Chrome, Safari, Firefox.

Performance testing has to be verified for Device, Network and Server performance.This type of testing validates the request and response time between the network and the server. We should also validate the battery consumption and Memory consumption test, this could be one of the reasons for the slowness of the application. For testing performance manually try using a light version of phones or some old model. Such devices would more obviously show all performance issues.

Checklist for mobile application testing

Mobile testing also has a big dependency on the hardware. It is important to test compatibility in different devices, screen size, resolutions, OS (and browsers for web applications). Of course it is impossible to cover all phones/ tablets models as well as many Android and iOS versions. We advise you to investigate the market usage in the country where your application would be used or analyse Google Analytics data of your application and cover the most popular.

Mobile type of devices foncees having different types of Internet connection. Please keep in mind testing major use cases with Wi-Fi as well as sim card mobile data connection. Cellular internet connection is commonly forgotten during mobile testing. Usually QAs have Wi-Fi only models of IPads to save costs of test devices and experience lack of sim cards for the testing. We were making the same mistakes and skipping quite severe bugs specific to cellular internet connection.

While testing on mobile we advice to keep in mind:

• Interface for different screen orientation (portrait and landscape).
• Making/Receiving/Rejecting voice calls while the app is running.
• Sending a mobile app to background (and resuming it).
• Keyboard for edit Fields as well as checking responsiveness of the drop-downs.

Big Screens

Set-top Boxes

Set-top Box is a device, which displays on connected output TV content. As input it can use cable television or IP TV signal. Thanks to STB the TV content can be shown on TV devices or even monitors.

While we talk about testing on Set-top boxes, we can categorise into 2 types of testing. One the extensive testing done on the set-top box device itself. This testing basically ensures the video & audio quality, tuner quality and also other interface testing. The other type of testing is done on the application that is deployed on the set-top boxes, that includes both middleware and the UI that we see on the TV screen.

Most of the Setup box UI testing is done manually and the stress or endurance testing is done by automation. The automation scripts are executed to the set-top — box and the logs are captured to validate the crash failures. This type of testing needs a lot of creative & out of the box scenarios to make sure the application works properly. Here are the few test scenarios that have to be focussed on the below functionalities.

• Electronic Program Guide — Test Reminders, Check for program time and date
• Digital Video Recorder (DVR) — Check for the recording capacity, rewind, play recording etc..
• Channel list — Look for the channel list and scroll for navigation
• Search list — Give multiple searches and check for the functionality based on program, movie, artist, etc..
• Video on Demand (VOD) — Check for the list of Content available, play stop, rewind etc..

While testing applications for STBs it is important to keep in mind testing all possible Internet connections (throw LAN or Wi-Fi). Also covering use cases with low speed or not stable Internet connection would be very helpful.

We advise you to include into the STB test approach installation testing. We would compare this part with the login module. If the user is not able to get updates on the hardware, he/she can not enjoy even last added features.

While testing on STB, it is important to keep in mind that every touch screen has hard and soft keys. The soft keys are part of the UI and they can be pressed by making the button focused and pressing on it by Ok (or similar button on the remote control). The hard keys are keys on remote control. The user can press back, exit, play, pause, record buttons on UI or directly on remote control. As QA you need to cover all hard keys and all soft keys on every screen.

If we would compare STB with other apps, it is important to include a test scope specific to firmware features (like peering of remote control, connecting storage devices, supporting several versions of remote controls or switching between different languages).

Other Big Screens

By the big screen applications we mean applications, which can be installed on Big Screen devices, which are usually used in living rooms. It can be Apple TV, Fire TV sticks/ boxes/ cubes, Xboxes, different smart TVs with Android TV (like some models of Sony or Philips) or OS created by TV vendors like Web OS (new models of LG TVs), Tizen and Orsay (Samsung TVs).

See more extended list of vendors and OS in section “Smart TV platforms utilized by vendors” in the Wikipedia article.

Those big screens can use native, hybrid or web applications. However if you are thinking about the test automation on big screen apps, consider the Suite.ST framework. It allows us to launch automated tests on real devices.

Like in STB testing, it is important to cover with tests LAN and Wi-Fi Internet connection, soft and hard keys as well as installation testing. Compared to STB those devices have the complexity level of different hardware and OS combinations.

Casting

Google Cast

Google Cast is a technology for playing Internet streamed audio/video content on compatible TV/monitor or audio systems. The user needs a sender and receiver compatible device for casting.

Sender is a place where the user controls the playback. It can be mobile/ web apps which support Google Cast feature. Mirroring the screen from Chrome desktop browser or Android mobile phones is also possible.

Receiver is a place where audio/video stream is played. The chrome cast receiver is commonly built into TVs with Android TV operating system, devices like Google Nest, some sound bars and speakers. Chrome cast devices can be also used as receivers. They are designed as small dongles that are powered by connecting the device to an external power adapter or USB. Video-capable Chrome casts plug into the HDMI port of motor or TV, while the audio-only model outputs sound through audio jack. There are many different models of the chrome casts: 1st Generation, 2nd Generation, Audio (for use with audio streaming apps), Ultra (supports 4k resolution), 3rd Generation.

While testing the Google Cast feature we advise using different test levels: component testing for testing separately senders and receiver apps and integration testing to check flow of casting. If casting is a new feature to you, it is worth checking test cases prepared by Google. Those 2 files suggest the essential scenarios of the sender, different types of receivers, multi-sender scenarios, playback or establishing connection.

For a successful integration testing approach, keep in mind different parameters which impact on casting and cover different combinations. Classical examples would be different receiver and sender devices and quality of stream. If you start mixing those parameters, your test coverage would be much bigger. How the scope may look like:

You can enlarge your approach with DRM or non DRM content, different types of players, different streaming protocols (more about them). Knowing the app architecture and talking to developers would help to define those parameters.

By including more parameters you may face large complexity and would not be able to manually design peers. There are many pairwise tools, which can mix combinations in smart algorithms (list). We offer Test cover, as it supports constraints when some combinations of parameters are not possible.

AirPlay

AirPlay is a technology designed by Apple that allows wireless audio/video streaming as well as screen mirroring between the compatible devices. Like in Google Cast the AirPlay uses the concept of sender (IPhones, IPads) and receiver (Apple TV).

For that reason it is worth following the same test approach to apps supporting AirPlay as for Google cast: component testing of each app separately and integration testing with different combinations of parameters.

Testing mobile, STB, big screen and casting applications has not so many differences compared to web applications. Of course such applications have bigger dependency on hardware and some new use cases related to specify how the applications are being used. If you always wanted to try testing them, switching would be very easy going. Just do it!