What You Need to Know Before Testing Smart TVs

While I was growing up, TVs weren’t smart, let alone flat! Although I am not old enough to have experienced black and white TV or switching between channels with a knob, TVs were just a glorified monitor that was maybe connected to a VCR or set-top box in order to watch your favourite shows. Binge-watching, at the time, meant renting the boxset from your local Blockbuster. The TVs had no pop-ups to check “are you still watching”, or buttons for us to “Start Over”. We had to manually rewind the tape to start anything on demand!

Fast-forward to 2020 and Smart TVs are our chosen mode of consuming content and with this, apps have moved across from the mobile space to the TV space. But where do they come from? And how are they so good? Designing, developing and testing new TV Apps on Smart TVs each require a unique skill set since there are many things to consider when doing so…

Who am I?

I have been quality-assuring streaming TV applications since 2013, testing across multiple platforms and devices, from mobiles and tablets to Smart TVs. I have worked on many projects in the last couple of years, from large broadcasters launching their first-ever Smart TV application to Pay-TV Operators who are replacing legacy services like cable or broadcast TV. Over these years, I have learnt a thing or two about testing on larger devices and would like to share some inside tips with you.

But before I start, if there is one thing you are going to take away from this, it is that the role of a Quality Assurance Engineer is not limited to testing the application at the end of the development cycle. It is about quality assuring the software development process, from start to finish. It starts with ensuring your device inventory is always up to date and covers a range of devices, through to validating the requirements and design and finally to ensuring that the application is ready for the market.

Even if you are a Quality Assurance Engineer with a lot of mobile testing experience, testing on Smart TVs is going to be a whole new ballgame as these devices are still on the way to becoming commonplace. Whilst testing services on these large screens, I’ve found three core things to consider. For starters, a TV is larger than the phones and tablets that you’ve been testing on, secondly, each device comes with its own complexities with regard to design or user experience, and finally, the endless fragmentations coming from the device manufacturers´ specifications.

Size Does Matter!

Testing on Smart TVs differs quite a lot from testing on smaller handheld devices. Firstly, the fact that TVs and screens are so much bigger, makes it difficult to find enough room to squeeze one in within your workspace. So how would you find the space for testing multiple devices on your desk or even a small room? It is a logistical nightmare to move and shuffle TVs around!

Additionally, you’ve got to worry about mixing up remote controls or worse, the remote controls interfering with each other — so you’ll need to practise some “social distancing” between TVs. What’s more, your testing may be interrupted by your colleagues! You can solve this problem by creating a testing lounge so that Quality Assurance Engineers can be “isolated” to test multiple TVs simultaneously in peace.

Smarter Designs for Smart TVs

Quality assurance isn’t just verifying the functions of an app on the TV, but also validating the UI/UX with the design guidelines of the device manufacturers, such as Samsung or Sony. The sheer size of the screen leaves more real estate to be utilised for design, but one needs to consider how to use that additional space. You should consider that the user is not holding the device in their hand, but rather sitting several meters away. Also, they are using a 4-way navigation pad on the remote control rather than their fingers swiping on a touchscreen. There are also other aspects to consider with magic remotes that are adding yet another way for the user to interact with the device.

Each of the device manufacturers have their own design guidelines requirements, that include both functional and design elements. For example, button actions, navigation, font size and specific feature requirements. You would need to be aware of all of these considerations not just at the initial design phase of the project, but also during the final quality assurance phase since some of these actions are not defined on UI/UX wireframes or workflows. Without conforming to most of these requirements, the application won’t be certified and therefore won’t be available on the store!

Down the Rabbit Hole…

Much like the design and development phases while creating apps on Smart TVs, whilst testing, one needs to consider the many facets, nuances and performance of each device. There are many key manufacturers such as Samsung, LG, Sony, Philips, Apple and many more. No, Google does not make TVs but just the Android OS that runs primarily on Sony and Philips Smart TVs. This gets more complex as the hardware and software varies between each manufacturer. And within that, they change from year to year, model to model and series to series. These variations have huge impacts on the performance of applications, where the same app will behave differently, regardless of your tech-stack.

Let’s take Samsung as an example. Their Orsay OS ranged from 2012–2015, and the Tizens were introduced in 2015 through to the present, each year group having its own nuance. Within each of these years, there are model groups, ranging from “Standard” to the top of the range “Premium” devices. Each of these model groups have multiple series that signify additional differences in hardware and software combinations. This part can get a little tricky to explain further as there is a limited amount of information available on this topic — “generally the higher the series, the higher the specifications on that model”. If you’re having trouble keeping up, don’t worry, here’s the complicated yet simplified guide on how to decipher your TVs model number.

While testing these devices you need to know or have handy information about the supported features on each model group. Keep in mind, the feature support on devices from just the same manufacturer can vary within a single year! For example, the Multitasking feature on a TV means the opening of a second application while the current application runs in the background, is not supported on all Samsung Tizen models. The consumer who is used to this on mobiles expects it instinctively. And for the creators, not just a Product Owner, but also the Quality Assurance Engineer should be aware of this technicality. With such complexities, one can’t assume things! Diving deeper into the rabbit hole, as this can get even more complicated, as each year and model group have varied support for video formats, encryption solutions, playback or player features like subtitles and internet connectivity specifications.

All of this can be difficult to comprehend for professionals in the field, let alone the consumer who needs to make an investment.

The Basics or 101s

While building the device inventory, it is important to have a good range of years and models that cover low-, mid- and high-end devices from each year. If you have a limited budget or resources, then you’re better off stocking and testing against the lower-end devices. I know what I said about assumptions earlier, but when the applications perform well on the lower-spec models, one can reasonably assume that they will perform better on the higher-end models. Another way to mitigate the problem is through automation and using a solution such as Cypress. This will allow you to perform end-to-end testing without the constraints of the physical device as you are testing javascript in a browser. Incorporating this into your Continuous Integration pipeline will allow you to improve your test coverage, catch bugs before they get to QA — no, automation does not make QA redundant — and therefore ensure that the overall product is of a higher quality. However, this does not mean that you shouldn't test on the actual devices.

Whether you’re an experienced Quality Assurance Engineer or newbie testing applications, switching from mobiles to Smart TVs can be daunting. The sheer size of the devices considered, multiple UI and UX guidelines adhered to, and technical disparities understood — there’s going to be something new that pops up with a new model and a new year just like the good old days of mobile software testing. The nature of new technologies is that it will evolve faster than we can grasp, but from my experience in Quality Assurance, the most important considerations are understanding the device capabilities and having the right information handy. All of these things should be documented, followed and considered from the onset of designing development and quality assuring Smart TVs apps. I’m only stating it because the quality is always expected and questioned only at the very end.

P.S. Need some help testing your TV Apps? We offer OTT testing as a service, including basic and premium OTT features, pre-defined and bespoke test cases, as well as device manufacturer certification checklists and app validation. Find out more and reach out here.