How DAZN automates tests on Tv devices? Enter the TVLAB!
Meet the TestOps team!
DAZN has always been a place with great and talented people. Not forgetting the technology stack as well as the product itself, people are what make DAZN a special place. When the pandemic started, the office became deserted. I miss these people, their energy and positivity were always major positive factors when working at the office. It was understandable that things were changing, and we could struggle to hit the same on-site scale. Even though I’m all for direct interactions, the only thing you can do is just accept it & adapt to it.
At DAZN, we try to move to a flexible way of working, which means that our teams can choose whether / when to work from home or from the office, but also bearing in mind that there may be situations where the presence of teams in the office may be needed.
TestOps team cannot really choose whether work from home or from the office. Why? Keep reading.
A bit of history
Almost 4 years ago a side project for TVs, Consoles, STBs called Test Automation Framework (TAF) was created. It was a project without a dedicated team, most of the work was done while managing other priorities. Along with the dynamic development of the company, we quickly realized that this is something that will help us grow and scale. Based on this approach, a special team called TestOps was formed. TAF wasn’t the team’s only responsibility. At the same time, we’ve started building our TVLab and Virtual Remote which was the obvious development and completion of our mission:
so, to provide a fast feedback loop for our internal teams on living room application quality.
TestOps focus is to ensure that other engineers are able to work efficiently from home, as they would be if they were coming to the office. You may think: “Why some people need to go to the office to help the others work remotely?”. I think you will be able to answer this question at the end of this article. But first, let me add more context here.
DAZN is a global over-the-top sport streaming service. The service is live on a wide range of devices, from phones and tablets to web browsers, smart TVs, game consoles, STBs and more. To specify, data shows that a high % of service consumption hours comes from living room devices (TVs, game consoles, STBs).
What do we do to ensure high quality products for that category of devices? We automate!
Test Automation Framework (aka TAF)
In the standard approach to E2E UI testing, Selenium has been the market leader for many years. The concept is simple: there is a driver that communicates with a web browser, which is a link between the test code and the tested application. This driver is installed directly on the computer/device/server which also performs the tests and runs the browser with the tested application. Implementing the same approach on Living Room devices is not that simple though. For example, we do not have a dedicated integration for a WebView engine on the Sony PS5 console. In this case, we have to create our own communication system between the tests and the application running on the device.
Our solution is based on the well-documented WebDriver standard. Considering that the test frameworks based on this standard communicate through standardized queries, we can easily add our own implementations or completely replace the logic behind each and every one. For the Testing Framework, it does not make a difference, because it’s sending the same requests as it would be when running in the standard configuration. Our solution implements an additional actor which we call “Taf-Middleware”. It’s a cloud service that processes queries sent by Tests Frameworks and routes them to a special script running within the application on the targeted device. The mentioned scripts are injected during the test session to perform actions and then return the results back to the Taf-Middleware.
Where is my screenshot
Screenshots are one of the most important test artifacts. It always helps the tester to understand, describe and report the error more accurately.
The WebDriver framework supports screenshots, but to take advantage of it on a Living Room device, sometimes we need an additional solution to bridge the gap. We can distinguish three main solutions:
1) Using the device/platform API to grab a screenshot (e.g. Android)
2) Using the HDMI out to capture recorded screenshot
3) With camera in-front of the screen
Device’s APIs are almost always the preferred solution but not all partners expose it. If we are lucky and there is API exposed with screenshot command, we can hit the DRM wall.
The camera in-front of the screen does not seem to be an ideal solution due to the quality of the image captured. Depending on the TV dimensions, the camera must be placed at a certain distance in order to capture the entire screen. Camera distance and its placement create a challenge by itself however, it is a universal solution that will work in any case. Unfortunately, the HDMI out screen grabbing that gives us a pixel-perfect screenshot is not supported by every device. Additionally, with the HDMI out we have to account for DRM-covered material so you could say that each solution can have its drawbacks.
How we are doing it?
Our approach is to have a general solution that will work on the majority of living room devices including Roku, Samsung Tizen, LG WebOS, FireTV and many others. We are always working on incorporating digital screen capture so that we can deliver visual regression testing.
Reporting
We used to use TestRail as one of our testing reporters, but since Allure reporter was smaller and local-oriented many teams decided that it would be better to create reports for their needs. Another tool we use is TeReBu which provides 2 main functions:
- It uploads the tests reports
- It also allows downloading historical trends for each testing group.
So having both: Allure and TeReBu we always can create the historical report from previous builds and analyze the changes.
Tagging
We cannot imagine not having a solution that does not support personalization of the automated tests. Our solution is supporting tags that define test categorisations: test type, region, target, feature, custom tags. For instance, you can easily define on which region and device type you want to run your automated tests.
A room full of magic
Let me come back to the topic of remote work for a moment. Most of the employees who have switched to remote work use a computer or laptop as the only tool needed to do their work. If we think about companies who support applications on TVs we can imagine it is impossible to provide necessary devices to all of their employees. We have our space limitations at home plus managing 20+ devices including Smart TVs is rather impossible. So how would we allow our employees access to devices to work efficiently? To answer this question we created TVLab.
TVLab is a room in DAZN Katowice Office with devices ready for test automation and manual usage by interacting with a Virtual Remote.
Like being in charge of the remote control
To better understand the programming process, it’s important to understand exactly how a common TV remote communicates with the device.
Whenever you push a button on your remote, it creates a signal beginning in remote’s circuit board and addressed for the receiver device. Basically, it’s sending a command code that tells your TV what to do. This is usually an infrared signal, so you need to be within 30 meters of the other device and within the line of sight. We refer to the remote as the “Transmitter” and the TV the “Receiver”.
How can you know what code is behind which pushed button? We can use a device called “infrared transceiver” to get the keycodes from the remote. But this is not the only way for device to receive commands.
There are some solutions in order to create the “Driver” for Living Room devices:
- Direct connection TCP/IP
- HDMI-CEC commands
- Infrared control
- Bluetooth control
How do we do it?
- Our approach was always to have general solution that will work on the majority of living room devices and focus on the individual implementation later.
- Direct connection over IP is very often stable but we sometimes have some paring functionality problems.
- HDMI-CEC has proven to be unevenly supported, where for example some of the TV models are supported fully, some are only capable of driving other devices.
- Infrared connection has always been a skeleton key for driving the solution for difficult platforms. IR solution works on most platforms but have one major weakness. IR signal can be received by other devices. Having 3 devices of the same type close to each other can make things complicated. There are some ideas how to sort this problem out, for example separate each of emitters to direct the signal to one device only.
- We have chosen to upgrade our solution to connect with the device via capture-card which gives us much more possibilities.
- We are also using HID interface to simulate user inputs.
Let’s collaborate
People from internal teams can interact not only with our solution by automated tests but also by playing with the app using the Virtual Remote. “What is the Virtual Remote?“ — you may ask.
Virtual_Remote is a small computer implementing real time communication, video recording, stream recording and TV/STB remote control. In order to achieve this functionality we have created patches for the system kernel and also created several system hoops that allow us to route the information stream easily. In fact, it’s a very simple solution, but fine tunned and very optimized. This setup allows us to send and receive signals, essentially controlling the device.
We have implemented an interactive learning mode in order to teach our solution any new device that has to be supported and linked that with IR emitters, API connectors and HID simulators.
Virtual Remote allows any authorized users to connect to a specified device by using portable computer as a proxy. On the new connection, we initialize the remote control features, offering the commands supported by the device and share the real time video feed.
So in general, we can teach the remote control actions or use existing ones that were already created by the community. We can control the TV remotely with help of the VIDEO feed shared in real time, that can also be recorded and saved.
Summary
You had a very long day at work. There’s a football match and you invited a couple of friends over to watch it on DAZN and order some pizza. When you turn on your TV you get a flashback of all the complications of the remotes, the testing, and the difficulties to work with those devices but you’re very tired and you just want to watch a game. Don’t worry. It’s ok. We’ve got you covered.
