Pointer navigation on TV

How we at Zattoo enhanced our existing OTT application to use LG’s Magic Remote

Everyone is used to using a remote control for their TV. When it first appeared it changed the way we surf through channels. People didn’t have to stand up and come over to the TV to change channels or to turn down the volume. Everything could be managed laying down on the couch a couple of meters from the TV. The first wireless remote controls used ultrasound, but this method had a problem that noises at the same frequency at which a TV receiver was tuned could trigger the actions. Another issue was a limited amount of actions. Those problems were gone when the infrared remote control was introduced to the world but it required you to point it to a receiver. The next breaking change was Bluetooth technology, which made channel switching possible even from another room. Decade by decade the TV was becoming smarter and could provide more content. Even with a good connection for the remote, the TV owner still has trouble navigating through modern interfaces with a lot of content.

The example above shows that a user needs to press a key on a remote 6 times to come from the bottom-right corner to the Back button.

What about some magic to cut intermediate steps out? LG has something for it. They have come up with the Magic Remote.

Way faster!

The article below describes how we at Zattoo enhanced our existing OTT application to use this “magic” device.

Intro

The technical stack for a web hosted OTT application is easy to predict. It’s JavaScript and some popular libraries and frameworks on top, such as React, Redux and Webpack. LG TVs uses the WebOS operating system which requires some specific html and metadata files to give the web app running on WebOS an access to the hardware and features that are available to native apps.

An LG TV requires the Developer Mode app to be installed to allow testing. This gives an opportunity to see how the application will be running on a real TV and how we can interact with it using the Magic Remote. However, the application needs to be uploaded to the TV each time we want to test it, which complicates the debug process.

To improve developer workflow we came up with a pointer simulation in the browser. At first glance, it appears that the pointer is always present in the browser and we don’t need any extra logic to make it appear. But we need to simulate WebOS’s pointer activation to have the same experience as happens on TV. To achieve this we trigger the cursorStateChange event in the browser, which gives us a similar behaviour of the pointer and makes the debugging process smoother.

Detect when the pointer is active

To detect when the pointer is on/off we use a custom event cursorStateChange, which is available in the WebOS browser. For more information please read Handling cursorStateChange event.

A Higher Order Component is used to add the logic that tracks when pointer mode is active or inactive. This component is a simple functional component that defines event handlers for the cursorStateChange event and connects it to the Redux store. When the Zattoo app is built for the WebOS environment the root component is wrapped by a HOC which introduces pointer navigation to the app. Later each component that needs to know if a pointer is active can get it from the Redux store.

Here’s how it works:

If a certain page wants to know when to show specific navigation elements, adapt the UI for the pointer, etc., it gets this information from the store using a selector and a React component connected to a Redux store:

That solves the challenge of how to detect when the pointer mode is activated within an app. Although it does not solve how to detect if the app was opened with the pointer mode already active. In this case even if the pointer is present on the screen, our app does not know yet if the app should be in pointer navigation mode. Let’s improve the previous code a bit.

When the app mounts it starts listening for two events:

• load for window
• visibilitychange for the document.

The former detects an initial load and the latter detects potential visibility changes while the app is running. For both cases, it provides a callback to track if the app becomes visible. On unmount it removes these callbacks:

The callback which is called when one of the events above is fired adds a new event listener for the pointermove event:

We check document.visibilityState === ‘visible' to make sure that the app is not overlapped by WebOS ui elements: e.g. settings or home menu, modal windows, etc., as seen in the example below:

Visually the app controls are still present on the screen, but the setting menu that opened up on the left hand side is overlapping and nothing in the app can be accessed while the WebOS UI element is open.

As the last step we need to add a callback to handle the pointermove event and activate pointer mode in the app. The need for this logic is required in order to cover a case when the app is started with the pointer mode already active. In this case, if pointer mode is active in WebOS and a cursor is moving over the screen, it adjusts the real state of pointer mode in WebOS with our state in Redux store:

If the pointer moves and pointer mode was not currently turned on in the app (visibility === false), we activate it. If it was already active, we just ignore this move. Finally we remove the listener for pointermove as now we will be tracking the state when the remote control has been shaken or the wheel is rotating.

Specific UI in Pointer mode

The application for WebTV uses CSS Modules to scope all component related css classes in one file. Nevertheless, when pointer mode is active the css class .pointer is added to <body> which can be accessed by all modules using the :global(.pointer) modifier. This makes it possible to disable some 5-way navigation specific css classes, set them to their defaults, and add the :hover pseudo-class for elements which can now be accessed with a pointer device:

When Pointer mode becomes inactive, the .pointer class is removed from <body>.

Switch between 5-way/Pointer modes

In 5-way navigation mode only one element across an entire page has focus, which is visually highlighted and triggers an action when Enter (OK on the remote control) is pressed. Pressing an arrow button on a remote control causes focus to navigate to one of 4 directions. Thus an application always knows not only what element has a visual focus, but also a specific focusIndex value which is bound to data, represented by a selected element.

Example:

Let’s say we have a menu list with categories. In 5-way navigation mode we start at the element with the index 1 and have to press down button on a remote control 3 times to reach the item 4:

In Pointer mode this concept of one element which is always focused — does not exist.

Let’s look at another example:

None of the area is focused in the screenshot. In this case if a pointer is over the Header, there are no focused elements that might trigger an action on Enter (OK button on the remote control) press.

Going back to the previous example with the menu list, the pointer doesn’t have a strict rule of its track. We can put it over the first item in the list, and then move it aside to the 4th one:

So far it’s not a big deal since a visual effect is set by the css :hover property and a respective focus index can be set at the moment when the mousedown event is triggered. But what if we want to provide a consistency between 5-way and Pointer modes in terms of which element has focus?

Example:

In our case pressing the up arrow on the remote control will dismiss pointer mode raising the question — what element will be focused next in 5-way mode? Since we are talking about consistency, we expect that the next element above would be selected next; that is, if in pointer mode the focus was on item 4, then moving up with the arrow should put it on item 3. To reach that point and avoid extra updates within a component tree each time a pointer is hovering over the next element, the item’s index update should only happen when the pointer is dismissed or when it leaves a respective area. Here’s how it works technically:

• props.highlightedIndex - index of a focused element which is used across different components. Initially designed for 5-way navigation, where technically “focused element” == “selected element”
• state.teaserIndex - index of focused element which is used within the list of items

Each time highlightedIndex is changed outside of the list, it is aligned with teaserIndex which is scoped by our teasers list. Then later when the pointer is moving through teasers, only state.teaserIndex is changed. It doesn't affect other components above in the tree:

This allows us to start 5-way navigation from the last point where the pointer was:

Pointer specific files

Pointer navigation for the Zattoo WebTV app is supported only for LG. In order to optimise bundle size and to exclude this specific logic from a base build two approaches were taken.

For simple cases we use code elimination with a feature flag $INJ.__POINTER_NAVIGATION__:

When the logic is more complex, we introduce class inheritance and import different implementations from files with an extra extension. Thus the base build without this extra functionality has a reduced size and also does not consume memory for methods, listeners or variables which will be never used.

Webpack has the resolve rule, which allows us to define how modules are being resolved. The property resolve.extensions defines all possible file extensions and their order. For example, this configuration allows importing files only with these extensions to a module:

It gives an opportunity to load specific versions of our files based on some features. Each feature name becomes a part of the extension preceding the main extension .js or .jsx. For instance, specific logic that allows pointer support would be exported from modules with the following extension: .pointer.js(x)?.

For a specific platform, in our case WebOS, where we want to add a pointer support, we have to modify the array with allowed extensions. The Webpack configuration for it would become:

While Webpack is resolving appropriate extensions for import, it will search for files with an extension earlier in the list first; if .pointer.js(x) is not found under the folder, it will load .js(x) files. This strategy excludes the feature-specific code from a base build. Feature specific files inherit from base ones. The shared base class doesn’t have feature-specific code and will be loaded either by a feature-specific implementation or directly if no features are requested.

Example:

We have PlayerToolbar component which implements a class which extends React.Component.

This file contains the base logic which would work on all platforms and it exports a component from ../components/playertoolbar/index.jsx.

../components/playertoolbar/index.pointer.jsx implements a class which extends PlayerToolbar from ../components/playertoolbar/index.jsx and adds on top pointer support like event listeners or class methods required for pointer support:

Since index.pointer.jsx is resolved only if the pointer feature is requested we don’t need to add verification to see if this extra code should be added.

When PlayerToolbar is imported to another module no extra logic needed:

We import PlayerToolbar and Webpack resolves which of theindex files under the folder has a priority. If we run a build for WebOS, Webpack uses index.pointer.js(x) file to import from and the class from this file extends a base version of PlayerToolbar. All dependencies have been resolved at build time and we do not need to manage them at runtime.

Conclusion

In this article I covered the main topics which characterize a specific functionality of a smart TV application with pointer support and how Zattoo Web-TV team solved them. The key points are:

• The absence of a constant focus on an element, which acts as a starting point for an action on 5-way mode navigation.
• Specific UI elements (e.g. arrow buttons) which appear when the respective area may provide feedback for wheel-scroll and contains more offscreen content.
• Pointer support logic is delivered as an addition to the main bundle. Which is achieved using file extensions and Webpack configuration.

Thanks for reading this article! Sign up for the Zattoo Tech Blog, to get the latest updates about developing, engineering and designing the future of television.

Thanks to pverkhovskyi, Vivien Luetticke and Tom Bridgwater for supporting in writing this article.