An approach to implementing HbbTV on Android TV™
Introduction
We at Zattoo, in cooperation with Fraunhofer FOKUS, wanted to explore the viability of implementing HbbTV within an Android application.
Technically speaking, HbbTv is an additional layer rendered on top of the live stream. It displays web-based content and consumes varying sets of key events, depending on its current state. Lastly, it communicates with the Android layer using Javascript interfaces, to request data (e.g. the HbbTv app URL for the current channel) and report user actions (e.g. starting nested playback). From the user perspective, it adds interactive content to the stream, enhancing the experience and even allowing for some multitasking, like browsing the guide or playing a game with the stream still present. It may also provide content of its own, from recorded to live.
While all this may sound simple enough from the general Android perspective (“just put in a WebView!”), the complexity lays in the details and unique use cases of Android TV. So, without further ado, let us go through the challenges that we faced and the solutions we devised for them.
Zattoo’s Android TV Set-Top Box
Zattoo, in partnership with Skyworth Digital, have begun offering its TV Platform customers an Android TV Operator Tier certified 4K/UHD Set Top Box product. Based on the AmLogic S905X2 chipset, the device offers end users a high performing fully branded TV experience powered by Zattoo’s customer launcher application. As Zattoo’s newest operator grade first screen device, it was important to ensure that those customers transitioning from more traditional TV devices could still enjoy content made available via HbbTV.
Evaluation of HbbTV Apps / HbbTV Features
To prioritize the features for the HbbTV WebView on Android TV, we evaluated the Top 10 HbbTV applications in Germany and documented the most popular features. As we were using devices offering support for HbbTV specification version 1.5, we used this as the reference for the implementation of a minimal viable product (MVP).
Below are a list of the most common HbbTV features used by applications today:
- Mediatheks / VoD Portals
- Alternative Live Streams (e.g. for sport events like Olympics with multiple live streams from different locations)
- 360° Video
- News, Games, Weather, Dedicated HbbTV Apps for popularTV shows/programs
HbbTV applications are Web-based, therefore they can be rendered in a standard WebView. We decided to focus on HbbTV-specific APIs, namely embedded objects, VoD, live playback, key events and application object as these are all broadly supported in this standard WebView.
A popular HbbTV feature, video/broadcast embedded object, is used to display the broadcast stream in Picture-in-Picture (PiP) mode within the HbbTV application. Similar in popularity is the VoD Playback feature, which is offered on almost all HbbTV applications as part of their “Mediatheken” offering. Typically, VoD’s streaming format is MP4 progressive download, but we decided to implement a HTML5 video solution which supports MP4 progressive download out-of-the-box.
At the time of evaluation, we were unable to find HbbTV apps offering alternative live streams which are usually enabled during special events such as sport events like the Olympic Games. As for KeyEvents with HbbTV, the applications are navigated using a remote control as opposed to a traditional web setup. For this purpose, we had to map KeyEvents as received by the set-top box in which the HbbTV app is running to HbbTV KeyEvents. In particular, the color keys of the remote control (red, green, yellow and blue) are crucial for navigating in HbbTV apps. For example, the red button is used on most channels to show or hide the HbbTV App. For this reason HbbTV Apps are also called Red Button Applications.
Lastly, HbbTV offers API’s to manage the lifecycle of the HbbTV application. These API’s, such as those to start or stop an application, are the glue code between the Zattoo application and the HbbTV WebView.
HbbTV Library for Android TV
When initially planning the HbbTV Library, we had to keep in mind the unique structure of Zattoo’s native code base. With 30 variants of the app spread across mobile, TV and TV launcher compatible devices, we needed to ensure that the HbbTV solution could be plugged in cleanly and also remain decoupled. The ability to reliably toggle the feature on and off between different variants of the Zattoo Android TV application was just as important. We determined that an Android library would be the best way to achieve this.
After the evaluation of the top 10 HbbTV Apps referred to earlier, we defined the main components of the HbbTV Library and specified the interfaces to be used for the communication between them as stated in the following diagram.
The main component of the library is the HbbTVWebView that extends the default Android WebView and implements additional functionalities and JavaScript APIs required by HbbTV. All these APIs are summarized in a single JavaScript file (HbbTV Polyfill) that is loaded into the WebView before launching the HbbTV App. This ensures that the App can use any of the supported HbbTV APIs after launch. We differentiate between 2 categories of HbbTV APIs:
- HbbTV APIs that can be implemented in pure JavaScript (Example: VoD playback of mp4 videos via the HTML5 Video Element)
- HbbTV APIs that need to interact with the Android Platform (Example: video/broadcast implementation needs to interface with the underlying live streaming player e.g. ExoPlayer).
Another main component of the HbbTV Library is the HbbTVLifecycleManager that monitors and controls the lifecycle of running HbbTV applications. It acts as a bridge between the HbbTVWebView and the underlying Player (e.g. ExoPlayer). This way, the HbbTV Library can be used with any player since there is no direct binding between the HbbTVWebView and the Player. The underlying application needs to implement the PlayerInterface (in the diagram above, the ExoPlayerInterface is a concrete implementation of this interface), which defines all functions required to communicate with the underlying player in both directions.
Examples:
HbbTVWebView → Player Communication: The HbbTV App requests to stop the live stream in the underlying player by calling stop() on the video/broadcast object. The stop request is propagated towards the ExoPlayerInterface which stops the playback in the underlying ExoPlayer.
Player → HbbTVWebView Communication: The playback state change event of the ExoPlayer gets propagated through the ExoPlayerInterface to the HbbTVWebView and the corresponding HbbTV event is triggered on the corresponding video/broadcast object.
Android TV challenges
Fitting the HbbTV layer into our application was another interesting challenge. Android TV can be quite capricious when it comes to navigation and focus handling, so it was important to stick to the platform logic. The goal was to ensure a high quality user experience by first ensuring the HbbTV app can be rendered correctly on top of the broadcast stream and ensuring that the UI interactions were as inexpensive as possible to avoid video interruptions or application crashes.
To deliver upon this goal, the first approach was to put the HbbTVWebView onto our Fragment stack. This would make it a first-class citizen of the application content stack, with full control over lifecycle and clean code encapsulation. Although initial prototypes were quite promising and very easy to implement, two issues became apparent:
- Fragment reordering — It turned out to be quite heavy on the system, causing stream stuttering. This was not acceptable as the HbbTV stream moving/resizing logic is an integral part of the implementation and needed to work flawlessly.
- Stacking logic — Our app did not like the concept of having more than two Fragments on the screen at any given time. Initially, we had the concept of the Player and Content Fragment with another in between, however the introduction of this third fragment caused side effects, including unexpected removals or getting stuck in a specific stack state.
While the second issue could be fixed with some development effort, the Fragment interactions posed significant challenges. With that in mind, we decided to try another approach, namely a single Fragment approach.
We tried anchoring the HbbTVWebView directly into our PlayerHostFragment, on the same logical level as the player surface. It turned out to be the perfect solution resulting in the following benefits:
- Flexibility — We could now reposition the Views against each other using simple logic, with minimal performance impact.
- Stability — There was no further need to track and sync the state of the player and HbbTVWebView as they were now children of the same parent component. Additionally, if we wanted to turn HbbTV off, it was only the parent component that needed to be triggered.
- Lifecycle alignment — When the player was present, the HbbTVWebView was present too.
Navigation
Our Android TV box includes a generous 47 button remote for which the majority of the keys are linked to functionality inside the application. This makes it difficult to “reserve” a set of keys for HbbTV applications and calls for a mode-switching approach. It also creates some interesting challenges when coupled with the aforementioned stacking logic as the user can navigate directly to some parts of the application without interacting with the HbbTV component.
Android provides a robust key handling logic that makes it easy for a component down the tree to consume the events based on some condition. The condition was the current mode and we simply decided whether a key event should be passed into the HbbTVWebView for consumption or allowed. The small exceptions here are the programmable color keys, which are always passed through.
As for the app navigation problem, we decided to make the Home button of the remote a “panic button” of sorts. As we could not always rely on the HbbTV apps to exit when the Back/Red buttons were pressed, we wanted to ensure that the user always has a reliable way of exiting. The next step was to apply this logic to other keys, including Search, Guide or Apps and Games.
As the mode-toggling logic fell into place, we faced another dilemma; what logic should be used to determine the current mode? We tried a couple of different approaches:
- Counting the programmable color key presses
The idea here was simple; as the user will normally enter the app with a color button and exit it in the same way, we could just create a counter and assume that a value greater than zero indicated HbbTV interaction, while zero meant we were in the native application. This did not work as the color buttons can be used by HbbTV for pretty much any purpose, quickly resulting in desynchronisation of the counter. Even the red button could not be assumed to always provide the only exit point of the HbbTV app, with some of them using the back button or a UI element. - Detecting whether the page shown is the red button prompt
Another idea was to utilise some already-present signaling logic to determine whether we were on the initial prompt page and deeper within the HbbTV app. We realized that there is no reliable way to do this as the pages don’t contain information on whether they are the initial page or not. - Directly monitoring the on-screen situation
In the end, we decided to inspect the WebView contents in a loop to determine whether there was a prompt page, a full-screen page or no HbbTV app visible on the screen.
The concept itself is very easy to implement in Android; all we needed was the AndroidX utility method View#drawToBitmap(). This gives us a Bitmap object with the current contents of an on-screen view component (including transparency, which was crucial in our case). Once we have the contents ready for analysis, we probe a set of pixels forming an uniform net.
If at least three of the probed pixels turn out to be non-transparent, we assume the HbbTV app is displaying full-screen interactive content and we start passing additional key events to it such as the back button or the D-pad keys.
We run this probing logic every three seconds in a loop. Real-time logic would have been ideal here, however our internal tests showed more frequent checks have an impact on the general performance of the device. Instead, we settled on the conditional as we can adjust it to different devices for performance optimisations as required.
HbbTV challenges
The main component of the HbbTV library is the Android WebView which is extended to support HbbTV specific functions as described before. The big advantage of the WebView is that it supports HTML5, CSS3, JavaScript and many advanced Web APIs. However, there are still many challenges when implementing specific HbbTV features, specifically:
- Support of HbbTV Content-Type HTTP Header: One approach is to intercept HbbTV App requests and download the requested HbbTV outside of the WebView. The response is then passed to the WebView via loadDataWithBaseURL() instead of loadUrl().
- Handling of remote-control color Keys: The color keys Red/Green/Yellow/Blue are mandatory according to the HbbTV spec. The main issue is that keyEvents triggered in the HbbTV App running in the WebView when a color key is pressed receive the same keyCode of 0 while in the Android application (e.g. in Android Activity) the key codes are set correctly. We solved this issue by intercepting the KeyEvents in the Android part of the WebView and triggered custom KeyEvents with correct key codes in the HbbTV App running in the WebView.
While these first two issues were fairly easy to reconcile, we faced multiple challenges regarding media playback during implementation of the HbbTV Lib due to the many content types available for streaming. To play any kind of VoD/Live content (or to embed the broadcast video in the HbbTV App as Picture-in-Picture), HbbTV requires a HTMLObjectElement (<object>) that is used to distinguish between different content types. The key ones include Broadcast (video/broadcast) and MP4 (video/mp4).
MP4 is the most relevant and most used media format for VoD playback in HbbTV. Android WebView supports MP4 video playback out-of-the-box in the HTMLVideoElement. The challenge was to intercept all HTMLObjectElements and create an associated HTMLVideoElement for each of them so that they would function properly in the Android application.
Broadcast content type that corresponds in the Zattoo App to the channel IP livestream and MPEG-TS content type are played in an external Player (ExoPlayer) since they are not supported in the WebView. The challenge however was to link the Broadcast HTMLObjectElement in the HbbTV App to the underlying ExoPlayer and let the application control the size, position and layer of the ExoPlayer to fit the position, size and layer of the corresponding HTMLObjectElement in the HbbTV App.
One limitation of this approach is that the external player can only be layered either on top of the WebView or behind it, but not within the layers of the DOM tree of the HbbTV App. To address this limitation, we are currently looking for a deep integration of the player in the HbbTVWebView to give the application full control of the appearance of the player within the HbbTV App.
Outlook / Future Work
This first version of the HbbTV browser library for Android TV is based on the HbbTV 1.5 specification. Later versions of the HbbTV specification, namely version 2.0 onwards, brings additional features such as parental control, DASH/DRM support, companion screen, UHD / HDR and Next Generation Audio to name but a few. As more broadcasters and content providers offer support for these later HbbTV specification versions, the opportunity to take advantage of these features within the HbbTV browser library is exciting.
Additionally, as Zattoo offers its Android TV application to many other Android TV devices such as Smart TV’s, the ability to support HbbTV on Android TV applications running on these devices is eminently doable.
Conclusion
This concludes our HbbTv adventure. We hope that we have given a good overview of the possible challenges of implementing HbbTv in Android TV and a bunch of solutions we devised. This is of course just the tip of the iceberg, as further HbbTv specifications contain even more fascinating features to implement and provide to the end-user. The journey to merge both technologies will probably not end any time soon, but we hope to have proved that Android TV and HbbTV are far from exclusive or competitive. It is certainly possible to have them integrated into a single product, to provide users with an even smarter smart TV experience!
