And you thought browser support was annoying…
I’m a frontend developer who makes streaming apps, and since I began working in this field I have been stunned by the complete lack of information on the very topic of TV apps, so in this article I would like to explain some of the more under-the-hood problems faced on the frontend side of streaming. Something which doesn’t really affect most web developers living in a fast-device age.
TV apps are something, that for a lot of us, are used every day. Whether you watch your favourite show, rent movies or just browse, you probably have interacted with them at some point. When you look at the business of TV apps it seems to be focused on the content provider, and everything is looked at from a non-technical standpoint, the content dictates everything. What I want to do in this article is to explore the technical side of TV apps and take a bit through know what technical problems we, the frontend team at Norigin Media, deal with on a daily basis and how we try to overcome them.
What’s familiar?
The commonalities between standard web/native applications and streaming applications are generally pretty high-level. Most modern applications in the browser and on TV platforms are taking advantage of web views native to that particular platform, so this means that using your most loved JS framework is as easy as it is on the web to get going, with some minor native shell set-up on the device at hand. Routing and everything web then becomes familiar and you can get something looking pretty ok pretty quickly. Below I will explain some of the key differences and core problems that we face when architecting and building streaming apps.
Navigation
This one is actually pretty fun. On a browser or on a mobile device, this is pretty straight forward (click and tap), but take this to a smart TV and suddenly you have four arrow buttons and you’re writing listeners for these and mapping focus to the next desirable component manually or you’re performing expensive operations on poorly performing devices to calculate the best place to move to next, leading to some really desperate and unreadable code, nevermind the performance overhead.
How do we tackle this: In most cases in the past a custom approach was valued over a generic one due to the lack of apps available on devices only supporting spatial navigation. In the past few years, however, smart TVs (and watching content from a games console, for example) have become commonplace in our lives and people demand the same usability they find in their browser or mobile device from their TV. This had led us to develop a generic method for dealing with spatial navigation, which ultimately abstracts this difficult logic from the app and allows the developer to focus on what matters, all with some nifty matrix algebra! The tradeoff of doing this is that some apps may require extremely custom navigation which can only be dealt with in a custom manner, so all use cases aren’t covered.
Getting logs…
If you’re working on a smart TV and you’re developing the normal way you do on the web, you’ll begin to notice something that is missing — logs. On a lot of devices, especially pre-2013, getting logs is a major pain and a huge time-sink. Generally speaking, you don’t get errors, your normal logs will just stop and you have to figure out where stuff is going wrong. You will need to tediously go through the timely procedure of opening the app and closing it multiple times if your app freezes or the TV gets a bit tired and laggy.
How do we tackle this: Oftentimes it’s patience, and just getting through development with some nifty try-catch logic. We also try using a remote debugging tool as the obvious starting place but these aren’t always super reliable (see JSConsole and Weinre). Some people in the office have written their own remote debugger, posting the stringified log to a local server with logic to handle some of the commonly known printing syntax for objects and arrays for example. Another alternative is to add on-screen logging, and this can be particularly useful when you’re working with player logic.
Streaming formats and different players
One thing that is really important to note with streaming and players in general is that there are a lot of formats of content (DASH, HLS, MP4… etc). To cater to all of these different formats, lots of different companies and communities have built different players with different capabilities and different support spectrums. In short, there are quite a few variables at play.
Picking the right player for your scenario is key… each browser has its own quirks and Apple really hates supporting anything but its own technology. Once you deviate from the browser things can become a lot more complicated. Generally speaking, iOS and Android can be pretty straightforward to get things playing on with some really neat players available (VOP for example). The problem lies in supporting many different types of devices, many TVs have their own native player with extremely poor documentation and a lot of bugs. A lot of the time you’re pushing the boundaries of the players on devices and having to call upon very small communities and manufacturer forums to get help, oftentimes in vain.
How do we tackle this: Abstraction, abstraction, abstraction. We like to take the generic approach with the things we write, and we love reusing our solutions over and over again. There is no better feeling than tackling a project knowing that your code is battle tested.
By providing a standard API to the application-specific code which encompases player scheduling and states, and then adding adaptations of the specific players (adapters) on a lower level in the architectures hierarchies, you abstract away the headaches. This means that the app logic doesn’t ever have to care about what device it is running on, it just asks for an operation to be called and our solution (which we call Player Manager) takes care of this in a simple API.
Advertisements
Advertising has become an intrinsic part of streaming. Streaming is afterall a business and advertising is one of the driving forces of it. This also means that developers have had to come up ways and standards which are designed to deal with advertising.
As this article is frontend-orientated, I will discuss ad-stitching in the context of client side ad stitching or CSAS. CSAS is basically where the client is in charge of inserting (stitching) the advertisement (the ad clip) into the playback and presenting it to the user. There are two main ways that this is tackled:
- When content begins and an ad is set to play, two players are instantiated on the page. One player is in charge of playing the ads, the other is in charge of playing the content. Oftentimes when an ad appears, the other player simply lays over the paused content in the background and plays the ad. The drawback of this approach is that anyone who knows how to use an inspector can easily destroy the player playing the ads and return to content. An obvious advantage of this method is the almost seamless transition between the different contents being played.
- Sometimes it isn’t possible to have two players concurrently playing on the page due to a native player API constraint on the device at hand. In this case, a fresh streaming URL will need to be loaded into the existing player and in some sense, you get a more true-to-the-word stitching experience. The main pro of this is that dealing with one player in your code is a lot neater as you’re not trying to juggle two player instances. The issue with this is that the desirable bitrate for the content may not be known at the beginning of each playback and can result in poor streaming quality for some time. There is also the possibility of some minor to significant buffering time between streams, impacting on user experience.
How do we tackle this: It is difficult to outline how we tackle this in the case of advertisement as there are a lot of variables at play, most notably, the device in question but also the infrastructure on the backend side of the application. But generally, the approach is to look at the pros and cons of both approaches in the context of that given application.
Tracking
This is a weird one, because pretty much every app has tracking, and this isn’t specific to streaming apps. I wanted to touch on this however as I have spoken a lot about abstraction. Sometimes abstractions can lead to a loss of context and in the case of tracking, this can be very dangerous and can lead to code with dispersed tracking calls, making for a readability-nightmare
How do we tackle this: In the case of tracking, I am going straight to how we deal with it as it gives an insight into a real benefit of working with streaming apps — players are stateful things. The player state is really the core of a solid tracking implementation, and being able to add some useful context to that player state also makes tracking a breeze.
A good example of this is that some tracking software may want to track differently for advertisements and the desired content. The player in this case, however, only knows about what it is currently doing and not about the context of the operations. By adding this context in a generic and reusable way, your tracking is reduced to a data driven event map (mmmm… nothing better). In our generic implementation the application code itself never has to know about any tracking code, leading to neater and more concise application code.
TV Versions
TV manufacturers are constantly trying to keep up with the latest trends and this also means that the industry is moving quickly. In response, developers need to move even quicker. Most large manufacturers have their own purpose built OS for their TV which means that different browsers are supported and things become messy very quickly for us frontend developers.
A good example of this is Samsung Tizen TV. Tizen is the upgraded OS from the Orsay (now legacy) Smart TV Operating System. Samsung made the switch in 2014 and the first Tizen TV came out in 2015. Each year, however, a new version of the OS has come out on the new models and this has meant some pretty annoying optimisations. Every year there are new issues, questions to Samsung and the plague of optimisation for an ever expanding list of devices — on just one platform.
Conclusion
Overall, streaming app development is an extremely diverse and broad scope of challenges and there are of course many more than I have outlined here. At Norigin Media, our goal is to take care of the complexities of streaming development in generic and reusable ways in order to build solid battle tested solutions.
I hope to delve into the more low-level aspects of each of these points in later articles and share some of my opinions on approaches taken by others as well as exploring some of the latest technology in the streaming technology world.
We want to hear from you!
We would love it if some of you guys could get back to us with issues and problems related to streaming and TV app development! We want to start building a community around streaming and enable other developers to build cool things in this space. If anyone has anything they think it would be helpful for us to share, then we will consider making that solution open source and involving the community. If there is anything in this article that you want to find out more about, please don’t hesitate to contact me at eoin.falconer@noriginmedia.com.
