THE REAL STAKES OF LIVE TV RECOGNITION (1/2)

Paul Chaumont
Mar 5 · 6 min read

Reminiz recently launched its Live TV recognition service, making it possible for the first time ever in the world to recognize all public figures appearing on your screen in a snap. This represents a major step in real-time metadata processing, giving new interaction levers on TV programs, channels and content producers.
As you have read in our previous article, TV recognition raises many challenges and so is real-time recognition. Our new service had to provide secured, on-time and highly accurate results as there is no way back possible with real-time recognition services. Here is an overview of the main reasons why.

We found you, Charlize !

THE LIVE TV RECOGNITION DILEMMA

First things first, when speaking about Live TV Recognition, what do we refer to? All three words have their own importance here:

· Live means real-time with no delay or, as we’ll see, very few, with regards to offline processing and any on-demand service. On the one hand, on-demand content has a beginning and an end that can be apprehended before starting processing and used to treat the whole content through parallelization. On the other hand, live stream implies processing without knowing what comes next. You’re in for surprises… As a result, the algorithm is more performing at the end of a live content compared to beginning, as it has way more information on the nature of content.

· Television also known as Linear Stream as opposed to any other medium broadcasting live streams. For Reminiz, live recognition service is not limited to television, but the TV medium represents its biggest challenge: capturing streams is harder, channels are infinite, programs are sometimes unknown.

· Recognition as linear and continuous detection and identification service of different in-image features.

In a nutshell, Live TV recognition is the ability to provide an instantaneous answer for all features appearing on your television screen with very limited delay and no safety net.

Indeed, the direct consequence of giving a real-time answer is that what is done is done and you will have no way to change it afterwards. Moreover, you will not use what comes next to validate any recognition: no retro-propagation of the information to put it in other words.


LIVE DOES NOT WAIT

Delay when it comes to live television is an every second challenge. The main reason is that once the feature is not on screen anymore, recognition is already out of date (or almost). Timing for live recognition includes multiple variables that should be considered when setting up such a service:

· Stream Latency: in other words, how long does it take to receive the stream information and are there any differences between channels and devices? Have you ever experienced your neighbor shouting after a scored goal, whereas you did not even get to watch the goal yet? Our tests have shown that on all 28 main French TV channels, the stream delay could vary between DTT and set-top boxes for a same channel, and this variation could be more or less important depending on the channel.

This variation is estimated to -/+ 3 seconds.

· Processing Latency: this is the time it takes for Reminiz to process and return an accurate and verified information to the client. It takes no more than 3 seconds for Reminiz to send an answer back. It could go faster of course but speed is linked to performance: the quicker, the less performing the algorithm. We have found the perfect trade-off between performance and speed.

This variation is then estimated to 3 seconds.

· User Latency: this is the time it takes for a viewer to use our service on their television, the delay between human stimulus and remote controller utilization. Based on tests made with a panel, the user latency revolves around 10 seconds: human eye recognition (2 seconds), device search (1 second), use of device (6 seconds)

This variation is then estimated to 9 seconds.

In the light of these estimates, one can ascertain that no more than 6 seconds (and usually 3) are necessary to send back a qualified result to the user, while he will in average take 9 seconds to make a request, leaving a 3 seconds buffer.

Illustration of delays between User and Reminiz point of views. We see that worst case, Reminiz has a margin of ~3 seconds after processing is done to answer the requested celebrity.

REAL TIME TV RECOGNITION FOR LIVE CELEBRITY FACES

One of the use case made possible to our clients thanks to our live TV recognition data is the face identification of any public figure appearing on-screen. Actors, Politics, Writers, Singers, Athletes and so on, our tech can identify them all in a snap. For the viewer, it only requires using the television remote and ask, whenever necessary, who’s appearing on screen in real time.

Live TV recognition poses the great challenge of the unknown. As it implies processing streams without stating upfront who will be appearing in it, the wide variety of programs and celebrities makes it harder to give an accurate and verified answer.

Digging deeper into an average TV program, one can quickly do the math. An average week on the 10 main French TV channels represents around 2540 programs, from under 10 minutes to above 90 minutes long. Each program has a different level of complexity, depending on the following factors:

· Number of celebrities appearing: the wider the research scope of celebrities to look-in for the algorithm,the less certain the results. Not only is it more complex to detect and recognize all of them, but it also implies having all the said celebrities within the database. It seems rather easy if you’re monitoring an episode of Friends. A bit less if you’re asked to recognize all actors from an unknown show rebroadcast from 1975. Not so easy either for shows such as Game of Thrones, with more than 600 actors credited to the cast.

Game of Cast

· Ability to anticipate the cast: our technology must look further than the regular electronic program grid. It must be able to anticipate casts, enrich it with the most complete and accurate data for pre-casted program. It must also be able to define who is trending and more inclined to be on-air today and make sure this person is already in our database. News programs or live events do not give any hint on who could appear and offer a true uncertainty when it comes to processing the streams. Thus, they are the trickiest. News TV channels require a really strong algorithm and a wide database, making it able to switch from last night football game’s result to the latest presidential elections in Ukraine.

· Detection context: the third criteria applied when assessing the complexity of a program is the context of appearance. As we already explained it in a previous article, good recognition implies seamless detection. A face is not always so easy to detect: from an anchor sitting behind a desk to a war scene in a movie with 1000 extras, the difference is big. One should not under estimate how unknown people can make it harder for the algorithm to recognize the “real” celebrities. The audience in the background can be tricky as well, as it can be blurry, in dark spots or simply in large numbers.

Taking these three levels of complexity, a football game is a good example of tricky live TV facial recognition: a live event with no heads-up on the outcome, lots of celebrities on the field and a tough context of detection in a crowded stadium.

But the harder, the greater the reward. Indeed, Live TV recognition opens amazing new ways to interact with the audience, new monetization services and a brand-new way to rethink television as a competitive medium…

Reminiz Insights

Reminiz is a world pioneer video understanding technology offering real-time facial and logo recognition. Augmented Content for a never-seen viewer experience.

Paul Chaumont

Written by

Product Manager at Reminiz

Reminiz Insights

Reminiz is a world pioneer video understanding technology offering real-time facial and logo recognition. Augmented Content for a never-seen viewer experience.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade