RCA for Live Stream Playback Issue on June 20th, 2017


This RCA addresses the live stream playback issue we discovered on June 20th, 2017 where SYNQ’s video player could not playback a live stream. This only affected playback of live streams with SYNQ’s player. Live streams played through the CDN url and regular video playback were unaffected.

Event Description:

On June 20th, 2017 at 14:17 UTC, while doing our manual post deployment testing, we noticed that we could not play live streams with the new player we had just deployed. Development was immediately notified and alerted of the issue.

At 14:27 UTC, the development concluded their investigation and determined that in order to fix the live stream issue happening in the new player they would need to do a hot fix. Operations updated our status page to let clients know the details of the issue and let them know we would have hot fix ready within a few days. We also notified clients that live streaming directly from the CDN URLs we provide were still working as expected.

On June 21st at 21:47 UTC , operations updated our status page letting customers know the issue was resolved, after development deployed the hot fix and operations did some thorough testing to make sure everything was working as expected including live streaming from the new player.

Root Cause and Remediation:

A new player was introduced and it was not configured properly to playback live streams. This made it so that when one tried to use the new player for watching a live stream, it did not function at all and only showed a blank image. This happened because the live stream CDN url was not being loaded properly into the new player as it was only expecting regular VOD content.

Future Preventive Measures:

  • When introducing something like a new player have more thorough testing around functionality that should continue working