How we took VG Live & Målservice’s latency from seconds to milliseconds by changing our API approach
In live sports, milliseconds matter. Delivering the latest news or goal updates milliseconds faster than the competition is a huge competitive advantage.
By Michał Drobniak (tech lead) & Tord Overå (product manager)
Our live sport platform, VG live, is Norway’s biggest sport web-site, with up to 40m weekly page views, and at peak, over 600k unique users use our platform daily, to follow football, handball, & ice hockey. In addition, we provide the same platform in Sweden branded as Målservice, generating over 10 million weekly page views. These sites & the underlying data platform are built end2end by us, Schibsted’s Sport Platforms team.
In other words, if we are able to improve our sports feeds latency by 2 seconds on average, we are reducing our users’ experienced latency by over 6 years per week (100 million seconds)
Previous versions of vglive.no were usually based on a polling approach for match updates — asking the API every eg. 10 seconds if a match score has changed. For many situations this is fine, but when PSG plays Barca, we get three problems:
- User requests can put a lot of pressure on the API, sometimes causing outdated data
- Because of the interest for such a game, the 10 second delay is 10–100x more impactful
- This delay can vary across users; imagine sitting next to someone on the bus, and them knowing about Messi’s goal against his old club 10 seconds before you. Not a great feeling
As the root cause of the delay was the way we were pulling the updates, we decided to explore how to push data.
Introducing Push APIs
Our first step was to utilize Server Sent Events which works as a “push from server”. It’s a one-way (from server to client) communication protocol, based on HTTP with standardized browser API, using the SSEHub software (kudos to Ole Fredrik Skudsvik). It served us well for a couple of years but we wanted to migrate to something with richer features and two-way communication so we can benefit from a consistent publish/subscribe interface.
Enabling two-way API
We started experimenting with the MQTT protocol which is a lightweight publish/subscribe messaging transport, very popular for IoT or mobile devices. We used EMQX software as our MQTT messages broker, and it served us quite well for some time. However, we did not find a good way to scale it up and down while maintaining all the MQTT features. It also has a relatively heavy browser library. But we loved the protocol as such, especially the topics concept (great for structurization) or message delivery levels (not all updates must be delivered exactly at once).
Getting to 15 ms latency
Recently we decided to use the WebSockets protocol with Eventhub as the message broker. It takes some concepts from MQTT while allowing usage of the browser’s WebSocket API. Currently we use it to deliver live updates to tens of thousands of clients at the same time with an average latency of 15 ms, with thousands of messages published per minute. Of course, this requires a lot of CPU power (cores), but in the world of sports these seconds are the difference between the best, and the second best services.
Join our team!
If you too would like to combine tech and sports on a daily basis, our team is currently hiring in both Oslo & Kraków.
Reach out to jack.stenberg@schibsted.com if you have any questions, or apply directly here (Oslo, Norway) or here (Kraków, Poland).
