Down the MCU road with GStreamer #WebRTC

Replacing rtpbin with an in-house solution

(Disclaimer: without some network/webRTC knowledge, you might struggle a bit with this post)

If you’re building your own MCU — just like us, at Tribe — then at one point you’ll need to handle RTP/RTCP. And if you chose to build upon GStreamer, it’s very likely you’ll be introducing rtpbin in your pipeline at some point.

rtpbin is a meta-plugin of sorts, that:

[…] combines the functions of GstRtpSession, GstRtpSsrcDemux, GstRtpJitterBuffer and GstRtpPtDemux in one element.

Specifically (and amongst other things, because it also takes handles the demuxing part), it takes care of re-ordering successfully received RTP packets and handling lost ones, which is all good when it works fine — but in our case, it unfortunately didn’t.

Or rather, we didn’t manage to tweak it the way we wanted, and started getting serious headaches as to why!

We found out rtpbin would send, for no reason, a disproportionate amount of NACKs leading to a drastic decrease in stream quality: not only did it actually have the packets, but asking for them again consumed much bandwidth, for nothing.

After a few days spent trying to tweak rtpbin in every possible way, we found out we had 3 options:

• dig deep into the code / rebuilt it to understand precisely what went wrong, in our case
• try and implement a workaround solution by directly using the sub-plugins rtpbin comprises (i.e, still use the GStreamer pipeline)
• implement our own solution

We did the maths, and realized patching it would require more work than simply re-coding our own, because we had already done all the RTCP parsing work previously, using Golang, so we figured out the “only” remaining part would be handling the jitter-buffer, so… that’s what we did.

I should point out that it’s really NOT easy. It’s a lot of work, but it’s also very beneficial: you end up knowing every nook and cranny of the protocols and you get to choose the language (in our case Go, and not C).

Tribe MCU, quick glimpse at the generic structure

But most of all, it enables deep and full control over how quality is going to be managed, mostly through packet reordering & jitter-buffer calculation/monitoring.

Here’s how we calculated the dynamic jitter:

Jitter = [RTT] + [3 * avgTimeBetweenRTPPackets]

In case of a NACK, the second part of the formula enables enough time to ask clients for missing packets, receive them and re-order them so as to send a consistent stream back to whatever peer the MCU was serving.

Let’s take an example : a 25 FPS stream typically has a 20ms spacing between RTP packets. So if your RTT is 100ms, that means your jitter is about 160ms, and you can put together quality control mechanisms, like:

• whenever you think a packet is late, if delay > 20 ms (for instance) then you send a NACK (because you’ll still have 40s left to squeeze it in)
• If packet is still not received, no need to NACK again: not enough buffer left to take it in anyway (but if you happened to have a shorter RTT, you could have sent several more NACKs, and maybe get that missing packet back)
• If that happens too frequently (you decide the frequency), it either means that your client struggles with encoding, or the network experiences congestion — in which case, you’ll probably send a PLI and request a whole keyframe, as well as lower bitrate expectations a bit, to smoothen things out until it gets better

What happens if you don’t handle that part? As I said before, pretty blatant quality loss: you miss packets, so intra-frames can’t be correctly predicted, and you end up with artefacts, that will last until you send PLI requests.

Artefacts, due to (a lot of!) packet loss

All is in order! Delayed packets correctly trigger PLIs, image is good

Sometimes, it’s paradoxically faster to take the long road, and do without easy-to-use, all-in-one bundle of plugins that you don’t really master nor own.

Sure, you spend some time doing it but you’ll also drastically increase both your knowledge of underlying mechanisms, and your ability to tweak and optimize them, so as to best suit your product or business.

At least, that’s the way we see things! What about you?

(With ❤️ from Tim, Sébastien, Marc & Germán @Tribe Video)