It’s been three months since we posted about the initial release of Elixir WebRTC. In that time, we’ve added a lot of features, made a bunch of improvements, and now we are about to release the second version of the library. Here, I’m going to tell you about what exactly we have done.

WebRTC improvements and new features

Here’s an overview of the new WebRTC-specific features that we have added to version 0.2:

• Support for different types of RTCP packets (these carry various information about the media stream, contrary to the RTP packets, which carry the media itself):
  - Transport Wide Congestion Control: our PeerConnection can create these based on incoming RTP packets (what are the delays between packets, which packets have been lost?) and send them to the other peer, allowing adjustments to stream parameters (like encoder’s bitrate settings) to better match network conditions. Without this, for example, Chrome will cap the video at a lower quality.
  - NACK: negative acknowledgments, our PeerConnection will send these when it detects lost packets. Then the other side can proceed with the retransmissions.
  - PLI: simplifying a bit, allows our PeerConnection to request a new keyframe from the remote peer.
• RTX (retransmissions): our PeerConnection can not only request retransmissions but also handle remote NACKs and retransmit packets itself.
• ICE features:
  - support for mDNS candidates: these are used to conceal private IP addresses.
  - TURN support: our PeerConnection can generate relay candidates and use a TURN server (previously, we could only handle remote relay candidates).
  - tracking remote keepalives, so our PeerConnection can properly detect when the remote peer was disconnected.

Broadcaster

We also created a WHIP/WHEP broadcasting server based on Elixir WebRTC called Broadcaster (not very original, I know). It allows for streaming media live from a WHIP source (whether it’s an OBS or a web browser) to multiple WHEP clients, typically web browsers. The selling point of WHIP/WHEP broadcasting is the very low latencies — we can keep them much lower than, for example, using OBS for streaming to YouTube.

Initially, Broadcaster was supposed to be just another example app (alongside Reco, which we presented in the last blog post), but we used it to stream the latest ECK Meetup, and it was a success. We decided to go with the flow and also stream some other upcoming events. If you are organizing an event and would like to use Broadcaster to stream it, reach out to us!

Benchmarks and performance

The work on benchmarking has started — we are not 100% sure of the results yet, but here’s a little sneak peek into that.

We created a simple benchmarking tool that allows us to test the transmission between two PeerConnections using an arbitrary number of tracks and bitrate. In the presented scenarios, we used about 1.6 mbit/s per track (200 packets/s, 1000 bytes of payload each, which mimics higher-quality video).

My MacBook M1 Pro was able to handle more than 200 tracks sent from one peer to the other peer (one direction only) and an AMD EPYC 7502P 32-core machine was able to handle about 100 tracks, utilizing less than 5 cores (even more in cases when the number of tracks was split between the peers, e.g. 50 tracks in both directions, as generally, the sender was the bottleneck).

It’s worth mentioning that the level of parallelization of a single PeerConnection is quite low at the moment (it only uses 3 OTP processes, for ICE, DTLS/encryption, and everything else), so the clock speed of a processor is much more important than the number of cores, but that hopefully will change, when we get to work on splitting the load of particular transceivers to different processes. In general, the possibilities for optimization are huge.

Remember that this is only a single PeerConnection, in a typical videoconferencing scenario you’d have a PeerConnection per participant. Even in big meetings (>100 participants) a single user usually receives 20–30 tracks (or how many video tiles an app can fit in a single view) simultaneously, and at that point, the performance of a single PeerConnection probably won’t be the issue to care about. Because of that, this metric is not immediately useful, but you can create a rough estimation of how the Elixir WebRTC could perform when used in the videoconferencing scenario e.g. as an SFU.

Plans for the future

We still have a lot to do, these are the things that we are planning to tackle before the third release:

• Adding support for Simulcast.
• More performance benchmarking and profiling.
• API improvements, especially in the area of codec configuration.
• Experimental support for Elixir WebRTC in the J̶e̶l̶l̶y̶f̶i̶s̶h̶ FishJam Multimedia Server (another project that we are working on and soon will go through rebranding).
• Further improvements to the Broadcaster app.

If you’re interested in Elixir WebRTC and want to keep track of the new features and changes, take a look at the roadmap. Also, check out the upcoming CommCon in London — we’ll be there to speak about Elixir WebRTC.

Thanks for reading!

GitHub - elixir-webrtc/ex_webrtc: An Elixir implementation of the W3C WebRTC API

The second release of Elixir WebRTC was originally published in Software Mansion on Medium, where people are continuing the conversation by highlighting and responding to this story.

For the last few months, we’ve been working really hard on an Elixir implementation of the W3C WebRTC specification and we are glad to officially announce its very first release ― the 0.1 version of ex_webrtc!

But why 🤔? Well, we’ve been using WebRTC in Elixir for a few years now, but it was mostly some C and Erlang libraries glued together using the Membrane Framework, and it was fine for the most part, except for some issues:

• sometimes we needed to contribute to the underlying libraries, or even worse, maintain our own forks ― that was a struggle,
• it was tough to introduce engineers from outside to our code, as it was completely different from the usual WebRTC API,
• Membrane is amazing for what it’s designed for, but we realized that the WebRTC API doesn’t really map well to the pipeline model, which made some things unnecessarily difficult.

With the previous experiences and these issues in mind, we decided to write an (almost) pure Elixir, W3C-compliant, open-source WebRTC implementation from the ground up, with the whole networking stack. Our goal is to create a solution similar to projects like Pion or WebRTC.rs, but for the Elixir ecosystem. So far, we’re on the right track!

Before getting into the nitty-gritty of what’s in version 0.1, check out this quick tutorial on how to use the Phoenix Framework, Elixir Nx, and Elixir WebRTC to create a web app that performs image recognition based on the web-cam video feed from a browser!

The app will be publicly hosted for some time 👉here👈.

What’s in the 0.1?

The first release focuses on three aspects:

• API compliant with the standard,
• ease of integration with the Elixir ecosystem,
• learning resources.

API that you’re familiar with

ExWebRTC tries to be compliant with the well-known JavaScript API while still being idiomatic to Elixir. The goal here is to make it easy for people familiar with WebRTC to jump in, but not compromise on the amazing features that make Elixir so powerful and unique. The version 0.1 already brings you most of the functions available in the JavaScript API.

For example, this snippet in JS:

const pc = new RTCPeerConnection();
const tr = pc.addTransceiver("audio");
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);

maps to the following Elixir code:

{:ok, pc} = PeerConnection.start_link()
{:ok, tr} = PeerConnection.add_transceiver(pc, :audio)
{:ok, offer} = PeerConnection.create_offer(pc)
:ok = PeerConnection.set_local_description(pc, offer)

See our API reference and examples for more.

Simple integration

One of the goals of Elixir WebRTC is to be appealing to the Elixir folks and easy to integrate into existing Elixir projects! With that in mind, we wrote the ExWebRTC Dashboard ― statistics visualization for the Phoenix Framework. Just add ExWebRTCDashboard as an additional page for the Phoenix’s LiveDashboard and you will be provided with something similar to Chrome’s chrome://webrtc-internals(but good looking 😎)!

# add this to your Phoenix app's router.ex

live_dashboard "/dashboard",
  additional_pages: [exwebrtc: ExWebRTCDashboard]

Learn some WebRTC

Understanding the W3C spec and all of its quirks is not a trivial task. We spent days reading the RFCs (my personal favorite has to be the RFC 8829, a great read for a winter evening with a cup of hot chocolate ☕), reverse-engineering Chrome’s implementation, and making issues with (sometimes dumb) questions in the webrtc-pc repo.

We wanted to gather all of that knowledge somewhere, and the first step in that direction is the Mastering Transceivers guide. It consists of a series of examples explaining some tips and tricks for WebRTC’s Transceivers. Each example contains both JS (you can quickly run it in the browser and experiment with it, all thanks to the JS Fiddle!) and Elixir snippet, which makes it easy to understand how the browser API maps to ours. We hope to add Livebooks for the Elixir code in the future!

What’s next?

We are extremely excited about what we have accomplished so far, and even more excited about what lies ahead of us! Our plan for the 0.2 is to:

• add support for Quality of Service features, like Transport-Wide Congestion Control (WebRTC sources, like browsers, use it to estimate how much network bandwidth they can use, and thus, how good the quality of the media may be) or Retransmissions (remember, WebRTC usually uses UDP, which is unreliable),
• add support for some ICE features, like mDNS ICE candidates (to prevent leakage of private IP addresses) or TURN servers (in some cases, P2P connection is not possible and TURN server needs to relay the network traffic),
• make QoL and performance improvements, and fix bugs (which are surely there).

We would love to hear your feedback and suggestions on what you need or what would be cool to add. Don’t hesitate to leave a comment on our Roadmap issue on GitHub! 🎉

Big thanks to Sean DuBois for creating the awesome Pion project and Philipp Hancke with the WebRTC community for answering our questions on webrtc-pc and the discuss-webrtc group!

Happy streaming!

GitHub - elixir-webrtc/ex_webrtc: An Elixir implementation of the W3C WebRTC API

Introducing Elixir WebRTC was originally published in Software Mansion on Medium, where people are continuing the conversation by highlighting and responding to this story.