Stream LITERALLY anything into and out of Unity using WebRTC

WebRTC is an open framework that enables Real-Time Communication (RTC) across web browsers. You can send several simultaneous streams of data, video, audio, or combinations of them using this resilient and low-latency protocol.

Unity also has render-streaming but the following type of implementation gives it massive control to create a stream across languages, frameworks, and deployments. For example, you can create a WebRTC stream of data from a Python server or browser client or Android app, or even another Unity game.

Here’s an example usage: https://www.youtube.com/watch?v=v4T8AzNyzqE&ab_channel=VigneshS.

Streaming video from browser into Unity using WebRTC

Firstly, can you explain WebRTC like I’m 5 ???

Give a quick glimpse over this article for the inner workings that will be necessary for further explanations:

https://medium.com/@trevahok/explain-webrtc-like-im-5-885371dd7186

Right, but you said WebRTC in Unity?

Parts of WebRTC and their inter-communication

Yes, slow down. You can create a WebRTC stream between two Unity game instances. That's the easy part and there are already examples for it on their website. To create a WebRTC stream across devices, you need to write your own implementations of the following:

• Unity WebRTC library — Create a library to create an abstraction and create a WebRTC DataChannel and to receive a stream.
• WebRTC JavaScript library — Write a simple library abstraction that sends a video camera feed as a stream from a browser to Unity. My implementation uses a client-side JS library but you can use Python or NodeJS to establish a server-side WebRTC stream too.
• Signaling server — To carry the SDP data from Unity to JS through WebSockets protocol. I wrote this in NodeJS using WebSocketServer.
• Serialization — Uses a standardized format like JSON or ProtoBuf to send and receive information between peers during signaling.

Let's start with the Signalling Server

A signaling server is essential to send the session information between two peers so they can set up the connection. You just need a way to send the SDP, offer, and answer data between two peers. Doesn’t matter how. Could be a POST request, HTTP request, WebSocket or write down on a piece of paper and key-in on the peer, doesn’t matter at all.

I chose to go with WebSockets.

1. On Unity: WebSockets-unity package
2. On JavaScript: native WebSocket API on browser
3. On NodeJS: I just set up a WSS server that broadcasts everything it receives to every other socket. Talk about laziness huh?

Seriously? That’s all on NodeJS?

Yah, literally that's it. If you get a message/event via one WebSocket, send it to all other sockets except itself.

WebSocket Server on NodeJS

Ok, now that we have a WebSocket server, what are the events that can happen between browser and Unity?

So what are the different events that can occur between two peers?

1. Offer — Browser sends an offer
2. Answer — Unity responds to offer from browser
3. IceCandidate — Both Browser and Unity share

Uhhh ok and what format are these sent in?

Ideally, you would use a standard marshaling/unmarshalling format and library like JSON serialization or ProtoBuf but I was lazy and the data wasn’t too complex so I just serialized it as a CSV string.

The format roughly looks like:

“ EventType, data1, data2 …”
Eg. “ice, sdp:candidate … ”

Example type of IceCandidate event

Got it, so let’s go through the flow of code to establish a WebRTC connection ?

Cool, as long as you follow along.

1. Browse creates an Offer and setLocalDescription() on the offer

2. Unity receives the offer setRemoteDescription() on the offer.

3. Unity creates an answer, setLocalDescription() on the answer and sends it to Browser.

4. Browser receives the answer and setRemoteDescription() on the answer.

5. Unity can now establish a DataChannel for bi-directional communication.

Wait a min, DataChannel? Can it also send Video and Audio?

Jeez, you’re quite antsy. No, DataChannel can move only binary and text data. You’ll need MediaStream for Audio and Video.

DataChannel setup is the hard part. Once that is done, you can quickly add a MediaStream to an existing PeerConnection. Pretty nifty huh? ;)

Add VideoStream from webcam

Conclusion:

Yah, not really sure what your antsy personality got from this but this is VERY cool stuff!

I haven't seen anything similar on the internet and I’m very excited to stream data from android, ios, or any other client that can use this type of streaming capability. You can even play a whole game from the browser if you use this type of connection!

Resources:

Set up a WebSocket server in Node.js | Postman Blog

WebRTC

WebRTC API - Web APIs | MDN