Concord: How I built a screen sharing application in two weeks
A couple months ago I was searching for a screen sharing application to collaborate with friends. I’m a software engineer by trade and I naturally am working on a couple of side projects at any given time. Finding a decent screen sharing app turned out to be much more of a headache than I could’ve imagined. Every app my friends and I used consistently had a caveat that rendered it unusable.
The Discovery Phase
I’m not sure how many people have tried using any sort of screen sharing app these days, but it’s a bit of a mess. You’ll have to either download an application, install a plugin, register countless accounts, send friend invites, or even pay hefty premiums. All the while the quality is still subpar.
Hangouts? Not optimized for screen sharing. Text is blurry and contains artifacts with no way to control the quality.
Slack? Screen sharing is a premium feature at $7/user in the Slack server.
Skype? Ha…
Eventually I discovered that Discord has a screen sharing feature. After giving it a try, we discovered that the quality was not good enough to share detailed text. However, Discord’s premium Nitro (now Nitro classic) service was $5/mo for one person to gain the ability to stream high quality video to any number of people. We decided to give it a try and… it worked very well! Occasionally the quality would degrade, depending on CPU/network quality, but for the most part it handled streaming 1080p video with legible text fairly easily.
The Depression Phase
Shortly following the hunt, I purchased a Macbook. The upcoming iOS apps I would be developing and the Docker for Windows fuckery I experienced (story for another time) made it an easy choice, if not an essential one. Surprisingly enough, I had gone through life until this point without owning an Apple device or even a laptop.
Almost immediately, I discovered that Discord’s screen sharing capabilities crumbled to the ground on my shiny new Mac. My 3rd generation Core i5 processor on my desktop handled 1080p streaming fine, but the same attempt on my core i7 6-core Mac (with discrete GPU) brought the thing to its knees.
I’m not exaggerating. Starting up the camera or a 720p/1080p screen share session rendered MacOS unusable. Every animation was jittery, typing was behind massive input lag, and that was just sharing my screen and editing a text document. A look at the activity monitor indicated that Discord was utilizing anywhere from 100–300% CPU. Yikers…
The Investigation Phase
Clearly, there is something wrong here. There is a bug in Discord’s video encoder that causes MacOS to fall over.
I started off by hitting up the Google. Didn’t take me long to find a promising lead: YouTube. Google is behind a video compression format known as VP8 and its successor VP9. YouTube has taken to encoding all of their videos and serving them using VP8 for 1080p and lower resolutions and VP9 for 2k and higher resolutions. The alternatives, h264 for 1080p and h265/HEVC for 2k+ were mostly abandoned by the video gargantua.
Enter stage left: Apple refuses to buy in to VP8 and VP9 encoding in its entirety. You might see where this is going… This has been an issue with Safari not being able to decode YouTube videos properly for years. When you browse YouTube in Safari, you will not be given the option to view 4k videos and will be forced onto 1080p, as that’s the only resolution YouTube will encode videos using h264. If you’re on a Mac and use Chrome, however, you can view 4k videos and will be served content using VP8/VP9. That’s fantastic, at least we can still watch 4k YouTube videos using Chrome instead of Safari! Well, hold up there Jimmy! Of course it’s not going to be that easy.
You know how people don’t like to use Chrome on laptops because it’s a resource hog and kills your battery? Here’s one reason why: When you view YouTube videos via Chrome on a Macbook, you’re going to use the incredibly inefficient software decoder to do so. An 8th generation Intel Core series processor has hardware acceleration that’s supposed to make this process easy for you, but Apple blocks that at the OS layer. So, when you’re watching a video on YouTube, your computer does not know it has the ability to do that much more efficiently because Apple decided they don’t like Google’s encoding formats. There is even a Chrome extension that lies to YouTube for you, saying that your browser does not support VP8/9 to force it to provide you h264 content, in only 1080p of course.
What does that have to do with Discord?
The Reporting Phase
You could probably guess what video encoder Discord uses for video sharing… VP8! This can be confirmed by looking at the call debugging statistics while in a Discord video call.
Voila! I typed my findings up into a bug report on Discord. Discord uses a community-based volunteer bug hunting group to report, reproduce, and approve bugs reported by their users. I joined the server, reported the bug, and explained to the community what I’d found. The report had a few people reproduce it on their Macs as well as a couple saying they were not able to reproduce. Confused, I explained in even more detail how it is not possible for a Mac to encode VP8 efficiently and how they should be checking for Mac users on any side of a call to switch to h264 encoding.
Finally, the bug was approved and it appeared on their Trello board. I was brought onto the Discord bug hunting program as a bug hunter and left feeling mildly satisfied with myself. But wait, I still don’t have a way to screen share with my friends. Goddammit…
The “Fuck it, we’ll do it live!” Phase
I don’t remember how I initially made the decision to roll out my own screen sharing app, but that’s exactly what I ended up doing. I spent the last two weeks building what I wanted out of a video calling app. WebRTC made this a fairly painless endeavor. WebRTC is a project that aims to bring a standard for real time communications to the modern web. The API’s are mostly adopted by the big 3 browsers these days, though it can be a little fragmented depending on which browser you’re using. For my app, I decided to focus purely on Chrome, to make things simple.
The end result is concord. A react app with dependencies on node.js, express.js, react-router, and peer.js. On loading concord, you’re given a copy-able link that you can share to others. When the user clicks your link, they are brought to concord and immediately connected with you in a peer to peer call. In this call, you have the ability to talk, screen share, and video share via a camera. All without downloading an app or plugin, registering for an account, or maxing out your CPU.
Starting an audio call is very easy:
//caller
await navigator.mediaDevices.getUserMedia({audio: true})
.then(mediaStream => {
let call = peer.call(id, mediaStream);
call.on("stream", (stream) => {
let audio = document.getElementById("speaker");
audio.srcObject = stream;
audio.load();
});
});//receiver
peer.on("call", (call) => {
navigator.mediaDevices.getUserMedia({audio: true})
.then((mediaStream) => {
call.answer(mediaStream);
});
call.on("stream", (stream) => {
let audio = document.getElementById("speaker");
audio.srcObject = stream;
audio.load();
});
});
Adding video is still simple:
await navigator.getDisplayMedia({video: true})
.then(mediaStream => {
//attach the stream to a video element similar to audio
});And you can control the video quality via content hints:
mediaStream.getVideoTracks().forEach((track) => {
if ("contentHint" in track) {
track.contentHint = 'text';
}
});Thought it was lovely that the World Wide Web Consortium included a content hint that I specifically was interested in using just a few months before this project booted up. And the result was very promising!
That’s concord streaming greater than 1080p at a bitrate that is optimized for viewing text while consuming 36% CPU. Here’s what it looks like on the other end:
Of course it’s difficult to convey the quality of a video session in a screenshot. So, if you would like to try it yourself, go ahead! https://concordapp.co should be live so long as it’s not hammered by the internets.
There is no database that keeps track of users and my server is only used to serve the JavaScript, HTML, and to provide the secure connection to the other person. There are times with certain router/firewall configurations that you might not be able to reach the other person directly, and in that case your data will be routed through my server and immediately relayed to the other end (TURN server info). This functionality is minimally tested because I ran out of people who’s network configuration conflicts with a peer to peer connection.
Unfortunately, as of this time, you need to enable Chrome’s experimental web platform features to be able to screen share (chrome://flags/#enable-experimental-web-platform-features). This is only meant to be a smaller project to see what I could do with WebRTC and React while also trying to build something I can realistically use with my friends. I think it’s turning out quite well having spent 2 weeks on it during my free time (and keep in mind, Smash Bros Ultimate came out last week…).
If you have a suggestion or an issue with concord, let me know!
bradziolko@gmail.com
Oh, and the result of that original Discord bug report? Cannot Reproduce…
🤔
UPDATE 18 Dec 2018:
I was informed midway through the day today that concord was non-functional. The Chrome 71 update released yesterday broke my implementation of the screen share functionality. My fault entirely, I for some reason did not think to test on Chrome Canary prior to release.
Concord has been updated to be compatible with the latest Chrome versions (stable 71 and unstable 73), so for those of you that couldn’t try it yesterday, you should be able to do so now.
Sorry for the inconvenience, folks!
This story is published in Noteworthy, where 10,000+ readers come every day to learn about the people & ideas shaping the products we love.
Follow our publication to see more product & design stories featured by the Journal team.
