Where to deploy TURN (or other relay) servers

Philipp Hancke
Aug 1, 2016 · 3 min read

I read an interesting article on NoJitter which described the way Cisco is deploying Spark. By rolling out media servers in data centers close to the users Cisco wants to ensure a sub-150 millisecond delay for anyone participanting.

Following up on Dag-Inge’s blog post which looks at connection setup time depending on TURN servers I wondered if we can measure something in the same spirit that tells us whether we are deploying TURN servers (and in the future SFUs) in the right locations or if we need to optimize our deployment. After all, as I wrote a while back, this kind of operations infrastructure is one of the most important aspects of running a WebRTC service.

Our current deployment strategy at appear.in is load-based. In every AWS data center available to us, we deploy one or more TURN servers, look at the load and fire up another intance if the individual instances get too much traffic. We then use route53 to route clients to the nearest TURN server by letting them resolve ‘turn.appear.in’.

But how fast and effective is that?

For peer-to-peer connections the time it takes to gather a relay candidate from a TURN server is a pretty good indicator of the round trip time between the client and that TURN server. This is easy to measure by looking at the delay between the RTCPeerConnection.setLocalDescription call and the first icecandidate event where the candidate type is ‘relay’.
Roughly like this:

var pc = new RTCPeerConnection({iceServers: […]});
pc.onicecandidate = function(event) {
if (event.candidate &&
event.candidate.candidate.indexOf(‘typ relay’) !== -1) {
console.log(‘took’, stop — start);
pc.createOffer({offerToReceiveAudio: 1}).then(function(offer) {
start = Date.now(); return pc.setLocalDescription(offer);

At the same time, it isn’t affected by clients doing calls across regions like measuring call setup time is. What happens under the hood is that (unless you are using some tricks like Facetime) the browser will ask the TURN server to create an allocation, get back a not-authorized error and ask again with credentials. So this takes two round trips (unless there is packet loss). The relay address in that candidate also gives us an indication of the relay server used. We did not gather that before but it was easy to add.

Since the data again contained quite a lot of outliers we’re cutting of things at 5000ms:

SELECT MIN(medianvalue) as median, datacenter
SELECT relayaddress, datacenter,
MEDIAN(gatheringtimeturnudp) OVER (PARTITION BY datacenter)
AS medianvalue
FROM dataset LEFT JOIN turnlocations USING(relayaddress)
WHERE gatheringtimeturnudp < 5000) t1
GROUP BY datacenter order by median;
median | datacenter
198.0 | Oregon
212.5 | San Jose
257.0 | Dublin
268.0 | Ashburn
276.0 | Tokyo
280.0 | Sydney
302.0 | Sao Paulo
313.0 | Frankfurt
332.0 | Seoul
406.0 | Singapore
442.0 | Mumbai

Intuitively we knew that running TURN servers close to users is a good thing so the results are pretty good. Remember that we are measuring two round trips so we can divide by two to get a (very) rough estimate for end-to-end latency.

Now we can measure to a degree where intuition does not cut it anymore. We can use these metrics to measure if deploying another TURN server in a particular region improves latency or if we should run servers in another datacenter. The Singapore and Mumbai datacenters are two examples of this.

One of the more odd results was that there is roughly 20ms higher delay in gathering candidates from Firefox than from Chrome. Pending further investigation that looks like there is an opportunity for Firefox to get faster :-)

Hear more about this at ClueCon in Chicago next week!

The Whereby Blog

The most recent blogs from the Whereby team

Thanks to Dag-Inge Aas

Philipp Hancke

Written by

doing things webrtc @appear_in

The Whereby Blog

The most recent blogs from the Whereby team

More From Medium

More from The Whereby Blog

More from The Whereby Blog

A Remote Community

More on Video Conferencing from The Whereby Blog

More on Video Conferencing from The Whereby Blog

Make your video meetings inclusive and engaging!

More on Video Conferencing from The Whereby Blog

More on Video Conferencing from The Whereby Blog

New integrations and improvements coming to our Pro plan

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade