Less jitter and delay in Chrome 52

More data nerding… I looked at the video jitter values in Chrome this time. Sounds incredibly boring, right? Well… jitter affects call quality so even if it is boring, it is quite important. I found that Chrome 52 increased the quality quite a bit!

I wanted to run an experiment to see if a certain change in our Javascript code lowers the jitter value our users experience (less jitter is better). Since we know that not all networks are equal, I started with looking up the jitter values by datacenter:

I was checking if the baseline of jitter values was constant for the last two months. But there was a trend towards lower jitter values. A somewhat strong trend even with a 30% decrease. Which was bad news for me since my experiment somewhat depends on a stable baseline to compare with. Lets see if we can get this a little more clear by ignoring the noise introduced by the datacenters and also look at the maximum and standard deviation:

So the maximum value of the googJitterReceivedMs statistic (I am a big fan of those undocumented goog* statistics) has lowered significantly. The median value (over all sessions) of the mean value (in a session) has gone from 120ms to 70ms. We did not change anything in that period. Was it Chrome 52 rolling out?

As you can see, Chrome 52 started to roll out to 10% of the users in the week of July 23rd, going to 20% one week later and 30% after another week and then went to 90%.

So why are the median values for the googJitterbufferMs statistics dropping that much way before Chrome 52 rolls out? Lets double check some other stats like the round trip time on the connection:

The median of the mean round trip time is relatively constant with spikes during the weekend (where we have a very different user population during that time). That is what I expected to see.

I poked the WebRTC folks in Stockholm and quickly got a “new Jitter buffer in M52” response. Ok… lets split this up by Chrome version:

Wow. This looks like the new jitter buffer has cut this value in half from approximately 120ms to 70ms. That is quite a significant reduction.

Going back to the way Chrome 52 slowly rolled out as we have seen above explains the roughly linear decrease in the second graph. It correlates quite nicely with the proportion of Chrome 52 users increasing and all of them have a lower mean value for this statistic. Which in turn changes the median of the mean values over all sessions.

Cross-checking with one of my favorite statistics explained a bug I had filed earlier I saw this and thought “not again…”:

So the average time the video has to be delayed to wait for audio has increased because of this? Sounds not so good anymore. Fortunately, the target delay (which is probably the most user-visible metric here) has still decreased by about 40ms (or 25%) as we can see here:

Which is awesome.

Good news for me: I can still run my experiment. But I have to only look at data from Chrome 52 onwards. Then my assumption that the median of mean jitter is constant holds.

More importantly, catching this also showed that it is possible to spot changes in the metrics we collect before the next version rolls out fully. Now it is just a matter of finding the right graphs to look at…