Test Pilot graduation report: Pulse
As we began ramping up for our release of Firefox 57 in November, our product intelligence team faced a difficult problem: they had lots of telemetry to help understand Firefox’s performance, but no way of understanding exactly what those numbers mean to users. Pulse was designed to bridge the gap between hard performance numbers and the incredibly vague concept of user satisfaction.
At what point does page size affect user satisfaction? On which sites do users report the most problems? To what extent can users perceive the impact of new Firefox features like e10s and multiple content processes? These were all questions we hoped to answer.
How it worked
When installed, Pulse tracked performance metrics for every page visited. The page weight, time to load, content process count, and much more were recorded. These were submitted to the Test Pilot team whenever the user filled out a short, four-question survey, which asked for details about their experience. This survey was presented to users in two ways:
pageActionwas registered with the browser, putting an icon in the URL bar that allowed users to report their sentiment at any time.
- Around once a day, users were directly prompted by the browser, using a
This arrangement allowed us to collect two distinct but equally important types of data: random-sampled data, reflecting the average experience of the user, and outlier data, reflecting when a user’s experience was so good or bad that they felt compelled to report it.
Since user sentiment is a notoriously challenging thing to measure, we tried to restrict analysis to submissions reporting the primary reason for a score as fast or slow. This reduced noise from users who may have been reporting, for example, whether they liked or disliked a website. We also used a segmentation methodology similar to Net Promoter Score to cluster users into positive- and negative-sentiment groups. Those who provided a rating between one and three stars were considered detractors, while those that provided a five-star rating were considered promoters.
What we learned
In short: a lot. Thanks to the dedication of our users, we collected over 37,000 submissions in just a few months. We can’t thank you enough for all the time and effort you put into helping us understand how to make Firefox great.
Product intelligence is continuing to comb through the data to look for meaningful findings. A few that stand out:
- Performance matters. Nearly every metric showed significant evidence that poor performance negatively affects user sentiment.
- Ads hurt user sentiment. One of the strongest effects on sentiment was our proxy for the number of ads: requests made by the page to hostnames on the Disconnect.me tracking protection list. This covaried with the overall number of requests, total page weight, and each of the timers, so it’s unclear if this effect is due to a specific aversion to ads, or to their consequences on performance.
- Developers should focus on
DOMContentLoadedto improve perceived performance. Timers were placed on a number of events in the page load cycle: the time to first byte, time to first paint, time to the
window.loadevent, and the time to the
DOMContentLoadedevent. The one that most consistently affected perceived performance was the time to
DOMContentLoaded. If you want to see strong returns on a limited amount of time to tune your site’s performance, try to reduce that number. Your users will thank you.
As we move close to the Firefox 57 release in November, the team will continue using the data collected by Pulse to make sure that it’s the fastest Firefox yet.