Firefox data platform & tools update, Q1 2017

The data platform and tools teams are working on our core Telemetry system, the data pipeline, providing core datasets and maintaining some central data viewing tools.

To make new work more visible, we provide quarterly updates.

What’s new in the last few months?

On the data collection side, scalars are now supported through the pipeline, so new flag and count histograms are now disallowed on Desktop in favour of boolean and uint scalars.

Event Telemetry is now ready for first adoption. A general events table is available, a sync events table coming up and further uses are being looked at.

For documentation, we re-worked the guide for adding new Telemetry and extended the detailed data collection documentation. The prototype for making probe history more discoverable now has daily updates and supports Nightly too.

For filing or finding bugs, there is now a new Data Platform and Tools product. Note that client-side bugs still go into the separate Toolkit::Telemetry component.

The data pipeline work powers results for re:dash and custom analysis among other things. Notable recent work here includes:

The data tools work powers tools that make data analysis more accessible across Mozilla. Updates here are:

  • For re:dash, the UI improved to make the dashboard list more accessible.
  • re:dash query issues were reduced by handling failing queries using exponential back-off.
  • There is also a python re:dash client (h/t to emtwo), allowing programmatic generation of queries and dashboards.
  • The distribution viewer is now live, making distributions of a set of important Firefox metrics available.
  • The analysis service gained features like persistent cluster storage and the ability to extend cluster lifetimes.

What is up next?

For the next few months, interesting projects in the pipeline include:

  • Work to decrease data latency, by sending the last ping of a Firefox session immediately. We will also start sending timely pings for new users and updates.
  • Rebooting documentation, providing guidance as well as tying existing documentation together.
  • Start supporting new data collection from add-ons in Telemetry, starting with events.

How to contact us.

Please reach out to us with any questions or concerns.