How To Test 5000 Websites in 7 Days — Recapping the Firefox Quantum Sprint
Early this month, from October 2–8th, hundreds of volunteers from 50 countries around the world came together to test websites and file web compatibility issues in a Firefox Quantum Sprint. By the end of the week 5900 website were tested and 1133 bugs were filed.
The approach was simple, engage our local communities of enthusiastic Mozillians to organize events and test Firefox Quantum in their region. Increasing the quality of Firefox Quantum by extending the reach of our testing to more parts of the world through volunteers.
Volunteers who registered on the site were pointed to a list of the 200 famous websites, by country, and instructed to report any bugs on the webcompat site.
As the campaign ran we responded in real time to feedback and metrics. We were thrilled with the volume of participants but identified early that we needed to increase the quality of the reports and immediately and shared additional guidelines by the webcompat team on all channels.
Gamification of contribution is often held up as a best practice. In the case of our campaign, we believed that the gamification was unhelpful — we were running a high number of empty or incomplete bugs while the leaderboard was up, and taking it down saw an immediate increase in the quality of submissions. Participants were there to help Firefox.
The campaign produced over eleven hundred bugs in a week, dramatically increasing the regular volume of web compatibility issues. The Web Compatability team reported that for certain locales they received bugs they would normally never have seen, for example some issues around Hindi and Malayalam fonts.
In spite of the dramatic increase in the scale of the effort, the quality of the bugs filed remained relatively stable, with a 20% validation, and 1% high value.
We believe we can still increase the quality of bug filing, and participants and staff alike were positive about the campaign and particularly excited by the enthusiastic response to a call to make Firefox better.
The Open Innovation Team is continuing to explore new ways to connect our communities of enthusiastic volunteers to meaningful projects within Mozilla. With each campaign or initiative we learn more about the best way to connect volunteers who care about Mozilla’s work to appropriate impact. From the perils of the leaderboard, to the importance of bite size instruction, we’ll be sharing more of what we learned in this campaign and how it’s driving our relationship with internal teams and enthusiastic non-coding contributors forward in the coming weeks.