Adios, Amigo
TL;DR Firefox Test Pilot is flying off into the sunset on January 22nd, 2019. Currently active experiments will remain installed for all users, and will be available on addons.mozilla.org after this date. Non-extension experiments like Firefox Lockbox and Firefox Send will continue in active development as standalone products. In fact, both products will have significant launches in the near future. Stay tuned for updates in the coming months.
I’ve been involved with Test Pilot since the project’s inception (true story, we called it Idea Town at that point) almost four years ago, so this announcement is definitely bittersweet for me. Nevertheless, I think it’s the right move for Mozilla, Firefox and for the Test Pilot team.
In this post, I’ll talk about the why of Test Pilot’s termination, the impact of the program at Mozilla, and what happens next.
Why is this happening?
Here’s where I feel like our team can rest on our laurels a little bit. Test Pilot being disbanded is more-or-less a symptom of our program’s successes both in terms of products shipped and cultural impact within the Firefox organization.
Prototyping as a culture
When we founded Test Pilot, my teammates and I wanted to address a dilemma at Mozilla that felt particularly acute at the time: we didn’t have a great way to get new Firefox features to market quickly or get feedback in a timely manner. Firefox ships to hundreds of millions of people, and follows a conservative release cadence as a consequence. While this supports the overall quality of the browser, it can be difficult to get feedback or understand user attitudes in service of feature development.
Test Pilot was envisioned as a response to this problem: what if we could validate products before we expended the energy to ship them at Firefox’s rather significant scale? What if we could leverage our most ardent supporters, solicit their input, and iterate toward better products or simply kill things that didn’t really work?
And Test Pilot scratched that itch. We helped engender what Mozilla now describes as a culture of experiments. When I started at Mozilla, prototyping and lightweight validation techniques were mostly alien to the company, but now they’re commonplace. Conversations at Mozilla about possible futures are now framed in terms of how we might experiment or prototype or conduct user research to make smart product decisions.
This is not to suggest that Test Pilot was alone in bringing about this change. We were maybe in the vanguard, but our thinking definitely co-evolved with many other teams at Mozilla, and this cultural shift is one of the reasons Test Pilot’s time has come. The company at large has learned a ton about privacy-respecting research, prototyping and product experimentation, and we’re no longer dependent on one team to drive these practices. As 2019 progresses, expect to see more opportunity to experiment with and participate in the development of new products at Mozilla.
Shipping and Receiving
Over the last three or so years, the Test Pilot team has built — or facilitated — a number of popular Firefox features like Screenshots, Containers, Facebook Container, and Activity Stream. We’ve also made a raft of popular add-ons like Side View, Firefox Color, Snooze Tabs, Tab Centre (RIP), and Min Vid, and facilitated Mozilla’s first non-browser mobile products in Notes, Lockbox, and Send.
We were always meant to be a small prototyping team, and while we paired with other groups internally to build some experiments, we wound up amassing a stable of products that we could not reasonably ask other parts of the organization to maintain. With each successful experiment, we shed some of our prototyping capability in services of developing and maintaining scaled product experiences.
To wit, when Screenshots became an early success in Test Pilot, it took three engineers and a designer off of building experiments for Test Pilot and onto the full-time job of managing and growing Screenshots in Firefox. Eighteen months later, Screenshots is a product that is now used 20M times a month and demonstrably improves Firefox retention.
At first glance this kind of problem might seem obvious: just bring in more people. Maybe in a perfect world that’s possible, but people and teams aren’t infinitely fungible or replaceable. Hiring is hard, as is bringing people into in-flight projects. Relative to other browser vendors, Mozilla is a small company that punches well above its weight. We don’t have a massive surplus of engineers, designers and product managers lying around in storage. Over time, our team simply had to grapple with a growing stable of products.
It’s a similar story with Send, a product for which we are preparing a new release featuring an Android client, Firefox Accounts integration and much larger transfer limits. In order to invest in this product, we’re peeling off more resourcing from prototyping. Long story short, we’ve self-cannibalized by building stuff now deemed important to the organization writ large.
From Features to Services
The other trigger for this change is that our team have been moving away from developing new Firefox features and starting to invest in services that stand apart from the browser and bring Mozilla’s core philosophical commitments to privacy, security and user control to new audiences.
Firefox Send in particular represents a service that works totally independently of the Firefox browser (in fact more people use it on Chrome than on Firefox). While Firefox Lockbox is definitely a companion app to the Firefox password manager for now, it’s also 100% mobile and backed by Firefox Accounts and Sync.
Whereas Test Pilot was initially intended to test Firefox features, products like these started to expand our horizons. Quickly, teams outside of Test Pilot started pushing into this space. Firefox Monitor — a service that lets you see if your passwords have been involved in a data breach was a real hit for our company in 2018.
These types of projects represent a new set of opportunities to expand our brand. As they take up more and more of our time, maintaining a standalone platform for feature experimentation in the Firefox browser becomes more difficult to justify.
What’s Next
First thing’s first: if you are using any currently active add-on experiment, you can keep using it. Nothing’s gonna change in your browser, we’ll just automatically migrate you off of the Test Pilot versions of the add-ons.
Firefox Send and Firefox Lockbox will continue in active development in 2019 as standalone products. Notes, Firefox Color, Side View, Price Wise, and Email Tabs will all remain available at addons.mozilla.org for the foreseeable future. (ed note: I’ll add links to these new URLs here once I have them early next week). Email Tabs in particular may undergo some changes to make it more flexible and less directly connected to GMail, but otherwise these projects will be accessible in their current forms with occasional maintenance releases going forward. Speaking for myself, I hope to add some new palette generators and color properties like borders, sidebars, new tab &c. to Firefox Color, a particular favorite of mine.
As for the Test Pilot site and add-on, we are replacing the site with a farewell message on January 22nd. Visiting the site after this date will automatically uninstall the Test Pilot add-on, but you can also just uninstall it manually if you wish whenever you like without affecting installed experiments.
This blog will remain available indefinitely. Since many of the projects that started in Test Pilot are still actively being worked on, we’ll continue to post updates to projects here as events warrant.
A final, slightly personal, note
Test Pilot has been an absolutely phenomenal project to work on for nearly four years. The Mozilla community gave so much life to the project, and my teammates and I remain deeply grateful for your support, encouragement, and willingness to contribute.
To have had the opportunity to evangelize prototyping and rapid, iterative development processes at Mozilla has been the greatest pleasure of my professional life. Sharing the journey with the incredible women and men on the Test Pilot team made the experience singularly amazing. It’s been a great ride, and I’m so excited to see what comes next.
Ad astra
