Shield Studies — Go faster, bet smarter

In my last post, we discussed the superpowers that Heartbeat unlocks for us by giving us a direct line of communication with our users. Today I’d like to introduce you to Shield Studies. Shield Studies are controlled A/B tests built right into Firefox. These studies allow us to compare proposed changes to the default experience in Firefox for small, representative, populations before shipping those changes to everyone. Shield Studies will be available on all release channels in Firefox 56, initially in English locales only and available globally later this year.

Heartbeat gave us the ability to talk directly to our users, to really understand their needs and desires. We use the insights from Heartbeat in addition to the amazing work of our market research, user research, and other teams throughout the organization to help us narrow down a list of many possible product changes to a short list of changes with strong chances of success.

Once an idea has been through several rounds of research and low-level validation we need to make a tough decision.

“Is this the right decision for our users?”

In the past, it’s sad to say that we would probably just use our gut.

“We *think* this is a good idea. We *think* users will love it. Let’s ship it!”

Unfortunately that’s not the right way to build really compelling products. My friend and colleague Brendan Colloran said it best,

“Every time we push out a new product or feature without doing proper validation testing, we are running a massive uncontrolled experiment on our entire user population.”

So how do we make more informed (data driven) product decisions without compromising our principles? Get feedback directly from our users! You see, nothing is quite as valuable as giving users something that they can touch, feel, and explore in the product. That is exactly what Shield Studies have been designed to do. Shield Studies tell us with certainty if what we are building is actually better than the current experience. So what does this look like in practice? Good question!

Test Pilot is an awesome project. It allows us to try new and innovative ideas in a safe environment. The ideas may not be bug free or highly polished at first, but by the end of their time in Test Pilot we know they really shine! So if a feature performs well in Test Pilot we should ship it in Firefox, right? It’s actually not that easy. Test Pilot is a heavily biased audience. It is predominantly early adopters and highly technical users. Just because a feature appeals to Test Pilot users doesn’t mean it will generalize to the broader Firefox population. So how do we decide if we should ship it or not? You guessed it, Shield Studies!

Shield Studies allow us to give newly graduated features from Test Pilot to small representative populations of the Firefox Release audience. These users will get a chance to use these features for a period of time and then be asked to provide feedback on the experience. If the feedback is exceptionally positive we’ll know the feature is safe to ship. If not, then maybe it’s better for the feature to live on as a web extension than as a core Firefox feature.

This is just one example of how we’re using Shield Studies to make smarter decisions. Not all Shield Studies will be big and flashy. They won’t all be new features. Shield Studies can be as simple as deciding on the right color, copy, or icon for a button. We want to be daring. We want to be innovative. There are so many things we *could* do in Firefox, but we need real users to tell us what we *should* be doing.

So how will I know if I’m in a Shield Study?

Most Shield studies will be subtle. If we’ve done our jobs right it won’t be a jarring experience, so you might not even notice. The changes we’re testing should feel familiar and comfortable. If you’re curious though, you can always type “about:studies” into the awesome bar and hit enter. This page will tell you if you’re currently participating in a Shield Study. If a Shield Study is interrupting your workflow or degrading your Firefox experience, please disable it. That is a really strong signal to us!

What if I don’t want to participate in Shield Studies?

Opting out of the Shield Studies program is easy. There’s a new option in the data collection menu that controls Shield Studies labeled: “Allow Firefox to install and run studies.” We’ve also got a handy SUMO article that will walk you through the process. We want to make sure all users feel in control of their Firefox experience and this is no different.

What data do you collect in Shield Studies?

Excellent question! Shield Studies can only collect Type 1 and 2 data. This is the same type of basic interaction data we collect by default as stated in our privacy policy. There may be instances where we want to collect data that is not covered by our default data collection policies. In those instances you would be prompted to opt into the study. You would be given a complete description of the study and any additional data that would be collected before enrolling. A great example of this is Pioneer, which is actually an opt-in Shield Study currently in the wild.