Min Vid Graduation Report
Min Vid is becoming a Shield experiment, and will remain installed for current users for as long as possible while we explore implementing the feature natively in Firefox. We have no set timeline for this work yet, but will continue to provide updates on this blog.
We launched the Min Vid experiment in Test Pilot in the Fall of 2016. Min Vid created a pop-out video player that let participants play videos in a small, standalone window that would sit on top of any other content on the screen.
Min Vid has been a success in Test Pilot, both in terms of usage, and in terms of what we learned in the process of building it. From the start, the feature proved extremely popular with our audience. It’s consistently been our most installed experiment since Page Shot left Test Pilot to become Firefox Screenshots.
At the same time, developing Min Vid was challenging for our team. In order to ship the feature quickly, we relied on older add-on code that first proved unstable and finicky, and later became completely unusable. The bugs that cropped up in Min Vid were sometimes hard to reproduce, sometimes hard to diagnose, and quite often required significant engineering effort to fix. Beyond this, the engineering, design, and product management efforts required for taking Screenshots to market in Firefox while maintaining a pipeline of new experiments meant that Min Vid often had less attention than it deserved given its popularity in Test Pilot.
In this post, I’ll discuss what Min Vid was, what we learned from the experience as a team, and what will happen next for the experiment. Unlike most graduation posts, I won’t dive into metrics here. I’ll save that for a later post after we test Min Vid with more Firefox users.
How it went down
Min Vid is the brainchild of Test Pilot engineer Dave Justice. Dave’s original pitch was something like: “I want to be able to play software development videos from YouTube in the corner of my text editor while I’m writing code.” The idea had instant appeal for members of our team.
Prior user research suggested that secondary media consumption — watching YouTube, Twitch or a baseball game in separate browser window while working on a distinct primary task— is a common use-case among Firefox users. It also seemed like an idea that would be easily understandable to users given that picture-in-picture video windows are a long time staple on mobile operating systems and televisions.
The initial proof of concept for Min Vid was actually rather simple to write. At the time, Firefox included an extremely permissive framework for building add-ons called the add-on SDK which was slated for demise in favor of the new, more stable, better documented, more performance friendly (but less permissive) WebExtensions framework. While we knew the SDK would disappear in future Firefox releases, we built the first Min Vid release using the older framework because of the speed and flexibility it afforded at the time. Taking this approach allowed us to ship a lot faster, but it ultimately lead to a lot of down-the-road refactoring work.
The first version of the player worked by registering a little button over YouTube video players and video preview thumbnails. Clicking this button would pop the player out into a standalone window allowed users to drag, minimize, scrub playback, and adjust video volume. We used YouTube’s public API to send videos into an iframe embedded in the Min Vid player window.
While we launched with just YouTube support, we went on to add several other features into Min Vid, most notably history and queue features. We also added support for Min Vid in Vimeo and SoundCloud. Because we relied on public APIs from these sites to pass media content into the Min Vid player, each new integration required a bespoke engineering effort.
What we learned
The Test Pilot program was pretty young and still a little inchoate when we launched Min Vid. From a personal perspective, I was still transitioning from my role as Test Pilot’s Design Lead to Product Manager when we launched, and I struggled with limiting the timeline and scope of the project.
Nevertheless, the experiment found its audience. That said, there are some things I wish we’d done differently for the sake of the experiment, its users, and our team.
Set goals from the start and then use them to limit work later
From the beginning, we identified some very simple success goals for Min Vid in Test Pilot: that it should show a steady retention curve, and that it should make Firefox more sticky for those users in the experiment cohort.
The problem was that at the time, we didn’t quite understand that setting goals like this can save time by limiting the eventual scope of work on a project. Instead, we got a small case of wouldn’t-it-be-cool-if-itis about new features in Min Vid.
Take for example our Vimeo and SoundCloud integrations. Seemingly pretty cool things do to? Well, this graph shows where Min Vid sessions originated last month.
Basically, these features got zero traction whatsoever. In hindsight, we should have been more aggressive about trying to move Min Vid out of Test Pilot and into the hands of test audiences in release Firefox before we started adding features to the experimental project. It would have saved us time and shaved off a lot of complexity in our codebase.
You’ll always underestimate how much refactoring costs
As I mentioned above, we originally wrote Min Vid as an SDK add-on. Over the last few months we’ve migrated it to the new WebExtensions framework (which, disclaimer, we’ve had to supplement with bootstrap code).
This kind of work — changing a codebase without changing functionality — is called refactoring, and it happens a lot in software development: libraries of code expire, frameworks become deprecated, code maintainers move on to new jobs, and so on.
While refactoring is a part of life in software development, it’s also one of the most difficult tasks in terms of estimating time and cost. There are always going to be unknown obstacles faced while migrating code from one paradigm to another: things don’t work as promised, unexpected errors arise, tasks that seemed simple from the outside become more complex in the process.
For Min Vid, the big migration from SDK to WebExtension was just such a slog. Dave made it happen — heroically — but it took a ton of work. We might have been better served holding the experiment for awhile and letting Min Vid ship later rather than writing the whole thing twice.
Focus on fewer things, and do them better together
This may be the biggest lesson of the first two years of Test Pilot. During this time, we shipped two major Firefox features, built an experiments platform, shipped more than a dozen large experiments, ran design sprints, and a university design course to boot. Our team has simply done too many things.
This hit me pretty squarely over the summer as I was deep in implementation details for Screenshots while managing the launch of Send, helping with priorities and metrics dashboards for Notes, and writing style code for Voice Fill at the same time.
In some ways, Min Vid has been a casualty of our team’s overeager schedule. With Screenshots’ launch imminent last summer we had to delay design, engineering, and product management tasks on Min Vid for long stretches of time. We simply did not have the resourcing.
The fact that Min Vid stayed successful is an absolute testament to the hard work of Dave Justice who dove into one complex coding task after another, often with very little direct help from anyone on our team.
As Min Vid leaves Test Pilot and moves on to the next round of testing on Firefox users, the Test Pilot team is busy prepping the next wave of experiments with this lesson in mind. We’ll be shipping fewer new projects, but we will be significantly more attentive to each.
What happens next
Our users love Min Vid and our team loves Min Vid, but to do the feature right would mean closer consideration of core browser components like windowing and media playback. Because of this, taking the feature from Test Pilot to release Firefox in a maintainable, performant manner will require a complete rewrite of the project’s codebase.
The next step toward figuring out if the engineering effort will be worthwhile is to ship a version of Min Vid to a small population of Firefox users through the Shield experiments platform. For our upcoming Shield experiment we’ve built a pared-down Min Vid experience by removing the queue feature, limiting the experience to YouTube, and reducing the Telemetry we’re collecting to a bare-minimum of event collections. We’ll also display a small on-boarding pop-up for one of our experimental cohorts to inform users about the feature. In our Shield experiment, we want to learn whether Min Vid usage is covariant with increased Firefox usage, retention, or user-reported satisfaction.
Depending on the outcome of this experiment, we will work with the Firefox Product and Platform teams to define specific goals and use cases for a Firefox version of Min Vid. We’ll continue to provide updates on this blog.
Thanks for everyone who participated in this experiment. And thanks in particular to Dave Justice for keeping it alive for so long!
