Test Pilot No More 404s Graduation Report
Editors Note: This post originally appeared on the Firefox Test Pilot site in February of 2017. We’re migrating all of our graduation reports to Medium to make them more discoverable.
Last winter, some folks from the Test Pilot team got together with some folks from the Internet Archive and hatched a plan. On the Test Pilot side of things, we were busy building our platform and getting experiments out into the wild. Meanwhile, the team at the Internet Archive was prototyping an add-on to help users avoid dead ends on the Web by checking if they had archived versions of sites available in the Wayback Machine for users who encountered 404 errors.
We launched the No More 404s experiment with a few goals in mind. We know from user research that 404 pages are a persistent nuisance on the Web, and a No More 404s add-on seemed like a really sweet utility to help steer users toward their intended content. Additionally, the Test Pilot team wanted to find out how a partner organization like the Internet Archive could help bring testable features into Firefox.
Here’s how it went down
When we launched No More 404s, the user interface was simple and unobtrusive. If a participant visited a 404 on the Web — specifically if a site that returned a HTTP header code 404 — the No More 404s experiment would send a request to the Wayback Machine to see if the service had an archived version of the page. If one existed, No More 404s would drop a notification bar on the page with a link to it. Users could then visit the content archived in the Wayback Machine, dismiss the notification bar, or just continue browsing as usual.
Later on in the experiment we made a few changes to the add-on. First, we added in more common error codes associated with missing Web assets. Then we made a much larger change when we asked our design team to increase the visibility of the add-on. We wanted to see how these changes would affect user perception of the experiment.
Here’s what we learned
Both teams wanted to get a sense of how many of the URLs that display 404 errors today had pages archived in the Wayback Machine. We tracked total 404s encountered by Test Pilot participants (1,275,165) as well as results returned by the Archive (185,081). Over the course of the experiment, we averaged a result 14.5 percent of the time, but also observed that over the course of the experiment the daily average rose.
In addition to 404s, which are common on the Web, there are other error codes users encounter. For example, users may reach a page with a 408 error code when a server takes too long to respond. In early November, the Internet Archive team decided to add more error codes to the experiment. We hypothesized that adding additional error codes would cause the overall percentage of sites returned by the Wayback Machine to drop relative to total queries made by the experiment. However, the opposite was true: in the graph above there is a modest uptick in the percentage of valid returns in early November after we made the change.
In early December we updated the user interface for the experiment and featured an adorable animated GIF. Changing the user interface affected how participants interacted with the No More 404s experiment. Prior to the changes, the experiment was tucked out of the way, so we wanted to see what happened if we made it more flashy. We assumed users might dismiss the UI more frequently than before, and they did, but we were surprised to see that the rate at which users dismissed the pop-up increased sixfold.
We also observed an uptick in the percentage of returned results that correlates to these UI changes. Of course, correlation is not causation, and this uptick remains a mystery: changing UI should not have affected the relative rate of successful returns from the Wayback Machine.
We also wanted to know how often experimenters encountered 404 errors as they browse. While this experiment did not track total sites visited by our users (because of privacy concerns), we did keep tabs on how many times each browser hit a 404 each week. Throughout the life of the experiment, roughly 50 percent of participants encountered at least one 404 per week. Of those who encountered 404s, well over 25 percent did so at a rate of four or more a week!
Here’s what happens next
While we’re graduating No More 404s from Test Pilot, and will be moving the add-on to the Mozilla Add-ons store so it’s accessible to all Firefox users. We won’t remove it from users who have it installed. If you currently have the add-on installed, you don’t need to do anything. The add-on will continue to update as usual. The Internet Archive team will also be adding some new features so users can add content to the Wayback Machine directly through the add-on! On the Firefox side, we will promote No More 404s on the homepage of the browser.
Once the add-on is in its permanent home, our teams will continue to work together on studies that answer key questions, including:
- Do average Firefox users see value in the add-on?
- Are there ways we could use the Wayback Machine to retrieve content when a user enters a URL for which there is no current DNS record?
- Could we integrate the Wayback Machine with other tools for archiving Web content, like the Page Shot experiment that is currently part of Test Pilot?
In addition to the next steps we’ll be taking in Firefox, the Internet Archive Team recently shipped a version of the add-on for Chrome and Safari support is on the way! The new add-on adds some features that let users save pages directly to the Archive so that the community can contribute directly to the health and growth of the archive.
Thank you to all the Test Pilots who installed No More 404s, used it, and told us what they thought! We are grateful for your participation.
Want to try a new experiment? Visit Test Pilot!
This post was co-written by Mark Graham, Director, Wayback Machine, Internet Archive
