Data from the People, for the People: Consumer Reports and CAT Lab’s Distributed Research Partnership

Over the last few years, we at Consumer Reports have been scaling up testing of digital products and services, while also conducting investigations. But Consumer Reports’s traditional model of comparative testing that we have been using for more than 80 years is not always going to be the most effective lever for change for digital consumer goods.

The marketplace for digital products and services is not only large and quickly changing, but also highly contextual. Whether browsing the web, requesting a ride via an app, or shopping online, data about you is being used to present different options, prices, advertisements, or opportunities to you. Since many of these services include a lot of data collection, some of the issues that we wish to track are just not going to be visible or applicable to all users.

We have traditionally done a lot of distributed research, like panel surveys, secret shoppers, story collection, and feedback on how reliable or satisfactory the products we test are; but we need to scale up this work in order to meet the market-wide challenge that connected products and services present. To that end, we have been working to leverage our membership base to drive our testing and advocacy as well through distributed data gathering. For example, we worked with volunteers in 2018 and 2019 to identify a missing Facebook setting that controls the use of facial recognition technology on the photos you post to the platform. By working with our members, we revealed that although the majority of users got the new Face Recognition setting, around 25% of users did not. This work led to a Consumer Reports article on Consumer Reports and a complaint to the Federal Trade Commission. The Face Recognition setting issue was one of the claims made by the FTC in their historic five billion dollar settlement.

We also worked with our members to gather over 800 cable bills, which led to our 2019 Cable Bill Report, documenting that consumers pay on average $450 per year in company-imposed extra fees. We began a new project called “You Test” in order to get quick feedback and information from consumers at the end of some articles. The information gathered from these You Tests are then fed back into the organization to inform our work. We also work with our members to help effect regulatory and market change through traditional advocacy efforts like delivering over 180,000 petitions to Equifax in the wake of their breach.

But we need to keep innovating, and figure out how to empower our members and community to reveal patterns and invisible harms in online life. And luckily, the Citizens and Technology (CAT) Lab is an ideal partner for this work. CAT Lab, a research group at Cornell University, works with the public to study the effects of technology on society and test ideas for change to better serve the public interest. They have been thinking through how to test the impacts of digital technologies through participatory studies for a long time.

Our partnership with CAT Lab to better engage our members in our efforts to ensure that the marketplace is safe and fair. By working with volunteers, we can observe real versus hypothetical effects, document disparate impact and personalization, precisely estimate product rates of behavior, and test social and behavioral promises. This distributed research will help Consumer Reports produce better ratings of connected products and services; inform our investigatory reporting; and support our advocacy efforts to protect consumers and their privacy.

Our first project of many we hope to undertake will use both organizations’ strengths in independent research to examine the privacy benefits of different browsers. So watch this space and check out Consumer Reports’s new Digital Lab and CAT Lab’s new homepage.




Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Katie McInnis

Katie McInnis

Katie McInnis serves as Senior Public Policy Manager US at DuckDuckGo, an internet privacy company.

More from Medium

Xenobots! Holy SH***T!

TIF 2.0: Through the participant’s eye

The fascinating world of Voronoi diagrams

The Failures of Thomas Alva Edison