How We Evaluated the Products in Mozilla’s *privacy not included Buyer’s Guide (2019)
By Rebecca Ricks and Janice Tsai
Updated Feb. 6, 2019, for Valentine’s Day.
The goal of Mozilla’s *Privacy Not Included buyer’s guide is to help consumers shop smart — and safe — for products that connect to the Internet. Understanding privacy in today’s world of connected devices — especially the risks and implications to consumers, personally — can be difficult. How will personal data be used by the company? What options do consumers have around what they collect or how long they keep your information? What could go wrong? It can be very unclear and almost impossible to find out.
To help consumers better understand the privacy and security implications of a connected device, we looked into a wide variety of criteria and then tried to simplify them to make it easier for consumers to understand what they should be considering as they do their shopping.
Here is the methodology we used to develop this guide.
Methodology
Product Selection
There are 70 products in the 2018 version of our buyer’s guide. These products fit into six categories — Toys & Games, Smart Home, Entertainment, Wearables, Health & Exercise, and Pets.
The goal was to select connected products that were likely to be popular during the holiday season and beyond. We selected products that were top sellers on Amazon Prime Day, products featured in the Target Open House, and products that were highly rated across a variety of consumer product websites such as Wirecutter, The Toy Insider, PC Magazine, Tech Radar, and Gear Brain.
Criteria
We looked into a set of criteria for each product across five questions.
1) Can in spy on me?
This question looks at whether a device and the app that controls a device uses a camera, microphone, and location tracking. We note that just because something can spy on you, doesn’t me it will. It simply means it could and you should be aware of that.
2) What does it know about me?
This question looks into whether a product uses encryption, what reading level the privacy policy is, and whether the company shares data with third parties for unexpected reasons such as marketing and advertising purposes.
3) Can I control it?
This question looked into whether a default password needed to be changed to a strong one, if the company pushes automatic security updates, can you ask the company to delete the data it stores on you, and if appropriate, does the product have parental controls.
4) Company shows it cares about its customers?
This question looked at whether a company has a system in place to manage security vulnerabilities on products once they are found. This includes a bug bounty program. We also looked into how easy a company made it for a consumer to contact customer support. Links to phone number, email, live chat, and Twitter are included for each product that has these methods of contact.
5) What could happen if something goes wrong?
We included this section to help people understand risk scenarios related to their privacy and each particular product. We aimed to identify risks that would feel relevant to consumers. It’s likely nothing bad will happen with most of the products in this guide. However, it’s also good to think through what could happen if something goes wrong. This question looks a potential worst-case scenarios for each product, in some cases for fun and in some cases based on things that have already happened with the product.
Minimum Security Requirements
Mozilla has established a set of minimum IoT security standards it has determined should be met by any manufacturer developing smart devices. For each product on our list, we tried to determine the answer to five fundamental questions:
1) Encrypted communications
The product must use encryption for all of its network communications functions and capabilities. This ensures that all communications are not eavesdropped or modified in transit.
2) Security updates
The product must support automatic updates for a reasonable period after sale, and be enabled by default. This ensures that when a vulnerability is known, the vendor can make security updates available for consumers, which are verified (using some form of cryptography) and then installed seamlessly. Updates must not make the product unavailable for an extended period.
3) Strong passwords
If the product uses passwords for remote authentication, it must require that strong passwords are used, including having password strength requirements. Any non unique default passwords must also be reset as part of the device’s initial setup. This helps protect the device from vulnerability to guessable password attacks, which could result in device compromise.
4) Vulnerability management
The vendor must have a system in place to manage vulnerabilities in the product. This must also include a point of contact for reporting vulnerabilities or an equivalent bug bounty program. This ensures that vendors are actively managing vulnerabilities throughout the product’s lifecycle.
5) Privacy Practices
The product must have a privacy policy that is easily accessible, written in language that is easily understood and appropriate for the person using the device or service. Users should at minimum be notified about substantive changes to the policy. If data is being collected, transmitted or shared for marketing purposes, that should be clear to users and, as in line with GDPR, there should be a way to opt-out of such practices. Users should also have a way to delete their data and account. Also in line with the EU’s General Data Protection Regulation (GDPR), this should include a policy setting standard retention periods wherever possible.
Can it spy on me?
For you to set up your new device, you will probably need to download an app. Both of these (the device and the app) will most likely need data to make things work. The app will typically need to request permissions for it to access your more sensitive data.
In our ratings, we evaluated if the device or the app required access to the camera, microphone, or GPS location information. We evaluated the device based on the product website and we used the Google Play store for Android to check on the permissions requested by each app. (Note: apps may access “approximate” or “network” based location. “Can it track me” was marked as “Yes” if an app requests any location information, including approximate location.)
For example:
The Harry Potter Kano Coding kit teaches kids to code using a wand. The Kano Code app is used to interact with the wand. The wand itself does not have a camera, microphone, or GPS. The Kano Code app accesses the following permissions:
Location: approximate location (network-based)
Camera: not applicable
Microphone: not applicable
It’s “Can it Spy on me” ratings are the following:
What does it know about me?
In addition to those five fundamental questions, we also evaluated the privacy policy for each company selling these products to determine (1) its readability; (2) its policy on data collection & sharing; and (3) whether or not users could request deletion of their data.
1) Privacy
If a company had a privacy policy or privacy FAQs that specifically addressed how data from the app and/or device was handled, then we marked this as “yes.” If the privacy policy only applies to the company’s web properties then it still passes this standard, but with the caveat that we cannot draw any strong conclusions about the product.
At a fundamental level, how easy is it to understand each company’s privacy policy? How readable is the policy to the average consumer? Privacy policies should be clear, readable, and communicate basic information to consumers about what happens to their data.
To evaluate readability, we used a tool developed by engineers at Carnegie Mellon University called Usable Privacy Policy Project , which builds on advances in machine learning and natural language processing to annotate and evaluate privacy policies. We used this tool to assign a readability “score” to each privacy policy, with reading levels ranging from middle school to college.
2) Data sharing
How do companies collect & handle customer data? What is considered a reasonable level of “expected behavior” in terms of sharing data with third parties?
In many cases, companies rely on third party vendors for services such as payment processing or fraud detection. If a company’s privacy policy appeared to share data with third party analytics services or ad networks, we flagged that behavior as “shares data with third parties for unexpected reasons.” Further, if the privacy policy states that the company sells anonymized data for marketing or advertising, then it does not meet that standard. This standard was aimed at highlighting the ways in which privacy policies condone questionable data sharing practices.
3) Deletion of data
If a company’s privacy policy included language suggesting that customers could request their data be deleted, in line with the GDPR’s Right to Erasure (“right to be forgotten”), then we marked this as “Yes.” If companies stated in communication with us that they could delete user data, then we also marked this as “yes.”
