How We Evaluated the Products in Mozilla’s *privacy not included Buyer’s Guide (2018)
By Rebecca Ricks and Janice Tsai
The goal of Mozilla’s *Privacy Not Included buyer’s guide is to help consumers shop smart — and safe — for products that connect to the Internet. Understanding privacy in today’s world of connected devices — especially the risks and implications to consumers, personally — can be difficult. How will personal data be used by the company? What options do consumers have around what they collect or how long they keep your information? What could go wrong? It can be very unclear and almost impossible to find out.
To help consumers better understand the privacy and security implications of a connected device, we looked into a wide variety of criteria and then tried to simplify them to make it easier for consumers to understand what they should be considering as they do their shopping.
Here is the methodology we used to develop this guide.
There are 70 products in the 2018 version of our buyer’s guide. These products fit into six categories — Toys & Games, Smart Home, Entertainment, Wearables, Health & Exercise, and Pets.
The goal was to select connected products that were likely to be popular during the holiday season and beyond. We selected products that were top sellers on Amazon Prime Day, products featured in the Target Open House, and products that were highly rated across a variety of consumer product websites such as Wirecutter, The Toy Insider, PC Magazine, Tech Radar, and Gear Brain.
We looked into a set of criteria for each product across five questions.
1) Can in spy on me?
This question looks at whether a device and the app that controls a device uses a camera, microphone, and location tracking. We note that just because something can spy on you, doesn’t me it will. It simply means it could and you should be aware of that.
2) What does it know about me?
3) Can I control it?
This question looked into whether a default password needed to be changed to a strong one, if the company pushes automatic security updates, can you ask the company to delete the data it stores on you, and if appropriate, does the product have parental controls.
4) Company shows it cares about its customers?
This question looked at whether a company has a system in place to manage security vulnerabilities on products once they are found. This includes a bug bounty program. We also looked into how easy a company made it for a consumer to contact customer support. Links to phone number, email, live chat, and Twitter are included for each product that has these methods of contact.
5) What could happen if something goes wrong?
We included this section to help people understand risk scenarios related to their privacy and each particular product. We aimed to identify risks that would feel relevant to consumers. It’s likely nothing bad will happen with most of the products in this guide. However, it’s also good to think through what could happen if something goes wrong. This question looks a potential worst-case scenarios for each product, in some cases for fun and in some cases based on things that have already happened with the product.
Minimum Security Requirements
Mozilla has established a set of minimum IoT security standards it has determined should be met by any manufacturer developing smart devices. For each product on our list, we tried to determine the answer to five fundamental questions:
1) Encrypted communications
The product must use encryption for all of its network communications functions and capabilities. This ensures that all communications are not eavesdropped or modified in transit.
2) Security updates
The product must support automatic updates for a reasonable period after sale, and be enabled by default. This ensures that when a vulnerability is known, the vendor can make security updates available for consumers, which are verified (using some form of cryptography) and then installed seamlessly. Updates must not make the product unavailable for an extended period.
3) Strong passwords
If the product uses passwords for remote authentication, it must require that strong passwords are used, including having password strength requirements. Any non unique default passwords must also be reset as part of the device’s initial setup. This helps protect the device from vulnerability to guessable password attacks, which could result in device compromise.
4) Vulnerability management
The vendor must have a system in place to manage vulnerabilities in the product. This must also include a point of contact for reporting vulnerabilities or an equivalent bug bounty program. This ensures that vendors are actively managing vulnerabilities throughout the product’s lifecycle.
5) Privacy Practices
Can it spy on me?
For you to set up your new device, you will probably need to download an app. Both of these (the device and the app) will most likely need data to make things work. The app will typically need to request permissions for it to access your more sensitive data.
In our ratings, we evaluated if the device or the app required access to the camera, microphone, or GPS location information. We evaluated the device based on the product website and we used the Google Play store for Android to check on the permissions requested by each app. (Note: apps may access “approximate” or “network” based location. “Can it track me” was marked as “Yes” if an app requests any location information, including approximate location.)
The Harry Potter Kano Coding kit teaches kids to code using a wand. The Kano Code app is used to interact with the wand. The wand itself does not have a camera, microphone, or GPS. The Kano Code app accesses the following permissions:
Location: approximate location (network-based)
Camera: not applicable
Microphone: not applicable
It’s “Can it Spy on me” ratings are the following:
What does it know about me?
2) Data sharing
How do companies collect & handle customer data? What is considered a reasonable level of “expected behavior” in terms of sharing data with third parties?
3) Deletion of data