Which Rating System Is Best for Your Users?

Sofia Quintero
Apr 4, 2019 · 6 min read

In 2017, after more than a year of exhaustive testing and internal experiments, streaming entertainment giant Netflix officially abandoned its five-star rating system in favor of a much simpler system in which users now simply give Netflix content a thumbs-up or thumbs-down.

Netflix’s star rating system was always a little different from most of the other similar systems out there. For one, the star ratings that viewers were shown weren’t averages of ratings given by all Netflix users but were instead tailored to individual users’ preferences; when users saw a movie with four out of five stars, this rating was a prediction made by Netflix’s algorithms of what Netflix believed a user would rate the title, not an average of actual ratings submitted by users. The problem was that Netflix didn’t communicate this to users, giving them what some described as a false impression of a title’s quality.

In an interview with Variety, Netflix’s VP of product, Todd Yellin, said that Netflix’s new thumbs-based rating system received 200% more user ratings in A/B tests of the new system.

Many users were deeply unhappy with the transition in Netflix’s rating system, but this was hardly the first time a major entertainment service provider had incurred the wrath of its users for making such a change. In 2009, YouTube made similar changes to its platform when it realized that a majority of users only rated videos they loved, with far fewer viewers rating videos they disliked.

One of the biggest problems with binary rating systems is that, while they might work well for gigantic media properties such as Netflix and YouTube, they may not be as useful for other products. Despite this, many design decisions made by large companies are often mimicked by smaller startups and product teams, sometimes to the detriment of their users.

How Rating Systems Drive User Engagement

When designing a rating system, it’s all too easy for user-experience designers to overlook the many purposes of rating systems. Before you can decide on the right rating system for your users, it’s important to understand exactly what rating systems can help you do:

  • Algorithmic personalization: Rating systems are one of the most effective ways to personalize a user experience based on real behavior. This is especially true of products with sophisticated algorithmic personalization systems, like Netflix. The more frequently users submit ratings, the more accurate the system’s personalization becomes, resulting in a richer user experience.
  • Aggregation of decision-making: Aggregate rating systems are useful in helping users make decisions. This works particularly effectively for products that have either high or low degrees of personalization; a brand-new Amazon user, for example, will still benefit from aggregated ratings, even if they don’t receive personalized offers as a result. This type of rating system becomes even more useful for products with longer decision times.
  • Identifying value: Rating systems can empower users to highlight the value or benefit of certain content and improve the quality of content over time in a continual feedback loop. This quality can apply to products with and without sophisticated personalization features.

Only you and your team can decide which kind of rating system is right for your product. Depending on which of the factors above is most important to you and your users, you may find that a binary rating system will suit your needs. That said, you may find that a more nuanced rating system is required.

Star Rating Systems Can Help Users Make Complex Decisions

While most companies want to reduce the time it takes for a prospective customer to evaluate a product and make a purchasing decision, some types of goods and services have inherently longer evaluation times. In these situations, a star rating system might be more effective.

For example, a customer who is in the market for a new pair of shoes might spend hours reading hundreds of user reviews before ultimately settling on a new pair. Zappos is a great example of this principle in action. In addition to providing browsers with a five-star rating system, the online shoe retailer also includes more detailed product metrics, such as its “fit survey” percentage, a metric designed to give prospective shoppers even more information about a product.

Star rating systems are often accompanied by more in-depth qualitative reviews and other product information. This makes the consideration process significantly easier for prospective customers because it gives them a three-dimensional overview of a product. This is critically important for certain types of products — including shoes — that often benefit from in-person evaluation.

Qualitative Reviews with Stars Increase Engagement but Delay Action

If time per session is a valuable metric for your product, increasing complexity with qualitative review information and a star rating system can be an effective combination.

One example of this is the review website Yelp. Although users have long been able to book reservations at local restaurants and reserve seating at certain events via Yelp, it’s known primarily as a review site. As with many sites that utilize star rating systems, Yelp also includes a lot of qualitative information to help users decide whether to patronize local businesses. From this, we can infer that one of Yelp’s engagement metrics might be measured by the average amount of time users spend on the platform. This might be more useful to Yelp than, say, the conversion rate of users who look up a restaurant before making a reservation.

The more freedom users have in their reviews, the more useful they are to other Yelp users. Yelp has done a fantastic job of leveraging this principle to grow its user base significantly, and many Yelp power users take great pride in the utility of their reviews. This itself creates a greater sense of community and engagement, both of which Yelp has cultivated across its platform.

Conversely, reservation platform OpenTable opts for a much shorter path to activation. OpenTable wants users to take the shortest possible path from identifying a potential place to eat to conversion by making an online reservation. As such, OpenTable features a blend of star ratings and a thumbs-up/down system to provide users with as much information as possible without bogging users down with unnecessary distractions.

Personalizing Content with Thumbs-Up Ratings

Netflix’s transition to its simplified thumbs-up/down system was deeply unpopular with many users, but it made a lot of sense for the platform. From the outset, Netflix had used its star rating system differently than many comparable systems used by other sites, and, by its own admission, Netflix did a poor job of explaining how the system really worked.

But many platforms have utilized binary rating systems to great effect, particularly within the realm of algorithmic personalization. Online radio platform Pandora has long relied on its thumbs-up/down system to craft increasingly individualized playlists and stations for listeners, as has music-streaming titan Spotify.

Products that rely on sophisticated algorithmic personalization and curation functionality often benefit from a binary rating system. But simpler rating systems can also encourage users to provide feedback to create larger, more robust data sets that may help users make more complex decisions.

Rating Systems Don’t Have to Be Binary

So far, we’ve focused primarily on the pros and cons of binary rating systems and star rating systems, but these are far from the only options available to UX practitioners seeking to incorporate user feedback into their product experiences.

Online movie database IMDb, for example, relies upon a 10-point rating system to give titles in its vast database a little more nuance than perhaps a five-point system would allow. Rotten Tomatoes, another incredibly popular movie-rating website, favors its 100-point rating system, which allows for even greater distribution of titles’ ratings. Other sites that rely on user-generated content, such as aggregation platform Reddit and the marketing community GrowthHackers, opt for up-vote/down-vote systems in which users can “vote” to promote content they like and dislike.

As with virtually everything in UX, there’s often no “right” answer when it comes to user rating systems. There are just too many variables for a one-size-fits-all approach to soliciting and acting upon user feedback. That said, hopefully this post has given you some insight into not only the kind of rating system that’s right for your product and your users but also the advantages and disadvantages of these systems.

Originally published at blog.getenjoyhq.com on April 4, 2019.

Hungry for Insight

A community for product people leading change

Sofia Quintero

Written by

Founder and CEO at https://getenjoyhq.com/

Hungry for Insight

A community for product people leading change

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade