Rating system for online marketplace
Thoughts on scale, binary and comment ratings
Marketplaces require rating system to evaluate the quality of the products/services listed as well as their providers. Most of the popular marketplaces around are using a variation or combination of scale (star, slider), binary (like/dislike) and written reviews. This post is my thoughts on each of the rating system and how they should be used.
Scale rating (5 star rating)
5 star rating can be found everywhere. It is a standard rating scale that’s being used by popular marketplace sites like Amazon and most app stores. Because this system is so common,it is intuitive to average users in terms of how they interpret and cast the ratings (clicking on desire number of stars). Visually, star ratings are very appealing and noticeable compare to number or text (number of reviews), therefore it is easy for users to scan and compare a list of ratings at once. Lastly, scale rating is extremely powerful and flexible because you can obtain feedback with great detail by increasing the number of scale level.
While it became almost a design pattern on marketplace sites, some designers argue that star rating is bad and ambiguous because users have different interpretation of each scale level and most of the time users either like something or they hate it. People also like to mentioned that YouTube switched from 5 star rating to a binary system, however it only demonstrated that their users were rating their videos in a binary way therefore it’s logical for them to convert to a binary system and not necessary saying that binary rating is better than scale rating. Despite the flaws mentioned, star scale rating is extremely effective and should be used unless your user can benefit from a binary setup.
Binary rating (Thumbs up/Thumbs down)
Text reviews are potentially the most useful reviews that can provide great insights regarding a product, service or person. However, we rarely see an online marketplace use only text reviews because it requires the users to invest their time and energy significantly in order to provide one (especially a high quality feedback). Therefore, a text review is almost always presented as optional and paired with either a scale or binary rating. In sum, definitely try to capture written reviews and provide helpful hints/directions to help them think and leave a feedback.
Some example of rating systems being used
Let’s look at a few popular (both product and service) marketplaces and their rating system.
Amazon uses a combination of 5 star rating with comment reviews on individual products as well as sellers. On top of that, Amazon uses a binary rating system to rate the reviews so that they can have a self-sustained and balanced review list. For a marketplace like Amazon, their rating scale is less ambiguous due to the high volume of ratings on products. In addition, the comment reviews turned out to be more useful than the 5 star rating and it appears that they have no issue obtaining a written reviews from users because they are so used to provide one. Therefore their rating system setup is certainly working well.
eBay appears to be using a 5 star ratings similar to Amazon. However, eBay’s star rating system is generated based on binary scores (positive, negative…well and actually neutral but let’s not be too technical) review counts and displayed in levels, which is reflected by stars. For instance (see screenshot above), a positive rating is equal to 1 point and users will have a gold star when they have between 10-49 points. As users obtained more points, their level increased and earn different color stars to reflect the quantity of their positive sales. In sum, eBay is using a binary model but converting that to a scale model. Of course eBay also has comment reviews but it is not being shown prominently as one of the primary quality metric for users to see.
One other rating system design for a marketplace is binary rating along with comment reviews. Take Etsy as an example back when they still use a binary system (They now use a 5 star rating following the steps of Amazon), they use a simple binary rating system to rate both the products and providers. By showing the percentage of positive reviews, it is sufficient for Etsy users to make a buying decision. If the users have further doubts, they can always dig into the review comments with an extra click or two.
oDesk, a service marketplace, utilizes a 5 star rating with comment reviews as their feedback system (See screenshot below). It is a fairly standard setup similar to popular product marketplaces like Amazon, however it appears that the star rating is very two-sided (see score breakdown below) because about 10% of their providers with ratings are below 3.0 stars while around 90% of their providers received a 4.0 star or above.
Therefore when it comes to helping the users compare and decide on which providers to work with, a 4-5 stars rating is enough to convince users that a provider is trust worthy and they’re safe to commit to a working relationship. It’s almost counter productive when users try to make decisions between 4, 4.5, and 5 stars providers since they are all considered to be good. In this case, due to how binary their rating is, oDesk is in a similar situation as YouTube was and perhaps they should convert to a binary rating system.
Another online service marketplace we can look at is TaskRabbit, they also use a combination of standard 5 star rating system with comment reviews but what’s notable is that they also include a level/badge (See screenshot below) system as part of the rating to aid users’ decision making process.
The level badge expresses the provider’s level of experience and it complements the 5 stars rating quite well. For a service marketplace it is also helpful to see how many tasks a provider has completed vs. the average rating because it better reflects the true quality metric. In addition, the 5 star rating on TaskRabbit is also leaning towards a two-sided metric but it might balance out with more reviews. One thing I noticed is that due to the simplicity of some of the services (tasks), they don’t necessary require a scale review. For instance, you probably don’t need to read good 24 reviews about a service provider when looking for a person to simply pick up and deliver a copy of today’s newspaper. Therefore, I’d argue that TaskRabbit’s rating system is working fine but it’s perhaps more complex than it should be.
Most online marketplaces use a combination of scale, binary rating and comment reviews based on their user needs. In order to figure out the ideal rating system design for your marketplace site, you need to consider how would the users rate your product/service and what quality metrics would help them make a purchasing decisions efficiently. As a follow up read, there is a very insightful Quora discussion on rating systems and what to pay attention when designing one.