Utilizing SingularityNET’s Reputation System to Prevent Scams

An exploration of the effectiveness with which SingularityNET’s reputation system can be utilized to prevent scams on digital marketplaces.

Introduction

The rise of digital marketplaces has served to drastically enhance the options available to consumers. No longer constrained by the limitations of their local stores, consumers can explore and choose from a nearly unlimited catalog of items and services.

The ability to choose from a variety of products, services, brands, and suppliers, has left the consumer with several questions. Most importantly: which product (or service), supplied by which supplier, at what price, with what delivery option, best fulfills their needs?

To address this question, digital marketplaces often use rating systems coupled with recommendation engines to aid consumers in their decision making. These novel mechanisms employed by digital marketplaces have fundamentally changed consumer shopping behaviors.

However, the widespread utilization of services that game the recommendation engines and rating systems has created considerable doubt about the veracity of the information being presented to the consumers. Recently, several scam campaigns involving financial incentives for the creation of fraudulent reviews have come to light. These campaigns affect up to an estimated 61% of reviews within certain product categories on Amazon.

In one of our recent publications, we conducted an in-depth study on the behavioral changes in digital marketplaces participants, to help develop a reputation algorithm that can perform more effective product recommendations.

In this blog post, we analyze the current state of different types of rating systems, explore how effectively our current reputation system distinguishes between honest and fake reviews, and discuss the development of SingularityNET’s reputation system that will, among other things, help prevent scams in digital marketplaces.

Weighted Liquid Rank Reputation Algorithm

The gaming of ratings and reputation systems in e-commerce platforms is a large and challenging issue. Various suppliers that are offering low-quality products and/or services, enhance their reputations by using groups of fake consumers. We refer to this kind of behavior as a scam.

We anticipate that the prevention of reputation gaming can be achieved with an advanced reputation system, such as the one that SingularityNET is developing to enable Liquid Democracy for Distributed AI Systems. In particular, we have provided the Reputation System Design for SingularityNET and explored applications of this design for specific marketplaces being affected by different sorts of reputation gaming attacks.

Our reputation system accounts for numerous factors, such as:

• time liquidity (in which reputation transfers throughout time and reputation scores from earlier times influence future scores);
• reputation flows from raters (buyers) to ratees (suppliers) (so that raters influence the reputation of products as well);
• time on the market (how long the supplier or consumer is staying on the market);
• rates of spending flow.

The algorithms powering our reputation system can be configured to suit different marketplaces, since they each have different market conditions and dynamics, to optimally assign the reputation to participants. Due to this liquid and flowing manner in which our reputation system assigns and transfers reputation, we call it the weighted liquid rank reputation algorithm.

Market Simulation System

To explore the mechanisms for preventing and resisting “scam campaigns” — in which providers of fake or very low-quality services attempt to fool buyers by manipulating ratings through fake ratings and reviews — the SingularityNET team created a market simulation system, and a reference implementation of the reputation system, both of which are available as open-source software.

We used the market simulation system to emulate real-world marketplaces with multiple buyers and sellers.

Some of the emulated buyers and sellers were considered honest. The sellers offer high-quality goods and services, and the buyers provide accurate ratings for these goods and services.

Other emulated sellers were asserted to be dishonest: they offered low-quality goods and arranged for temporary communities of fake buyers to provide (in a massive amount) highly positive ratings for those goods and services. They were simulated to act in campaigns so once the community discovers the scam and downvotes, bans or evicts the scammers, they have to close the campaign under current identities and start another campaign under new identities sometime later.

The simulations were run with different durations of typical scam campaigns and various configurations of our reputation system were employed for the prevention of the scam campaigns on the digital marketplace. These configurations were:

• No reputation system: participants are making decisions relying only on their memories and not referring to any reputation system.
• Regular reputation system: Accounts for only the plain ratings made by any buyer to any seller, without factoring their history, amounts, or purchases.
• Weighted reputation system: The computation of reputation ranks of sellers is based on the ratings made by buyers, with accounts to financial volumes of the buys associated with specific ratings.
• TOM-based reputation system: The computation of reputation ranks of sellers is based on the ratings made by buyers with accounts to “time on the market” (TOM) that the buyers spend on a given marketplace so that the ratings issued by newly created accounts (including fake ones) are given less value.
• SOM-based reputation system: The computation of reputation ranks of sellers is based on the ratings made by buyers with accounts to “spending on the market” (SOM), based on the cumulative spendings made by the buyers providing the ratings, so that the ratings issued by fake accounts that do not make any real purchases and spend only on short-term scam campaigns are making less impact on the reputation of the sellers.

Scam Prevention

To evaluate the performance of the reputation system against different scam campaigns, we came up with the following two financial metrics:

• Loss To Scam (LTS): the sum of the buys made by all honest buyers from the dishonest sellers, divided by the sum of all buys made by honest buyers. This metric shows what proportion of money spent by honest buyers is spent on the products offered by dishonest sellers.
• Profit From Scam (PFS): the sum of the buys made by all honest buyers from dishonest sellers, divided by the spendings of dishonest (fake) buyers aimed at gaming the reputations of dishonest sellers. In other words, the return on the money spent by dishonest buyers on their reputation gaming campaign, showing how profitable it is to run fraudulent campaigns in our marketplace.

We have performed the simulations for markets with different durations of reputation gaming “scam period”, such as 10, 30, 92 and 182 days.

Table 1 shows how our financial metrics perform depending on the type of reputation system configuration for every period.

Table 1. The dependency of financial metrics on “scam period” and type of reputation system configuration. The right-most columns “LTS Relative Decrease” and “PFS Relative Decrease” illustrate the performance of different reputation systems under different scam periods. “LTS Relative Decrease” is the relative increase of loss to scam in relation to having no reputation system. Similarly, “PFS Relative Decrease” is the relative increase in profit from scams in relation to having no reputation system.

We can see that for any scam period, regular reputation system does not provide any improvement but rather makes things worse by making the LTS and PFS greater. In turn, weighted reputation system provides stable improvement, decreasing loss to scam and profit from scam. Finally, we see the best improvement in cases of TOM-based (assessing raters’ reputation with time on the market) and SOM-based (assessing raters’ reputation with spendings on market) reputation systems with shorter scam periods.

It is clear that if no reputation system is used, then the losses of those agents to scam and the profits of scammers increase when the scam period is shortened. On the other hand, weighted reputation system always shows improvement — increasing with shorter scam periods. The performance of TOM-based and SOM-based systems for longer scam periods of 182 and 92 days exceeds that of the weighted reputation system. We also note that these systems improve significantly with decreases in the scam period and provide the best improvement across all scenarios with the shortest scam period of 10 days, making losses of honest consumers and profits of scamming suppliers nearly 10 times smaller.

The capability of the reputation system to discriminate between honest suppliers providing generally real services from dishonest suppliers charging money for fake services is illustrated in Figure 1 below.

Figure 1. Discrimination of dishonest and honest suppliers based on their reputation ranks using TOM-based reputation system configuration. The vertical axis represents reputation rank in a range from 0 to 100 while the horizontal axis represents market suppliers, dishonest in red on the left side and then honest in blue on the right side.

In Figure 1, multiple dishonest suppliers are shown — including all aliases being active across all scam campaign periods. Every alias of dishonest supplier agents is represented separately. It is observed that all dishonest suppliers except one are found below the reputation rank threshold of 50, while most of the honest suppliers, with few exceptions, are found at a much higher rank.

We conclude, therefore, that by using our TOM-based and SOM-based weighted liquid rank algorithm reputation systems, online marketplaces can better allocate products and prevent scams.

Product purchasers will lose significantly less money, on average, than they would without any reputation system. Just as importantly, the reputation system makes the marketplace prohibitively less efficient for scammers and the scams are much less profitable.

In a decentralized marketplace with no centralized service for scam prevention, it will initially be necessary for buyers to spend some money on scam sellers, because at the beginning there will be no way of knowing whether someone is selling good or bad product — only after some feedback from buyers and transactions will we be able to take appropriate action to allocate recommendations better.

What’s Next?

For a greater level of detail on our exploration and a more detailed conclusion please read the full version of our analysis.

You can also meet with the SingularityNET team during the poster session of the AI for Social Good Workshop at the IJCAI-2019 conference in August 2019.

If you would like to learn more about SingularityNET, we have a passionate and talented community which you can connect with by visiting our Community Forum. Feel free to say hello and introduce yourself here. We are proud of our developers and researchers that are actively publishing their research for the benefit of the community; you can read the research here.

For any additional information, please refer to our roadmap and subscribe to our newsletter to stay informed about all of our developments.