Researching a Reputation System for Online Social Communities

Published in

THXprotocol

10 min readJan 16, 2020

THX is dedicated to unlocking the value of online communities; we reward healthy member engagement, making communities more sustainable, using an improved reputation system based on blockchain technology.

(Update 14 April : I successfully defended and published my thesis. Read the full paper “Evaluation of Reputation in the Context of Online Social Communities”)

I have conducted an extensive study on accurately evaluating and measuring the reputation of members in online social communities as part of my Master’s thesis for the department of Behavioural, Management and Social Sciences at the University of Twente. I gathered data while working at Open Social, an online community platform. You’re here to learn why online reputation is vital to the success of online communities and how one could develop a framework that accurately measures reputation. For my current team’s recent findings be sure to read the THX white paper.

Communities are everywhere, can you trust their members?

Think back to the last time you went online. Did you scroll down Instagram? Leave a review on Amazon? Share a meme in a Whatsapp group?

These are just some examples of communities that you rely on for social interactions; and the online social world, to be frank, extends much further than that. People use online spaces to build companies, communicate across borders, connect volunteers, share the pain of mutual suffering, collaborate creatively, and much more.

Online activity has leaked into all areas of our lives and it’s only natural for us to assume that we can do it safely. But can we?

Online communities promise its members an open, democratic space in which they can find people with similar interests and goals, and share and debate ideas in a safe environment.

It sounds simple enough but when looking closer at communities today, most fall short of standards required for safe, active online interactions. We regularly experience uncertain credibility and reliability, leading to issues like fraud, internet bullying, catfishing, fake news and unknowingly having our personal information sold to the highest bidders.

Well, no wonder we tread online with suspicion and mistrust. The lack of a system of trust not only results in low activity in online communities but in a threat to our online identities and well-being. It’s a fact: we need a better way of measuring, evaluating and displaying reputation online.

Let’s talk about online reputation

Offline, reputation is the opinion the public forms about an individual based on previous behavior and characteristics. It’s generated by the external opinions of others, and serves as accountability and trust. These concepts have been thoroughly researched in psychology, sociology, and philosophy, and we find them to vary greatly according to people, situations, and environment.

Using reputation online is, point blank, quite difficult. We have little personal information when we meet others on the internet and often, there are no previous interactions that we can use for evaluation. As a consequence we need to rely on a third instance like a reputation system, that collects data and displays it readily on the user’s profile.

Measuring reputation online is not a new concept; previous research and practical examples are commonly found in the e-commerce sphere in the form of ratings and reviews for feedback. What do you do before buying a product from Amazon? Read the reviews. These systems have worked successfully in evaluating the quality of a product or service but are difficult to adopt in a social environment where personal characteristics and actions are the focus of evaluation.

What we need is a reputation system that fairly builds a public opinion of an individual online and presents reputation in such a way that it can be used as an evaluative tool to improve trust. In order to design such a system, I first needed to know which constructs are involved in the evaluation of the concept reputation.

Keep reading to find out whether I was successful in finding underlying constructs for reputation.

Creating a reputation framework for communities

I conducted various qualitative experiments to discover whether I can find concepts that can be used to create a system to evaluate members of an online community.

Qualitative methods

Reputation is an abstract concept. It consists of underlying constructs that reflect reputation. How we process reputation in our brain is not completely known yet. However, every concept saved in our brain is a mental model. Ergo, mental models can be used to find the underlying constructs of reputation. The problem is that we are often unaware of our mental models. The known methods, used to elicit mental models are word association and card sorting; two techniques often used to evaluate abstract concepts, such as reputation, by discovering concepts that reflect the understanding and expectations participants have of a said topic.

Word association was used to discover which words people associate with reputation in the context of social online communities. The method involves providing a target word and asking participants to list words they associate with the target. In our case, the listed words included ‘fake’, ‘like’, ‘social’, ‘fame’, ‘followers’, ‘privacy’, and ‘addiction’, to name only a few. This first part of my study gave me an idea on what underlying constructs there might be. However, this method only gives you a number of constructs. The card sorting is needed additionally in order to find out if these constructions have an underlying structure.

Card sorting helped reveal the structure of the mental models people built for reputation. It’s a qualitative method also suited to build website navigation structures, for example, because one learns how participants organize information. I provided participants with a set of words (obtained from our word association study) and asked them to sort these words into meaningful groups. I analysed the collected data with the Jaccard coefficient in order to obtain a similarity matrix that presents which constructs belong together in an unsorted way. Next I used a vector analysis to analyse and sort the overall score table. The results are presented in a dendrogram and heatmap, both presenting which constructs belong together in one cluster.

Exciting results (summarized)

All in all I conducted three card sorting studies. The first one was to get an overall idea of underlying constructs and their structure. The results suggested that there might be two reputation domains mixed together: automated and peer to peer.

Automated reputation system: a system that collects data related to reputation from individual users (for example, number of posts) analyses it, and displays it in an understandable manner to the rest of the community.

Peer-to-peer reputation system: a (more subjective) system based on peer evaluations. The feedback is collected and processed, and a final value is calculated to represent the subjective reputation of an individual.

I decided to use the list obtained in the word association study, sort the words into the two domains and conduct two card sorting studies separately, one for every domain. The results were very insightful. For both domains I found a number of clusters and subclusters. I compared the clusters from the heatmap and the dendrogram and created a tentative cluster structure on the basis of both of them. In the following I want to show you how the heatmap and dendrogram for the automated reputation domain looked like and I want to present the tentative cluster structure.

The heatmap presents the distances between the different constructs. Red means that there is a very strong association between two constructs and yellow means that there is a weak association. The black rectangles mark possible clusters. It can be seen that there are a few bigger clusters that have an additional cluster inside of them. There are 12 clusters and four subclusters.

The dendrogram just like the heatmap represents distances between construct. Additionally the dendrogram also shows how far the different clusters are apart from each other. The darker grey rectangles underline the 9 clusters.

The tentative cluster structure was created on the basis of the clusters obtained in the heatmap and the dendrogram. The tentative cluster structure presents a possible structure for the constructs of reputation. Blue are groups indicated by the dendrogram and green shows groups obtained by the heatmap. The structure served as a basis for the categories and subcategories — serving as a basis for an automated reputation system — that were formed later on. Some elicited categories included: Participation (involvement, social engagement), Attitude ( individual, character trait, etc.) and Achievements (badges, rewards, etc.).

(Insightful) conclusions

Previous research and real-life examples have already proved that measuring online reputation is possible, however, we still need a system specifically developed for online communities.

Our conclusions pointed to a combination of two types of reputation systems; an automated system and a peer-to-peer system. Say what? Here are the definitions:

The results of card sorting studies helped understand which aspects of an individual should be measured by these systems; or in others words, which behaviors are important and valid indicators of someone’s reputation.

Is it the amount of posts? How about the amount of likes someone receives on a blog article? This is as important to understand as the method of collecting, evaluating and presenting the final values of reputation. The categories and subcategories conducted from the tentative cluster structure presented above, can be used to build different kinds of automated and peer to peer reputation systems. In our study we provide an example of how a peer to peer and an automated system could work and look like.

In this article we want to give you some insights into how an automated reputation system could be designed.

Building blocks for an automated reputation system

These are the categories I identified through my research (see tentative cluster structure above) for measuring reputation for an automated reputation system.

The subcategory socialising is used as an example to explain the data collection process in more detail. First data is collected on how active a user was in his network and how many active connections there were. This is measured frequently. It is important to notice that only active networks and connections should be taken into consideration. The next step is for the system to calculate an overall score for the subcategory. Additionally the system goes through the found data like posts and comments and detects positive and negative connotations. In other words, there will be a scale later on where another can see how social the other person is right now in the community and also if the person is social in a positive or negative way. Let’s take a look at how such a system could be displayed on the profile of the user.

On the basis of the reputation categories and subcategories found in my study I designed a prototype of how a reputation system could look like:

Here you can see that the user in this case Jane Smith is pretty active in the community in general. She seems to have a big reach and a lot of achievements. However, you can also see (Participation and Content Creation) that while she is very active she might be involved negatively in the community often too.

All in all, my findings help one understand which systems can be used to measure reputation, and how these systems accurately reflect the way we think about social reputation. However, it has yet to be proven that these will work in practice. Regarding future research and the implementation of such systems, we need to be careful to create systems that are conform with data privacy of users and ensure safety rather than increasing danger. In the future one or more of the categories can be used when creating reputation systems.

An online reputation framework for a better future

THX will provide an ecosystem for online communities that encourages healthy behavior. And one of the ways it will do so, is by solving uncertain. credibility and reliability.

A reputation framework, that allows users to fairly evaluate others and see whether they can be trusted and reliant, will help solve the uncertainty of social interactions online.

The framework should accurately reflect the idea individuals have about the concept reputation and be based on fair and representative information displayed on a user’s profile. This information is, in some cases, collected automatically and, in other cases, subjectively from other community members themselves.

Looking to the (near) future; privacy and reputation

My work isn’t done, obviously. Further research needs to be done to test the validity of these reputation systems in real online communities. In addition, we need to ensure data privacy for the members of online communities. When we measure reputation, we are collecting and displaying data about individuals. Therefore, ensuring that we are not interfering with the online privacy that everyone deserves is a must and a key focus in our efforts.

Our next steps with the research on online reputation systems for THX are as follows:

Evaluate the upcoming DIN spec 4997 by the ‘privacy by blockchain design’ research group
Implement a reputation system minimal-viable-product (MVP)

Are you a community manager and want to know more about reputation systems for user engagement and safety? Or do you want to get a more detailed look on my study? Feel free to click on this link to read my full thesis: “Evaluation of Reputation in the Context of Online Social Communities”.

Do you have any questions, comments or suggestions? We welcome feedback! Leave a comment below or contact me directly for more information.