This AI Predicts Online Trolling Before It Happens

How do you keep online trolls in check? Ban them? Require real names?

, a post-doctoral research fellow in computer science at Stanford University, is developing an AI that predicts online conflict. His research uses data science and machine learning to promote healthy online interactions and curb deception, misbehavior, and disinformation.

His work is currently deployed inside Indian e-commerce platform Flipkart, which uses it to spot fake reviewers.We spoke to Dr. Kumar ahead of aon healthy online interactions at USC.

In my research, I build data science and machine learning methods to address online misbehavior, which transpires as false information and malicious users. My methods have a dual purpose: first, to characterize their behavior, and second, to detect them before they damage other users. I have been able to investigate a wide variety of online misbehavior, including fraudulent reviews, hoaxes, online trolling, and multiple account abuse, among others.

I develop statistical analysis, graph mining, embedding, and deep learning-based methods to characterize what normal behavior looks like, [and] use this to identify abnormal or malicious behavior. Oftentimes, we may also have known examples of malicious behavior, in which case I create supervised learning models where I use these examples as training data to identify similar malicious entities among the rest.

The key problem that I helped address on Flipkart was of identifying fake reviews and fake reviewers on their platform. This is a; recent surveys estimate as much as 15 percent of online reviews [are] fake. It is therefore crucial to identify and weed out fake reviews, as our decision as consumers is influenced by them.

My method, which is called REV2, uses the review graph of user-review-product to identify fraudsters [who] give high scoring ratings to low-quality products or low scoring ratings to high-quality products. REV2 [compares] our recommendations to previously identified cases of fake reviewers.

It is possible to proactively predict when something may go wrong by learning from previous such cases. For instance, in my recent research, I showed that it is possible to accurately predict when one community in the Reddit online platform will attack/harass/troll another. This phenomenon is called “brigading,” and I showed that brigades reduce the future engagement in the attacked community. This is detrimental to the users and their interactions, which calls for methods to avoid them. Thus, I created a deep learning-based model that uses the text and community structure to predict, with high accuracy, if a community is going to attack another. Such models are of practical use, as it can alert the community moderators to keep an eye out for an incoming attack.

Absolutely! A natural and exciting follow-up work is how to discourage bad actors to do malicious acts and to encourage everyone to be benign. This will help us to create a healthy, collaborative, and more inclusive online ecosystem for everyone. There are many interesting challenges to achieve this goal, requiring new methods of interventions and better prediction models. Enabling better online conversations and nudging people to be their better self is going to be one of my key thrusts going forward.

One of the major reasons for me to follow this direction of research was seeing some of my friends being harassed by social media trolls. This led to look for non-algorithmic ways to curb this problem. Being a challenging task, it piqued the interest of the scientist inside me and I eventually learned to create data science and machine learning methods to help solve these problems.