Moderating Online Content by Highlighting High-Quality Comments
How Highlighting High-Quality Comments Could be an Effective Online Moderation Strategy
If newsrooms didn’t moderate their comments, many of the comments would be rude, biased, misleading, and trolling. However, when discussing comment moderation strategies, most people think only of excluding problematic content. For instance, commenters can flag and report problematic comments so that other commenters don’t see and interact with these comments.
In addition to filtering improper comments, professional moderators and editors at the New York Times (NYT) also moderate the comment sections by highlighting high-quality comments, judged as “the most interesting or thoughtful,” and giving these comments a Times Pick badge.
These orange badges can signal desired content so that commenters are less likely to misbehave. Upon receiving a badge, or “the gold medal of the commenting Olympics,” one super commenter mentioned felt:
“the same pride I feel when I’ve done a particularly good job building something.”
But what exactly is the impact of this moderation strategy on the comment section?
Inspired by this question, we analyzed the correlation between NYT Picks and commenters’ behaviors in our recent paper. Specifically, our results show that NYT Picks are correlated with an improvement/increase in
- First-time receivers’ next approved comment quality
- Newcomer receivers’ commenting frequency
- Observers’ comment quality
To come to these conclusions, we collected and analyzed more than 13 million approved NYT comments (~2.5% were NYT Picks) via the NYT Community API from 2007 to 2015. To quantify comment quality, we defined a comment’s quality score as the probability of being selected as an NYT Pick and used a machine learning model to predict the score. We then validated our model’s scores by comparing them against human ratings of comment quality.
Apart from machine learning models, we used Propensity Score Matching so that our observational study mimicked a randomized controlled experiment. Using comment quality scores, we matched Picks and comments of the same comment quality score that were not picked, to compare the behavior difference between users who received a Pick (treatment group) and users who maybe should have received a Pick but didn’t (control group).
Let’s walk through some of the questions we answered using these data and methods.
Do commenters receiving NYT Picks improve their comment quality?
TL;DR: Yes.
In the figure below, we plotted the median comment quality before and after receiving Picks between users who received Picks and users who should have received Picks but didn’t. Order 0 corresponds to the first Picked comment.
Pick with 95% Confidence Intervals highlighted in dark gray lines. Order zero corresponds to the (matched) Picked comment. For every order, the data contains all commenters who commented at that order.
The figure shows that commenters receiving NYT Picks did improve their future comment quality (see the blue line), compared to the control group (see the orange line). After receiving the first Pick, users significantly improved the quality of the next comment. The quality then steadily decreased until the sixth comment after the Pick, when it reached a level similar to the quality before the Pick.
Do commenters receiving NYT Picks increase their future commenting frequency?
TL;DR: Yes, for the newcomers.
Our findings show a general trend of comment intervals that decrease after receiving Picks. To determine whether the decrease is natural or a result of the Picks reception, we calculated the gradients of comment intervals for every user. For the first two comments, receiving Picks made the gradients more negative than not receiving Picks. This result suggests that receiving Picks during early tenure on the site (i.e., for their first two comments) may motivate commenters to return to the section more quickly to make their next comment.
scores with users who didn’t receive Picks at the order, with 95% Confidence Intervals shown as dark gray lines.
Do commenters observing NYT Picks improve the comment quality of replies to those Picks?
TL;DR: Yes, but the impact from receiving a Pick is smaller than the impact from the parent comment quality.
To examine whether reply quality is simply associated with the generally higher quality of parent comments that are Picks or if the visibility of Pick badges may also play a role, we built a generalized regression model (GLM) to predict reply quality. We noticed the coefficient of the Parent Pick is close to but larger than zero and less than the coefficient of the parent comment quality score. This result suggests that observing Picks may positively impact repliers’ comment quality, though less than the impact from the parent comment quality.
How can we better moderate and design future comment sections?
Our findings emphasize the importance of reinforcing and maintaining comment sections’ positive commenting behaviors. We believe that the investment in professional moderation of news comments could improve online discourse, and highlighting high-quality comments is a sound moderation strategy for comment sections.
Based on our findings, there are several design opportunities comment moderation communities can consider:
- Because users tend to increase their comment quality after receiving their first Pick and return to the comment section more quickly in their early stage of commenting, designers might try:
Highlighting high-quality comments from users who haven’t received any Picks in their early stage of commenting.
2. Because users tend to increase their comment quality and frequency after receiving Picks, designers might try:
Sending notifications to users to return to the comment section after they receive Picks.
3. Because users’ reply comment quality is positively correlated with the parent comment quality, designers might try:
Ordering the comments in the interface from high-quality to low-quality for users.
We hope these design opportunities inspire newsrooms and comment moderation communities to rethink the moderation process and the comment section design. We also welcome your comments on how to encourage positive and constructive online discourse.
Please check our full paper for more technical details:
Wang, Y., & Diakopoulos, N. (2022). Highlighting High-quality Content as a Moderation Strategy: The Role of New York Times Picks in Comment Quality and Engagement. ACM Transactions on Social Computing (TSC), 4(4), 1–24. https://dl.acm.org/doi/10.1145/3484245
