Developing a scalable moderation solution

For many large online communities finding a scalable solution to the moderation and curation problem can be a tricky balancing act. Try to moderate every item of user generated content (UGC) and the costs soon spiral out of control. Resign to just removing bad content once it has been reported or an argument breaks out and you can cause damage to the community spirit, something that is so important for successful communities.

Here at DiscussIt we have been working with clients like the Washington Post to come up with an efficient strategy to tackle this problem.

The basic premise is to create a priority queue for the duty moderator, consisting of items we consider as high risk and likely in need of deletion or human intervention of some sort.

Tools in our arsenal

The DiscussIt platform pushes every item of UGC through a series of filters, that rank items for degrees of aggression, intelligence, sentiment and ‘spamminess’, along with a whole series of other tests and checks, such as a black list of banned words and a grey list of suspect words.

All of this metadata is added to the UGC item so that when it passes through the next stage of our pipeline, the rules engine, we now have a lot of item properties to use in the formulation of our rules.

One of the possible actions to perform when a rule is matched is the ability to add a marker to an item. We use this action to add a ‘priority_queue’ marker to any items that we want a human moderator to take a look at.

So now we can create any number of rules with varying levels of complexity that add items to our priority queue. For example below we have a rule that will add a priority marker to any item that has either been community flagged or has a grey word count greater than zero and an aggression score higher than 0.7 (or 70%).

Quantifying the efficiency of your queue

Once we have our priority queue, we can look at the ratio of approvals to deletions, with the aim of optimizing the rules used to build the queue. The chart below shows the volume of items going through the queue and proportions of actions taken upon them.

We can govern the volume of the items going through the priority queue, by adding more rules to add markers based on criteria such as low user reputation or low content sentiment and high aggression. The number of items matched per rule can be tweaked by adjusting the trigger points for the filters, reducing the aggression level to 60% for example would increase the number of items matched, whereas increasing it to 80% would decrease the matches.

Ideally we want the number of items being deleted from the queue to be greater than the number approve, as this indicates that we are pulling out the most contentious content from the UGC stream.

Automation vs Human Moderation

Whilst it would be great to have filters accurate and intelligent enough to fully automate the moderation process, we believe that this is still a way off. Moderation requires a light touch and is not an easy job. If we were to delete all the items placed in the priority queue automatically, they would only represent a small proportion of the content going through the platform and it might be tempting to think that the level of false positives is within tolerable levels, but even a small number of false positives can have a significant effect on the balance of trust and confidence between the members of the community and the company running it.

We therefore advocate the use of technology like our filters and rules as a means for augmenting human moderation, rather than a way of replacing it.


About DiscussIt

We are specialists in the management and exploitation of user generated content. We craft bespoke solutions for large online communities that streamline moderation, improve engagement and automate marketing.

We look to work closely with a small number of the largest online communities, helping them to prosper and grow. If you believe that you have a community that fits this description or have ambitions to make your community a major player please contact us to arrange a consultation.