Grooming Detection Part. 2: A Rule-base Matching Solution

Photo by Asterfolio on Unsplash

In a previous blog post, we introduced grooming detection with a state-of-the-art review and some existing models and datasets. Today, we will present an example of a Rule-base matching solution to detect grooming in chat messages.

What is Rule-base Matching?

Rule-base Matching aims to detect words and linguistic patterns in text. Detecting turns of phrases with words and patterns is an effortless yet efficient and precise way to filter texts. This solution alone can already give great results for some use cases, but they are even greater combined with model training or another kind of automation.

It can be done with regular expressions, but regex can be very time-consuming if you have lots of patterns and different topics you want to detect. Lexicons, dictionaries, and Natural Language Processing (NLP) pipelines can be very handy in helping you match specific features very fast. Here, we use spaCy Matchers to catch words and patterns not only on their surface form but also on linguistic features kept in spaCy tokens: we mainly use POS-tags and lemmas, but morphological and syntactic traits are also available.

Task Definition

Before developing any solution, you must define your task and dataset to find a better way to detect grooming.

Know your Use-case

Do you want to detect grooming in chat data or social media profiles? If you have chat data: is that one-on-one or a public chat? Is it a teenager community (in-game chat, for instance) or an adult community (like dating websites)? You can consider your detection based on the overall context by defining your task. For instance, if you are in an adult community, you can first detect underage users. If this is a teenage community, maybe sexual content or age-related conversations can give you a hint. That way, you will know how many things you want to detect and have a general idea of what to put in your Matcher(s).

You also need a general idea of how precise your Matcher will be. If it is used alone, you can start with general patterns and slowly precise them in the testing stage by checking what it catches in data. Suppose you have some human resources and expertise to review your filter results before making a final decision. In that case, you may be careful not to miss any suspicious texts (especially with grooming), so they can be sorted in a second step (i. e. target a high Recall). In that case, your Matcher can be more generic, depending on the time you have to review the results.

If your Matcher occurs in a specific pipeline, this is trickier:

• If it is placed in the first processing steps (for instance, as data augmentation to training a model on top of it), try to find a nice balance between Precision and Recall by focusing on the F1 score.
• If it occurs in the last processing steps, maybe you can develop a more Precise filter with very specific words and patterns to filter “good” and “bad” results.

Some more insights about Matcher’s evaluation will be given in the “How to Evaluate a Matcher” section down below.

Linguistic Features and Data Exploration

If you are lucky enough to have annotated data (or courageous enough to annotate it 😉), an optional yet very interesting step before developing your Matcher would be to explore your data. It can help you gain great information about what you want to detect. In our Grooming use case, you can start by splitting your dataset between grooming and non-grooming texts (depending on your labels). By only reading a sample of each label set, you can better understand what happens in your data. When you read it, try to focus on what regularly stands out of each set and gather some similarities in semantic fields. For instance, you can bring “baby,” “bby,” “little princess,” etc., into things related to endearment in grooming chats. Do not hesitate to bring out lots of examples into various categories: the more categories you have, the better it will be to merge or delete them if you need to.

You can also do some data analysis using NLP tools. For instance, extract linguistic features from each set of texts with spaCy or NLTK libraries and then compute some basic frequencies, word clouds, or even bi-grams and tri-grams analysis. Here is a tutorial to create word clouds. If you gather linguistic features in lists, here is a quick code to get frequencies using NLTK.

#Import your libraries 
from nltk import word_tokenize
from nltk.util import ngrams
from nltk import everygrams
from nltk.probability import FreqDist
nltk.download('punkt')

#Create your big token list. 
#You can also build lists of spaCy POS tags, lemmas, NER etc.
tokens = []
for text in df_grooming["text"]:
    text = nltk.word_tokenize(text)
    for token in text:
        tokens.append(token)

#Create your lists of unigrams, bigrams, trigrams and everygrams
bigrams = list(ngrams(tokens, 2))
trigrams = list(ngrams(tokens, 3))
everygrams = list(everygrams(tokens, 1, 5)) 
#everygrams is a combination of every possible ngrams from 1 to 5 tokens

#Count list frequencies
freq_dict = FreqDist()
for gram in tokens: # <-- you can change "tokens" with every list you have
    freq_dict[gram] += 1

#Print the 10 most common words
freq_dict.most_common(10)

You can also compute collocates with different measures. Useful code to do so is mentioned in NLTK documentation.

You can also check occurrences of those words, bigrams, trigrams, and collocates to understand better what happens. At least some other features can be extracted very easily. For instance, think of text length in terms of the number of words, the number of symbols in texts (like “?” for questions, “@” for emails …), and things like that.

Matcher Development

Okay! Now that you know better what you want to detect and how to let’s jump into the development part.

Words and Patterns Creation

Once you know a bit better what you want to add to your filter, you just have to implement some patterns. This previous blog post will teach you everything you need to know about spacy Matchers and how to create, export, and reuse your Matchers properly.

You can also enrich your filter by applying Word Embeddings methods. As an example, FastText processes out of vocabulary words (oov) like “s3x” instead of “sex.” It can be an interesting model to check for chat data that is often not normalized. This blog post can help you apply Word Embeddings methods to some words of your Matcher to find new, similar words. If you have some data normalization problems, this blog post on NLP attacks will suggest some solutions.

Iterative Testings

Once your Matcher is all set, you can finally test it on a simple prompt or a whole dataset. Again, all the code is available here.

The next steps are kind of a rolling process: you test your patterns on your data, check the results and modify your Matcher regarding if you miss some word/patterns or if there is so much noise in the matches. This process is endless, and you need to decide when to stop, i. e. when you find the scores sufficient.

The iterative process of filter development

How to Evaluate a Matcher?

Talking about evaluation now, how to properly check the quality of your Matcher? What metrics may be important?

Data Annotation… or not

What is great with Rule-Base Matching is that you don’t need annotated data to develop and test it. By checking the matched texts, you can already compute a Precision score. However, suppose you want a full understanding of your Matcher efficiency on your data. In that case, it is better to apply it to a representative sub-sample or even a whole dataset that has been annotated. That way, you will also have access to a Recall and an F1 score and thus have a solid quality check for your solution. Here is a quick guideline for data annotation for NLP. You can also check this blog post on which we already went through the text annotation process.

Useful Metrics

The image below is a reminder of what are False Positives (FP), False Negatives (FN), True Positives (TP), and True Negatives (TN). The FP and FN represent what your Matcher is classifying wrong (on the picture, “False” is depicted in red and grey), while the TP and TN represent good results (the green parts are “True”). The circle of selected items represents the texts caught by the Matcher. The overall square represents the total amount of data. If data is not annotated, you will only have access to the circle (= the Matcher results).

A visualization of True Positive, True Negatives, False Positives and False Negatives

On that basis, you can compute a Precision and a Recall as shown in the image above. The F1-score is calculated based on those Precision and Recall scores:

An Accuracy, which is more a general score of what has been well classified, can also be computed with the following:

Mismatches Checking

In addition to the scores, check some mismatches manually as a last step. It is the best way to understand your Matcher's strengths and limitations fully. For instance, if you do not have annotated data, you can take 100 matched texts to review and check which ones are TP or FP. If you have annotated data, you can quickly check each sub-sample of FN, FP, TN, TP ; focusing on the wrong results to improve later on.

Use-Case: Detecting Grooming Risks in Social Media, Dating Profiles, and In-Game chats

Our client data contains social media profiles, dating profiles, and in-game chats. To detect grooming on those, we developed 4 Matchers thanks to 100 000 texts from our clients + the 160 310 grooming texts in PAN12+PJZC (see our previous blog post):

• grooming contains explicit grooming formulations, public denunciations, and people searching for underage: perv, pervert, you jerk, i’m looking for a young lady, are you home alone …
• sexual contains explicit sexual terms: ass, cock, vagina, fuck …
• it is completed by sexual-related which contains more implicit sexual content that can be related to grooming. This Matcher contains subcategories like:
• body parts: hips, tongue …
• photo sending: send (0:2) photo, pic of you …
• sexual emojis: 🍑, 😈 …
• sexual fantasies: in bed with you|me, cuddle (0:2) you|me …
• sexual orientations: gayteen, with a girl
• sexual actions re-framed as acceptable: touch it, take your/my flower, …
• undressing: take off your|my, pyjama, …
• underage-21 detects underage users (less than 21 years old). This original Matcher contains pattern variations: I be (0:2) 8:21 years? old?.
• a second version of this Matcher underage-21_v2, has been developed to increase the recall and match some more things, even if there is a bit more noise in the results (variations of the pattern: 8:21 years old?).

We detect not always proper grooming but suspicious messages that may contain grooming. Those suspicious texts are then sent to a manual queue to be reviewed. However, because moderators can’t verify many texts, we want to keep a high Precision. The combination of some filters can be interesting for some clients and use cases: for instance, underage-21 + sexual can detect underage users looking for sexual favours (useful in dating websites).

Here are the main results of the Matchers on a representative subset (N=3 011) of our client + open-access data. This subset has been re-annotated regarding if they contain underage users, sexual terms, or none of those.

Our filter results to detect grooming risks

We can see that the sexual and the two underage filters are very precise, which is what we targeted here. Unfortunately, the sexual-related Matcher still contains lots of FP, but it allows us to catch some ambiguous texts that could contain grooming. Its Recall is very low because some of those texts are already caught by the sexual Matcher. The second version of underage seems to be better knowing we have manual review behind: despite the lower Precision, it stays acceptable regarding the higher number of matches and Recall. The grooming filter catches a very small amount of text. If we read them, they do not always imply grooming (it contains patterns related to sugar babies that an underage user does not always produce). This last filter works well on open-access data but needs some investigations to adapt to the client type containing too little chat data.

Conclusion

In this blog post, we shared the importance of task definition regarding Grooming Detection, as it is hard to handle. We also shared with you a way to develop and evaluate a Rule-base Matching solution applied to this topic.

Our first results are working well as we have human expertise behind them to review the results. Detecting grooming risks is already a great improvement as a human should review not every text.

However, if Rule-Base Matching is great for explicit turns of phrases that can be modelled in some words and patterns, it can’t generalize a phenomenon as you need to know exactly what to put in your Matcher. As we saw in our previous blog post, grooming is a phenomenon full of implied messages and formulations that are hard to catch. Language not only passes through words but also through context and world knowledge that helps us understand the meaning of a specific word or text. For instance, if a groomer says, “Did you touch it in the shower. Lol,” we know “it” refers to a sexual body part, but the machine does not have this knowledge. Detecting those kinds of texts by adding this linguistics knowledge to machines is the hard part of NLP.

As an extension to this solution, we trained a BERT model (known for its language knowledge representation) to see to what extent the grooming phenomenon could be generalized by a transformer model used alone. This solution will be presented in our next and final blog post about grooming detection. Stay tuned!