Fighting Spam using Clustering and Automated Rule Creation

Cathy Yang | Software Engineer, Trust & Safety

One of our biggest priorities at Pinterest is keeping Pinners safe, and that includes protecting them from spam. The Trust & Safety team’s goal is not only to catch spam, but to remove it as quickly as possible to minimize Pinner impact.

The goal of spammers is to make money, and the best way to do this is to spam at scale. It’s a numbers game: one million spam emails are much more effective than one spam email. In order to remove spam quickly, we look at common trends in spam attacks to identify suspect behavior.

To achieve the scale required to be effective, spammers must automate their actions, and each of these “attacks” can be thought of as a cluster. Each event within the attack cluster may share some common features, but different clusters will have a different set of common features.

For example, during an attack where a large number of Pins are created, a spammer might point all Pins to the same domain. While the domain may change between attacks, spammers are still trying to direct traffic to the same spam site.

One of our spam mitigation tactics is our rule engine, Guardian, which helps to identify common features in spam attacks.

Motivation

Previously, when a spam attack happened:

  • An alert would fire in our system statsboard, and an on-call analyst would investigate
  • The analyst would identify the common trends of that attack
  • The analyst would create a “patch rule” (a specific and temporary rule to address the attack) then apply it retroactively to catch old and future spammers

This process was problematic for many reasons: it was very manual for the analyst, there was a delay between the spam attack and our response, and it was a potential negative impact to Pinners.

However, what if we could automatically detect the bulk actions from spammers, find the shared common features within each cluster, create those patch rules, and catch these kinds of spam attacks?

Patch Rule

Patch rules are intended to be short lived and highly specific to an attack. The following is an example of a patch rule designed to deactivate spam users that:

  1. Are new accounts
  2. Are creating Pins with an Android device
  3. Begin the Pin descriptions with text “Check out our website”

This rule would be very effective in catching the spam attack and unlikely to impact any good users. In GSQL, our custom variant of SQL, the rule looks like this:

ACTION deactivate_user(user_id) WHERE event = ‘PIN_CREATE’ AND device = ‘ANDROID’ AND user_age_days < 10 AND pin_detail starts with ‘Check out our website’

Guardian makes it pretty easy to create SQL-like rules. Just by using the where clause, we can deactivate users with those conditions. To automate the creation of such a rule, we only need to focus on the where clause, or the conditions, of the user deactivation.

As mentioned above, most simple attacks have some common features even though they are using different accounts. Thus, our goal is to extract the common features from a set of users.

Anomaly Detection and Clustering

One of the tactics to detect bursts of activity in spam attacks is time-series anomaly detection. Anomaly detection is good for detecting spiky activities (like the ones in Figure 1) which are highly likely to be spam activity.

On Oct.25th, spam users created around 1000 Pins/hr ~ 500 Pins/hr. On Oct. 26th, 11:00~12:00, spam users created around 3000 Pins/hr. On Oct. 27th, there was almost no activity till noon, then they created Pins for 2 small 1 hour spikes (1000 Pins/hr)
Figure 1: Spiky activities

At Pinterest, we have an internal real-time anomaly detection system. We will provide the model with a time series of events from which the model can learn. If the model’s expected value is different from the actual value by a certain threshold, there’s an anomaly.

However, if we do anomaly detection on the time series of all events with all features, there would be too much noise with normal user activity, and the burst might be neglected by anomaly-detection algorithms.

An example that shows normal user activities, we have less events during night, more events during the day time. There are small spikes in the time series, but too small to get detected
Figure 2: A lot of noise on all user activities

Instead, we alert on certain event types with certain features. For example, to alert on Pin creation events, we might do anomaly detection on IP address, Pin details, geo location, or some other features. The goal here is to narrow down the events that we want to cluster on without introducing too much noise from normal user activities.

Take Pin creation time series divided by city as an example:

Left figure shows abnormal Pin creation time series in a small town: no activity for 16 days, then suddenly the Pin creation starts and peaks to 14000 Pin creates per hour, so the model expected count moved a lot. Right figure shows a normal pin creation time series, model expected count is almost always flat
Figure 3: Abnomal Pin creation events from a small town that usually doesn’t have any Pin creation activities vs. normal Pin creation events from San Francisco

Yellow line: model expected count per hr

Blue line: actual Pin creation event count per hr

The graph on the left shows the number of Pin creation events was at almost 0 for quite a long time before suddenly spiking. Since the count spiked so much compared to the moving average, our anomaly detection algorithm told us that Pin creation events from that town during that time period were highly suspicious. That narrowed our focus to the events during that time. We then gathered the relevant data and stored it in S3 to do further clustering.

Clustering

The goal within analyzing the data we store in S3 is to find users that share a certain pattern, as they are very likely coming from one spam campaign. The idea is:

  • Group by one feature of interest: if we group by domain, we will have group google.com, group instagram.com, group something.gr, etc.
  • For each group, we look at the value distribution of all other interest features
  • If there are features that have dominant values (such as 95% of IPs in the group are 1.1.1.1), we know there is a cluster

We see a lot of spammers trying to direct Pinners to their website by creating multiple Pins that link to their own domains. In the following example, let’s consider looking at a Pin’s domain as the feature we’re interested in:

  1. Anomaly detection tells us that today, starting at 13:00, city A has an anomaly
  2. We query and get all the data during that time span, then group by domain (ab123.blogspot.com; bc456.blogspot.com; something.gr; google.com; Instagram.com; Other domains)
  3. For each domain, we calculate the data distributions of several other features such as IP, account age, and Pin details. Then we will know how often each value occurs under that feature.

For domain ab123.blogspot.com:

Anomaly value Normal value IP 95% 100.120.13.4 5% other random ips Account age 90% less than 10 days 5% more than 10 days Pin details 100% contains “our website” 0%

Compare the results above with a known trustworthy domain, like Google.com:

Normal value IP 0.1% 1.1.1.1, 0.3% 2.135.2.3, … Account age Evenly distributed 1 day ~ 3000 days Pin details 1% home decor, 0.3% nails, 1.3% chicken, …

4. Google.com is a trustworthy domain; all interested features have good random distributions. However, domain ab123.blogspot.com has three dominant features:

pin_details contains ‘our website’ AND IP = ‘100.120.13.4’ AND user_age_days < 10

5. Now we can create a rule with all the conditions we have using GSQL, our query language, to write rules in Guardian:

ACTION deactivate_user(user_id) WHERE event = ‘PIN_CREATE’ AND domain = ‘ab123.blogspot.com’ AND pin_details contains ‘our website’ AND IP = ‘100.120.13.4’ AND user_age_days < 10 AND region = ‘A’

6. Some other criteria might also be helpful in reducing false positives, such as only creating the rule when > 10 users meet the criteria, or only when they’ve created more than 100 Pins. We can also make the group by condition be a regex pattern, such as ‘^[a-zA-Z]+[0–9]+[.]blogspot[.]com$’, instead of an exact domain.

Evaluating the Results

Agent review

It’s rare for normal users to have a similar pattern in a certain time span, but there’s still a possibility that the users are good if the cluster size is not very big. To reduce the impact of any false positives, we send sampled users we clustered to PinQueue, our in-house content review tool, so an agent can evaluate the rule’s accuracy before we turn it on and allow it to act autonomously on users.

Create the rule in Guardian

a rule saying: ACTION deactivate_user(user.id) where event = ‘PIN_CREATE’ and domain = ‘ab123.blogspot.com’ and pin_details contains ‘our website’ and IP = ‘100.120.13.4’ and region = ‘A’ and user_age_days < 10
Figure 4: Auto-created rule

Once we’re confident that the rule’s accuracy meets our standards, we add it in Guardian.

Archive after a certain amount of time

Since these rules are intended to be temporary mitigations of specific spam attacks, we archive the patch rule after a certain amount of time — usually after we see that the attack has died down. Not only is this a matter of good housekeeping, but it also limits the exposure of any false positives. In the future, there might be a good user who meets all these conditions and gets accidentally deactivated by the patch rule if we don’t archive it.

Spammers will mutate, but we can automatically create new patch rules!

When spammers find out that using this IP is not working anymore, they will surely mutate by purchasing other IPs or other domains to try again. But as long as there are some patterns, we can automatically create new patch rules.

Summary

With anomaly detection, clustering, and auto rule creation, we can now react fast, detect most of the naive spam attacks, and then take action on them. Below, Figure 4 shows an overview of the process:

Anomaly detection detected anomaly on region A, we gathered the data and group by domain, got 3 domains, a1.blogspot.com, google.com, sad.ga. Then we do distribution analysis, found for a1.blogspot.com, 95% of the Pin creation is from 100.120.13.4, 90% of account age is less than 10 day, and 100% of the Pin details contain “our website”. If it’s a confident bad cluster, we directly create a rule in Guardian. If we are not confident that the cluster is bad or good, we send it to human review. Rev
Figure 5: Automated rule creation with anomaly detection diagram

Acknowledgements

This blog post is made possible with a lot of help from Harry Shamansky and Kate Flaming. Thanks to Jiaqi Gu for developing the anomaly detection system. Also, thanks to Preston Guillory, Hongkai Pan, Rundong Liu, Alok Singhal, Dennis Horte, Carmen Dekmezian, Weiqi An, Yuanfang Song and the rest of our very own Trust & Safety team for their help and suggestions!

To learn more about engineering at Pinterest, check out the rest of our Engineering Blog, and visit our Pinterest Labs site. To view and apply to open opportunities, visit our Careers page.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store