How Pinterest runs Traffic-based Interlinking experiments for SEO

James Ouhyoung | Growth Search Traffic, Bruce Yu | Growth Search Traffic

Search Engine Optimization, or SEO, is an important channel for people to find ideas and inspiration on Pinterest, and can also help expose new users to first use cases, such as hairstyle tutorials or a home office setup.

One of the ways we gain a successful SEO presence is by surfacing the best content (among billions of Pinterest pages) via the use of interlinking.

On the Pinterest Growth team, we champion the use of experiments to validate the hypothesis and verify results. After some extensive research, we determined our best path forward was to develop a standard interlinking framework of our own, which we’ve been using to measure the impact of interlinks.

What is interlinking?

Interlinking is an SEO strategy that creates internal links that connect one page of a website to another page in the same domain. This is helpful because:

1. it helps users navigate within a website.
2. it helps Search Engines understand the structure of your websites.
3. it helps determine the page authority and ranking power within the site.

For the reasons above, we often see that search engines reward websites that have good interlinks with higher ranking, in turn driving more SEO traffic to useful websites that fulfill user intent.

An example of internal links at Pinterest looks like the following. On many Pinterest pages, we have a section called Related Pins (highlighted in blue in Figure 1) that takes a Pinner from this page to other Pin pages on Pinterest.

Figure 1. Pinterest Related Pins interlinks.

Using the example above, let’s suppose you have a new recommendation algorithm to choose a better set of Related Pins to improve the user experience. To understand how this change affects the SEO traffic on the current page, the old Related Pins, and the new Related Pins, you need to run an interlinking experiment.

Why is an interlinking experiment special?

People working on growth are typically familiar with the concept of A/B testing. However, taking a closer look at what happens when you change an internal link, you will see that the effects are twofold, and a simple A/B experiment is not sufficient. To explain:

1. The first part of the impact is the change in SEO traffic on the source pages (the pages that contain the internal links), and we can measure the impact on the source pages with a simple A/B experiment.

• control group: choose a set of pages and use the old internal links.
• treatment group: choose a set of pages and use the new internal links.

2. The second part of the impact is the change in SEO traffic on the destination pages (the pages the internal links link to), and we can measure this with Pinterest’s interlinking experiment framework illustrated later in this article.

• control group: choose sets of pages that are the destination of the old internal links.
• treatment group: choose sets of pages that are the destination of the new internal links.

TLDR on learnings

1. Pick the existing winners. If we are looking for maximum SEO sessions, picking destination pages that already do well in search results (thus driving good SEO traffic sessions) actually help boost more total traffic.
2. Existing SERP (search engine results pages) ranking is more sticky than you might think. As you might expect, when we point an internal link to a new page, the new page receives a boost of traffic from the search engines. We typically see this increase within 2 weeks. Surprisingly, when the internal link no longer points to an old page, the old page does not suffer much. (The metrics stay flat for 8 weeks.)
3. Expect fast results. We acknowledge that the time to impact for new internal links depends heavily on the domain, crawl rate, etc. With that said, we have seen the traffic shifts start as soon as within 4 days and stabilize as fast as approximately 2 weeks.

For the people looking to run interlinking experiments, let’s jump into the actual Pinterest Interlinking framework.

Pinterest’s interlinking framework

Step 1: Define Hypothesis

If we think about what happens when we change the destination of a link from page A to page B, we signal to search engines a promotion to page B and a demotion to page A.

To fully confirm that your new selection algorithm selects the best destination pages, one needs to verify the below hypothesis.

1. If you have 2 identical new webpages (same content, layout, domain authority, creation time, etc.), the page linked from an internal link would have more traffic than another page that was not linked.
2. The SEO traffic increase from linking to a new page should be larger than the traffic lost from unlinking an old page. This is to ensure we avoid the complete cannibalization of simply shifting traffic from page A to page B.
3. An interlinking (or selection) algorithm typically involves ranking pages destinations in the form of priority or scores. Then linking to a page with a higher score should have a higher gain in session traffic than linking to a page with a lower score.

In practice, to prove all the hypotheses above, we need to create three experiment groups based on the scores of destination pages: Top_1 Target Pages, Top_2 Target Pages, and Original Target Pages.

• Top_1 Target Pages is the set of destination pages ranked with the highest scores from your new selection algorithm.
• Top_2 Target Pages is the set of destination pages ranked with the 2nd highest scores from your new selection algorithm.
• Original Target Pages is the set of destination pages that we previously linked to but are removed now with your new selection algorithm.

By comparing sections of Top_1 Target Pages vs. Original Target Pages, we can prove hypotheses (1) and (2).

By comparing Top_1 Target Pages vs. Top_2 Target Pages, we can prove hypothesis (3).

Step 2: Set up Experiment groups

Now that we know what the three groups are, let’s visualize the experiment.

Notice that for most situations, there will be overlaps among Original Target pages, Top_1 target pages, and Top_2 target pages. (The target page “Cute cat collections” can at the same time be the first choice to the link “cute kittens” and the second choice to the link “adorable pets”)

Figure 2 below reveals the overlaps:

Figure 2. Experiment groups with overlaps

Next, to get the traffic gains and losses of these groups, we divide the target pages and create 3 sets of the control-vs-enabled group.

From the target pages that uniquely belong to Top_1 target pages (highlighted in green), we randomly pick 10% of the pages and break them into a 5% control group and a 5% enabled group, as shown below.

With this pair of the control-enabled group, we will be able to measure the traffic gain per 1 Top_1 pages will get after getting promoted by interlinking. To put this into writing:

• TFC_en1 is the daily average traffic per Top_1 pages get after promoted by interlinking (enabled group)
• TFC_ctrl1 is the daily average traffic per Top_1 pages get with no change (control group)

Figure 3. Top_1 gain: control-vs-enabled groups

Similarly, from the unique Top_2 target pages pool, we obtained the 2nd control-vs-enabled group. We will then measure the traffic gain per Top_2 pages after getting promoted by interlinking.

• TFC_en2 is daily average traffic per Top_2 pages get after promoted by interlinking (enabled group)
• TFC_ctrl2 is the daily average traffic per Top_2 pages get without any change (control group)

Figure 4. Top_2 gain: control-vs-enabled groups

Lastly, we construct the unique old target pages pool and get the third experiment groups from it. With the third set experiment groups, we will be able to measure the traffic loss per old pages after losing the promotion from interlinking. And we defined the following:

• TFC_en3 is the daily average traffic per old pages that can get after losing promotion by interlinking
• TFC_ctrl3 is the daily average traffic per old pages that can get without any change

Figure 5. Original target pages loss: control-vs-enabled groups

Step 3: Calculation & Analysis

First, let’s define following:

• ∂TFC1 = TFC_en1 — TFC_ctrl1
• and ∂TFC1 is the per page gain daily in traffic to Top_1 Target Pages.

Similarly:

• ∂TFC2 = TFC_en2 — TFC_ctrl2
• and ∂TFC2 is the per page gain daily in traffic to Top_2 Target Pages.
• ∂TFC3 = TFC_en3 — TFC_ctrl3
• and ∂TFC3 is the per page gain daily in traffic to Original Target Pages.

To prove the hypothesis #1: All things being equal, a page linked from an internal link would have more traffic than if the page was not linked.

We need:

• ∂TFC1 > 0

And also, good to have but not required:

• ∂TFC2 > 0

To prove the hypothesis #2: The SEO traffic increase from linking a new page should be larger than the traffic lost from unlinking an old page.

We need:

• ∂TFC1 + ∂TFC3 > 0

To prove the hypothesis #3: Assuming that your new interlinking algorithm assigns scores to destination pages, linking to a page with a higher score should have a higher gain than linking to a page with a lower score.

We need:

• ∂TFC1 > ∂TFC

If all three assumptions are satisfied, your new algorithm is indeed much better than the old one. As a result, you could get more SEO traffic by unlinking an old page and linking to a new Top_1 target page.

Future work

The growth search traffic team has done several experiments on interlinking algorithm changes with this interlinking framework and will continue to do more.

We hope this illustrates the necessary process to experiment and evaluate your new interlinking strategy for your website, taking into the account population overlaps and the cannibalization effect.