Let me spin you a tale about Aragog and his taste for 404 pages…
This year, we have been testing Accelerated Mobile Pages (AMP) on Little Green Dot (littlegreendot.com).
Little Green Dot runs Wordpress and it had been getting a little bloated, so we did a big speed optimisation sprint at the start of the year to bring the PageSpeed scores up to 95+.
AMP seemed like the next logical step, but upon deployment, it never moved the needle in terms of SEO.
Worse, the AMP pages had lower visitor engagement — higher bounce rates, shorter time on site, poor conversions.
So we pulled it down.
As per Google’s instructions, we removed the rel=”amphtml” link from our pages, we submitted an updated sitemap, removed the AMP pages and returned a HTTP 404 Not Found status for the removed pages.
How Google handles AMP removals
Here’s what visitors would see if they came across a page in Google’s AMP cache (*.cdn.ampproject.org) that had been removed:

The visitors that chose to continue to littlegreendot.com would show up in our Google Analytics as a 404 page hit, like so:

It took one full week for Google to purge all of the removed AMP pages from its cache. After a week, Google correctly referred all traffic to the non-AMP canonical page.
(As an aside, after removing AMP in July, we saw a 15% increase in search traffic from Google compared to the previous period, validating our hypothesis that visitor engagement is a better metric to optimise for, over raw speed.)
How Pinterest handles AMP removals
Because Aragog’s URLStore contains the location of a page’s AMP HTML equivalent, the Pinterest mobile app is able to direct users to the AMP page instead of the canonical page.
When AMP pages are removed from a site, this causes 404 errors, like this:

So far, so the same as Google.
But when we look at the 404 hits from Pinterest in Google Analytics, it’s a different story:

After a full month, Aragog’s URLStore has not been refreshed to remove the reference to the AMP pages.
While only a small proportion of Little Green Dot’s total visitors are affected by the 404 errors, it makes up a large percentage (100%?) of the visitors from the Pinterest app.
This does not mesh with Pinterest’s goal to provide “Pinners with the best possible browsing experience whenever they click through on a Pin”.
How do we fix the browsing experience for Pinterest app users?
Aragog fetches and parses billions of URLs every week, so it’s reasonable to expect a lag between a site owner changing a page and that change getting reflected in the Pinterest’s URL metadata.
This time can be reduced by using Pinterest’s URL Debugger to fetch new scrape information.
While this isn’t a good solution for entire sites, it can be easily done for the top 20% of pages that receive 80% of referrals from Pinterest.
However, looking at the URL Debugger for a popular page, we see the following:

Pinterest’s metadata is already up-to-date and understands that there is no AMP equivalent page.
But although Aragog has this information, it’s not acting on it. So we can only assume that the Pinterest app will direct users to the removed AMP URLs for all time.
A server-side fix for Pinterest’s metadata problem
Because we can’t rely on Pinterest purging the outdated URL metadata from its systems, we have to take action on our side.
Luckily, this is an easy fix.
Because littlegreendot.com was using the Automattic AMP plugin, all of the AMP URLs have a standard URL pattern — the canonical URL appended with amp/, e.g.
https://littlegreendot.com/homemade-body-butter-recipe/amp/So we can add the following single rule to our .htaccess file to catch all AMP requests and redirect them to the canonical URL, without triggering a 404 error:
RewriteRule ^/(.*)/amp/$ /$1/ [R=301]And, if you’re scared to tinker with your .htaccess file, you can achieve the same thing using the Redirection plugin:

But here’s the thing…
This is a terrible practice!
In order for AMP to be valuable, we need to serve valid AMP HTML to clients that are expecting it.
If I redirect my AMP HTML page to a regular-old HTML page, I risk breaking the experience for users of that client.
It’s better to say, “This content is gone” rather than “Here’s some content you didn’t request”. That is, serve a 404 instead of a redirect.
(There’s an additional argument to say that by skipping Google’s AMP caching and pre-rending and serving the publisher’s AMP page directly, you are missing out on most the benefits of AMP, but that’s a story for another time.)
Recommendations for the Pinterest Engineering team
When I started writing this, I thought my recommendations would be simple:
- increase the rate at which Aragog crawls AMP URLs to check their validity, and
- allow publishers to invalidate/refresh their Pinterest URL metadata in bulk.
However, now that I know that the URL metadata is already up-to-date, my recommendation is simpler still — find and fix the logic in the app that is serving up those AMP URLs, even when Aragog knows they have gone away.