Understanding and overcoming Search Console limits with Botify

Co-author: Matthieu de Vivie

Google Search Console (also called GSC) is a tool from Google that helps developers, website owners, and SEO professionals understand how their site is performing on Google Search. The metrics are made available through a dedicated UI, but also through an API (which is used by Botify through the RealKeywords App).

The tool is essential: if you want to learn about which keywords users are searching for and how your pages are ranking for those keywords (with related impressions & clicks), the best option is to go to the source of truth, Google. Scrappers will only tell you part of the story, with limited information about keywords that you already thought about.

There is some debate in the SEO world about the supposed GSC limitations. For small websites, the GSC UI is often enough (for each view, the UI shows you the top 1.000 results). For larger websites, you might want to use the GSC API or a dedicated app like Botify RealKeywords in order to have a more complete view of your organic search performance metrics.

Recently, Google gave some information. Let’s look deeper into GSC metrics.

By property vs by page metrics

Little known but quite important, the GSC provides two types of metrics: by property and by page.

By property metrics de-duplicate your statistics so that when your website appears multiple times on a given Google search result page, only the best ranking is taken into account:

• Typically, when you want to check how you are ranking on a given query, you will look at by property metrics.
• But when you want to monitor your pages performance, you will look at by page metrics.

You can read the detailed metrics definition in the GSC help center and see a quick recap here:

✔️ Botify RealKeywords showcases both by property and by page metrics, in a similar fashion as in the GSC UI:

• Showing by property statistics by default.
• Switching to by page statistics if you ask for page-related metrics or filter your report on a page-related scope.

Anonymized queries

What are anonymized queries? Per the GSC documentation:

Very rare queries (called anonymized queries) are not shown in these results to protect the privacy of the user making the query. Anonymized queries are always omitted from the table. Anonymized queries are omitted from the chart totals when you filter by query (either queries containing or queries not containing a given string).

When filtering by query, you can lose long-tail data in the table or the chart. When grouping by query, you can lose long-tail data only in the table. This is most noticeable in very large sites.

✔️ Botify RealKeywords also includes metrics from anonymized queries.

Understanding GSC limits

Per GSC API documentation, limits are applied when breaking down statistics:

When you group by page and/or query, our system may drop some data in order to be able to calculate results in a reasonable time using a reasonable amount of computing resources.

Some GSC limits are not documented in the new Google article, but thanks to Botify GSC integration, we are able to understand those.

For a given day:

• When requesting by page metrics, you will only get the top 50.000 (query, page, device, country) combinations.
• When requesting by property metrics, you will only get the top 50.000 (query, device, country) combinations.

We call Unknown queries the sum of queries that are not included in the top 50.000 combinations and anonymized queries. It is impossible to distinguish those. Related to those Unknown queries, for a given day:

• When requesting by page metrics, clicks and impressions from Unknown queries are only available for the top 5.000 pages.
• Clicks and impressions from Unknown queries cannot be associated with a specific device or country.

By page metrics

Let’s request performance metrics for query, page, device, country:

This “metrics by page” table (the usual “basic” GSC API integration) is limited to a maximum of 50.000 lines.

ℹ️ Digging deeper:

• If you remove a dimension (ex: device), it will only remove a column (device) from the results table. You won’t get additional data.
• If you filter on a dimension (ex: device = desktop), it will only remove lines (device ≠ desktop) from the results table. You won’t get additional data.
• On many GSC properties, the number of results is very close to 50.000 (greater than 48.000), without being exactly 50.000. Most of those properties also reached limits, but we have no proper explanation for this behavior.

By property metrics

Let’s request by property performance metrics by query, device, country:

This “metrics by property” table is limited to a maximum of 50.000 lines too.

ℹ️ Digging deeper:

• The above remarks apply. Removing a dimension or filtering on a dimension does not help getting more data.

Unknown queries metrics, by page

Let’s make a second request targeting by page metrics, but this time, with page as the only dimension:

This metrics by page table includes Unknown queries statistics, but is limited to a maximum of 5.000 lines. Adding device or country to the request, and Google will drop Unknown queries statistics.

By confronting this table with the by page metrics table from above, we can deduce the statistics related to Unknown queries and combinations that were not included in the 50,000 top ones:

We have 14000 impressions on URL A, with 8000 impressions for the query foo, 2000 impressions for the query bar, hence 4000 “remaining” impressions for Unknown queries and non-included combinations, noted N/A in the table.

✔️ Botify RealKeywords consolidates results in order to also display Unknown queries metrics.

Unknown queries metrics, by property

We can apply the same logic for the by property metrics, with device and country as the only dimensions.

Overcoming GSC Limits

In Theory

As per Google documentation, you can create a property that includes an entire domain (example.com) or a property that is limited to a single branch (https://www.example.com/clothing). Hence, if your website has a diversified organic search traffic, over several directories, you will greatly benefit from having multiple GSC properties matching those directories (sub-directories, etc).

From the above example:

• If, on a given day, you get (close to) 50.000 results for a given request on https://www.example.com/, you can create sub-properties: https://www.example.com/clothing, https://www.example.com/sport, etc.
• Then, if you still get (close to) 50.000 results for a given request on https://www.example.com/clothing, you can create sub-properties: https://www.example.com/clothing/women and https://www.example.com/clothing/men.
• Then, if you are still close to 50.000 results, you can iterate again, till you get all the results.

At Botify, we saw tremendous gains with our clients, with many of them gaining visibility on more than 6x keywords and 8x pages vs a “basic” GSC API integration.

In Practice

Let’s look at two examples of current Botify clients. Website 1 with a region breakdown:

• One Top property (website1.com).
• 9 sub-properties (/fr/, /de, /it, ...).
• Only 5% more clicks & 16% more impressions thanks to sub-properties.
• But +600% more keywords and +800% more URLs.

Website 2 with a product breakdown:

• One Top property (website2.com).
• 9 sub-properties (/shoes/, /tee-shirts/, /jeans/, ...).
• 23% more clicks & 72% more impressions thanks to sub-properties.
• +390% more keywords and +280% more URLs.

If most of your traffic is coming from URLs under the same directory, you won’t be able to create valuable sub-properties. Wikipedia is the perfect example, with all the articles under /wiki/ directory.

✔️ With Botify RealKeywords, you can consolidate multiple GSC properties in one place, overcoming most of the GSC limits. You will also gain visibility over the limits by GSC property, for clicks and impressions, so you know where to add additional properties. In addition, you can combine GSC metrics with more than 1.000 other metrics for advanced analytics and intelligent insights.

Conclusion

GSC limitations recap

The Google Search Console is a powerful tool, but often misunderstood by the market. With Botify RealKeywords and proper integration, you can:

• Better monitor your keywords performance, by automatically seeing your statistics by property when relevant.
• Get more accurate statistics, thanks to Unknown queries metrics.
• Learn about your long tail performance on both queries and pages, overcoming most of the GSC limits, thanks to the integration of the relevant GSC properties.