How to Scrape Google Search Results in 2024

tl;dr: the easiest way to scrape Google search results is to use a scraping API such as ZenRows or ScraperApi.

Scraping Google search results can be very beneficial if you’re trying to track how you rank compared to your competitors, or if you’re trying to do some market research.

Unfortunately, scraping Google SERP can be quite difficult. In this article, I’m going to show an easy way to do google scraping, that I’ve been using myself.

Using a Scraping API

The easiest way to scrape google is to use a Scraping API. The reason for this is that Google heavily protects their pages, and using normal proxies or calling the search URLs from within normal HttpClients will not work, and you will simply get a blocked page back.

So instead of trying to build your own infrastructure to get around all of this, it’s simply easiest to use someone else’s API, and let them handle this. The scraping APIs all use their own rotating proxies, have measures in place to get around captchas etc., so that they can actually reliably get the google search results for you. This is the closest thing you can get to a Google Search API.

Example

It’s very simple to use a scraping API for scraping google. I’m going to use ZenRows scraping API for this example as I use them myself, but other companies such as SerpAPI, SerpDog or SearchAPI.io works just as well.

As an example, let’s say you’re trying to scrape search results in the US for “iPhone”. Then you would simply call the scraping API (substitute your own API Key) and provide US as the country, and the URL as https://www.google.com/search?q=iphone:

https://api.zenrows.com/v1/?apikey=YOUR_KEY&url=https%3A%2F%2Fwww.google.com%2Fsearch%3Fq%3Diphone&premium_proxy=true&proxy_country=us&autoparse=true

The API returns JSON giving you the results in a nice format (I removed the majority in this example):

{
  "aproximate_results": "13160000000",
  "organic_results": [
    {
      "description": "Discover the innovative world of Apple and shop everything iPhone, iPad, Apple Watch, Mac, and Apple TV, plus explore accessories, entertainment, ...",
      "displayed_link": "https://www.apple.com",
      "domain": "https://www.apple.com",
      "link": "https://www.apple.com/",
      "ratings": null,
      "summary_points": "",
      "title": "Apple"
    },
    {
      "description": "The iPhone is a line of smartphones produced by Apple Inc. that use Apple's own iOS mobile operating system. The first-generation iPhone was announced by ...",
      "displayed_link": "https://en.wikipedia.org › wiki › IPhone",
      "domain": "https://en.wikipedia.org",
      "link": "https://en.wikipedia.org/wiki/IPhone",
      "ratings": null,
      "summary_points": "",
      "title": "iPhone"
    },

    ...

}

And it’s literally as simple as that. You can then retrieve the results from the JSON, and start tracking the search results.

Conclusion

Using a SERP Scraping API is by far the easiest option to scrape google results. Since you’re using an API, you can call it from all programming languages, Python, C#, Java, Go, JavaScript, etc. And you don’t have to maintain any expensive infrastructure that you have to pay for and keep updating because Google changes things.

There are many other scraping APIs for scraping google results than the ones mentioned in this article. I would suggest signing up for free trials on all of the popular scraping APIs, and playing around with them to see which one you like best. Finally, I want to mention that some of the links in this post is partner links, and I may receive a small commission from them.

Note: You should check yourself if you violate the TOS of Google before scraping them. I do not recommend scraping Google before you have checked if you can legally do this.