Uncovering Google Search Intent with SerpApi and Python

Benjamin Burkholder
Apr 20 · 6 min read

Determining search intent has long been a notoriously difficult task for SEO professionals to nail down. This article looks to provide you with a script I’ve written to help provide insight for one essential question:

What SERP (Search Engine Result Page) features correlate with what kind of intent?

The script in question connects and pulls data from SerpApi, allowing the developer to query against the API and pull all imaginable data about the Google SERP. The script connects to this API and, based on what SERP features are present for a given query, categorizes each query depending on what the search intent is. The script also pulls in the domain addresses present on each paid ad for a SERP to give additional context to what the intent is, and which competitors are bidding on those queries. The top five organic listings are also pulled in to provide a snapshot of who is ranking above the fold for a give query and intent.

It’s important to point out that the categorization of intent in this article is opinion and subject to change, so don’t take it as an unmovable fact.

Why is Search Intent Important?

Search intent is just what it sounds like, it’s the intent of the searcher when they’re querying on a search engine. Typically the intent falls under a few chief categories:

  • Informational — The user is searching for more information about a certain topic. In the script, this intent is determined if: answer box is present, paa(people also ask) boxes are present,
  • Navigational — The user is looking with intent to find a specific website, not for local searches. In the script, this intent is determined if: knowledge graph is present, site links are present.
  • Transactional — The user is looking to purchase something via the web or in-store. In the script, this intent is determined if: shopping ads are present.
  • Commercial Investigation — The user is looking to purchase in the near future, but is doing research on various products beforehand. In the script, this intent is determined if: paid ads are present, answer box is present, paa’s are present, if there are ads present at the bottom of the SERP.

Shoutout to my colleague, Drew Schwantiz, for providing the intent categories above and stipulations on what SERP features should be present for each intent.

Based on the intent of the query, SEO professionals can better inform what keywords to create content around and where the user is at on the buyer’s journey.

What does the Search Intent Script do?

Now that we know why search intent is important, we’ll move into the specifics of what my script does.

Here is the high level breakdown:

  1. The script reads queries loaded from a local TXT file.
  2. Each query is queried against the API.
  3. The JSON results are filtered by each “intent” listed in the section above. If each stipulation is met for a given search intent, the query is categorized as that intent. Each query can have multiple intents.
  4. If paid ads are present on a SERP, the domain addresses present on the ads are pulled in. This provides insight into which companies/competitors are bidding on given queries.
  5. The top five organic results on a SERP are pulled in, providing insight into which companies are currently ranking for given queries.
  6. The results are written to a local csv for further analysis.

Initial Setup

Now I’ll walk you through what you need in order to leverage this script and the API it pulls from. I will refrain from including all of the steps around installing Python, Git and cloning from GitHub as I’ve included these steps in my previous articles around the PageSpeed API, Knowledge Graph API and Google Natural Language API.

  1. First, you need to purchase an API key from SerpApi. Simply navigate to their website and, under “Pricing”, sign up for the plan that best fits your specific needs. The “Developer” plan gives you 5000 queries a month, which should be plenty for most users. If you need more though, there are two higher tier plans which have a much higher query limit.
  2. Second, navigate to SerpApi’s GitHub account to either fork (if you have a GitHub account) and clone the repository with your language of choice to your machine. For this article, we’re using the Python variant.
  3. Third, once cloned to your desktop (or wherever you want to keep it), the directory should by default be named“google-search-results-python”.
  4. Fourth, navigate to my GitHub account and clone the “serp-intent-categorizer” repository to your machine. You’ll notice that there are actually two Python files in the repository, this article is focusing on the ‘serp-intent-categorizer-static” variant but check out the README for more info on each module and use cases.
  5. Fifth, take the files you’ve cloned from my GitHub and drop them into the “google-search-results-python” directory you’ve cloned from SerpApi. This is so my script can access the necessary modules.
  6. Sixth, create a new TXT file and name it “serp-api-key”. Then paste in the respective SerpApi key you received when you signed up. Then drop this TXT file into the “google-search-results-python” directory. This is important because the script reads the TXT file to pull in the API key needed to query the API.
  7. Seventh, you should see a TXT file present in the repository cloned from my GitHub account named “serp_intent_queries”. This is the file you will load with queries to query against the API. Make sure each query is on it’s own line, as the script reads the file and runs the queries sequentially. I’ve loaded this file with a test query just so you can see how it would look.

Once you’ve successfully followed all of the setup steps above, you should now be ready to run the script.

Running the Script

  1. Open the IDE of your choice (I use Pycharm) and open the “serp-intent-categorizer-static.py” file located within the “google-search-results-python” directory.
  2. Run the script.
  3. You should see activity in the terminal, indicating the query being analyzed and status of each request.
  4. Once the script completes, you should see a new csv present in the directory named “serp-intent-results.csv”. This file contains four columns: Query, Search Intent, Paid Ad Companies, Top 5 Organic Results. For each query that had the necessary intent features, you should see it as a line item in the results.

Summary

Hopefully this script proves useful in determining the search intent for given queries. I’ll be updating the script periodically as we alter and change the stipulations for each intent, as well as adding additional features when necessary. Be sure to clone out a fresh copy of my repository if you return after an extended period, that way you’ll get the latest updates.

As stated before, be sure to checkout all of the SERP results available via SerpApi. There is a lot of great information to be gleaned from their API connection if you have the ability to parse through it.

Finally, if you have any questions/suggestions for updating my script leave a comment and I can potentially implement it. If you have any questions about the SerpApi, simply shoot them an email. They are very responsive and agile at making updates, I’ve actually worked with them to add a new data point not present when I originally discovered their API. So if your suggestion makes sense to their team, they very well may add it in.

As always, feel free to shoot me an email or comment if you have any comments or questions.

Benjamin Burkholder

Written by

Digital Integration Specialist | Python Developer in Cleveland, OH.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade