Spy on Ranked Pages Using Python
Get top 5 ranked pages for your search query
Python is a really powerful language that can do a lot of things. Nowadays, Python developers are using many creative libraries to automate recursive works.
In digital marketing, there are tools like SEMrush, Ahref, etc. that give creative and informative tools for digital marking experts. Gap analysis or competitor analysis is one of them. Both Search Engine Optimization and paid media experts can use this to gain useful knowledge from competitor analysis.
Here, I am about to show you how you can build your own python keyword gap analysis/competitor keyword analysis tool to spy on your competitors. And also get to know among them who are your actual competitors in google search. The reason behind the tool is that sometimes the competitors locally might not be your competitors online. They are different platforms. So, keeping an eye on the online competitors is an easy way to know about the market easily.
So, Let's get started.
Libraries Required :
from googlesearch import searchimport pandas as pd
If you’ve not installed the libraries and if you’re on windows, you can use pip to install the libraries.
Just go the terminal and write these commands to install them
pip install googlepip install pandas
Now, let’s say our search query is “Digital Marketing in the UK”. If we want to go to google.com (assuming I am in the United Kingdom) let’s search for “Digital Marketing in the UK”.
Here are the top 5 ranked pages. We need to have these 5 pages as our results. So, let’s start coding.
query = "Digital Marketing in UK"list_of_urls = []competitorslist = []for j in search(query, tld="co.uk", num=5, stop=5, pause=2): list_of_urls.append(j)
Here we put the query string in the search function of the search library and appended the results in a string by a for loop function. Here it will loop 5 times because of num=5.
The parameters of the search() :
search(query, tld=’com’, lang=’en’, num=10, start=0, stop=None, pause=2.0)
- query: query string that we want to search for.
2. tld: tld stands for the top-level domain which means we want to search our result on google.com or google.in or some other domain.
3. lang: lang stands for language.
4. num: Number of results we want.
5. start: The first result to retrieve.
6. stop: The last result to retrieve. Use None to keep searching forever.
7. pause: Lapse to wait between HTTP requests. Lapse too short may cause Google to block your IP. Keeping significant lapse will make your program slow but its a safe and better option.
8. Return: Generator (iterator) that yields found URLs. If the stop parameter is None the iterator will loop forever.
list_of_urls.insert(0,query)competitorslist = [list_of_urls]competitors = pd.DataFrame(competitorslist,columns = ['Query','Rank1','Rank2','Rank3','Rank4','Rank5'])print(competitors)
In this block of code, we added the query as the first item of the list_of_urls and a nested list of competitors list. Then, we create a panda dataframe for competitors and name the columns. So, now if we print competitors we would have the list of 5 ranked pages for the query Digital Marketing in the UK.
Query Rank1 ... Rank4 Rank50 Digital Marketing in UK https://clutch.co/uk/agencies/digital-marketing ... https://www.hotcoursesabroad.com/india/trainin... https://www.postgraduatesearch.com/pgs/search?...
The image above will give you a clear picture of the dataframe.
What to do this data :
You can create an excel sheet or spreadsheet to collect the data to the sheet when you’ve all the queries and the data copied. You’ll have a complete understanding of your competitors who are competing online.
Then you can sort them by the most found pages and start analyzing their contents, keywords, and rankings specifically. This will help the on-page SEO to create content to rank and also the PPC marketers to avoid expensive keywords or weather to compete for it.
Conclusion:
It’s a really simple way of finding your competitors on Google. You can iterate this in a list of queries to have the bulk of results and save it as a CSV file to view it as a table. Also, use your imagination to use the library to create more creative projects.
