3 APIs for Automating Keyword Research

Take tedious data entry, whiteboard brainstorming sessions, and a couple of hundred mouse clicks and you have the formula for many keyword research exercises. While this works for one-time efforts (and trust me, I will be the first to pull out a dry erase marker if the time calls for it), enterprise-level research requires a more sophisticated approach to data sourcing and analysis. From automating the day-to-day keywords sourcing to creating programs that semantically connect keywords together by SERP listing, tapping into new avenues of collecting keyword data will save you time and unlock new opportunities for your next SEO campaign.

So, whether you represent a larger organization, or want to bring your company/clients a larger pool of resources, an API driven keyword research process is right up your alley.

All of the examples present will be in Python (my preferred programming language), but all the APIs presented are accessible via other languages such as Java, PHP, and JavaScript.

SEMrush’s Analytics API

I’ve talked about the SEMrush API in the past, and with new updates being added regularly to reflect SEMrush’s current offerings, it has never been a better time to begin exploring. SEMrush’s API uses a REST framework for lean access to SEMrush’s massive and continuously updated keyword database. So it’s quick and easy to begin development and implementation with little hassle in the initial setup. And, with new additions such as their Phrase Questions (pulling questions-focused terms based on root inputs), and Broad Match Keywords (similar to their Keyword Magic Tool or Google Ads’ Broad Match Modifiers), keyword collection is a breeze. Here is a quick example script of how quick you can get started in the API.

import pandas as pd
import requests
import urllib
api_key = 'insert your api key'
service_url = 'https://api.semrush.com'
### For streamline URL development for call
def build_urls(phrase):
    params = {
        "?type": "phrase_related",
        'key': api_key,
        'phrase': phrase,
        'export_columns': "Ph,Nq",
        'database': 'us'
        }
    data = urllib.parse.urlencode(params, doseq=True)
    main_call = urllib.parse.urljoin(service_url, data)
    main_call = main_call.replace(r'%3F', r'?')
    return main_call
url_call = build_urls(phrase='keyword research')
r = requests.get(url_call)
data = r.json
df = pd.DataFrame(data)

Speaking from my own personal experiences, SEMrush’s keyword capacities are best utilized for mass sourcing when you have a bulk of root terms/topics that you know will have greater depth and areas of opportunity. Being able to iterate over hundreds of root terms and storing the data in one place as led this workflow to be a core part of my own process that has saved me countless hours and given me keywords I never would have thought of on my own.

Google Ads API — Keyword Forecasting.

Taking the data right from the source, Google Ad’s API featured a robust backend data stream for grabbing the best source for search volume. So why wouldn’t I put this one at the top?

Google’s APIs, especially those connected to their core advertising and analytics offerings, can have a steep learning curve in terms of jumping right in. Their documentation is impeccable and their full SDK can unlock endless opportunities. But, it can feel like drinking from a fire hydrant when first starting. SEO’s with not as strong developing skills may have larger issues with accessing the data and building lean keyword research tools compared to some of the other APIs in this article.

from __future__ import absolute_import
import argparse
import six
import sys
from google.ads.google_ads.client import GoogleAdsClient
from google.ads.google_ads.errors import GoogleAdsException
def main(client, customer_id, keyword_plan_id):
    keyword_plan_service = client.get_service('KeywordPlanService')
    resource_name = keyword_plan_service.keyword_plan_path(customer_id,
                                                           keyword_plan_id)
try:
        response = keyword_plan_service.generate_forecast_metrics(resource_name)
    except GoogleAdsException as ex:
        print('Request with ID "{}" failed with status "%s" and includes the '
              'following errors:'.format(ex.request_id, ex.error.code().name))
        for error in ex.failure.errors:
            print('\tError with message "{}".'.format(error.message))
            if error.location:
                for field_path_element in error.location.field_path_elements:
                    print('\t\tOn field: {}'.format(field_path_element.field_name))
        sys.exit(1)
for i, forecast in enumerate(response.keyword_forecasts):
        print('#{} Keyword ID: {}'.format(i + 1,
          forecast.keyword_plan_ad_group_keyword.value))
metrics = forecast.keyword_forecast
click_val = metrics.clicks.value
        clicks = '{:.2f}'.format(click_val) if click_val else 'unspecified'
        print('Estimated daily clicks: {}'.format(clicks))
imp_val = metrics.impressions.value
        impressions = '{:.2f}'.format(imp_val) if imp_val else 'unspecified'
        print('Estimated daily impressions: {}'.format(impressions))
cpc_val = metrics.average_cpc.value
        cpc = '{:.2f}'.format(cpc_val) if cpc_val else 'unspecified'
        print('Estimated average cpc: {}\n'.format(cpc))
if __name__ == '__main__':
    google_ads_client = GoogleAdsClient.load_from_storage()
parser = argparse.ArgumentParser(
        description='Generates forecast metrics for a keyword plan.')
    # The following argument(s) should be provided to run the example.
    parser.add_argument('-c', '--customer_id', type=six.text_type,
                        required=True, help='The Google Ads customer ID.')
    parser.add_argument('-k', '--keyword_plan_id', type=six.text_type,
                        required=True, help='A Keyword Plan ID.')
    args = parser.parse_args()
main(google_ads_client, args.customer_id, args.keyword_plan_id)

That being said, if you can leap over the learning curve, Google Ad’s keyword tool offers a ton of opportunities to decrease time and pull the most accurate data. Most importantly, it’s free! Well, the test accounts are free and limit the number of requests you are able to make in a day. But, regardless, for those who run with smaller budgets but still wish to bring API frameworks into their keyword research process, I would highly recommend giving it a look.

Searchmetrics

Typically used for enterprise-level SEO, Searchmetrics features a comprehensive keyword and domain database with a scalable API to match. With access to over 900 search engines and countries, Searchmetric’s API is a comprehensive way of gathering keyword data in niche markets where data is limited via conventional channels.

While the SDK for Searchmetrics is written in PHP, its thoroughly-detailed API documentation allows for smooth integration and ease of access for any programming language.

import requests
import urllib
API_KEY = "xxxxx"
API_SECRET = "xxxxx"
service_url = 'http://api.searchmetrics.com/v3/'
call_type = 'ResearchKeywordsGetListKeywordinfo.json/'
data = {
    'grant_type': 'client_credentials'
}
response = requests.post('http://api.searchmetrics.com/v3/token', data=data, auth=(
    API_KEY, API_SECRET)).json()
access_token = response['access_token']
def build_urls(keyword):
    params = {
        "?countrycode": "us",
        'keyword': keyword,
        'access_token': access_token
        }
    data = urllib.parse.urlencode(params, doseq=True)
    first_url = urllib.parse.urljoin(service_url, call_type)
    second_url = urllib.parse.urljoin(first_url, data)
    main_call = second_url.replace(r'%3F', r'?')
    return main_call
keyword_call = build_urls(keyword='seo')
r = requests.get(keyword_call)
keyword_data = r.json()

Searchmetric’s API is only available with their “Suite Enterprise” package, so utilizing their data will require a larger investment by either your clients or organization. But, with access to a larger pool of data, such as additional access to social media data and YouTube data, Searchmetric’s API is for the keyword researcher who truly wants it all.

The Bottom Line

Going off of my own personal experiences working with and writing code for each API, I would say each has its clear pros and cons. Scalability, request limits, pricing, all of these aspects can make or break a keyword research API into your workstream. And while there are many other excellent tools that I haven’t personally worked with (Ann Smarty wrote a great article highlighting some other APIs that I haven’t had experience with), no matter which you choose it’ll be hard for you to ever imagine going back to manual keyword research.