How to use Google Custom Search Engine (GCSE) in your Python code?

Lynn Kwong

Follow

Apr 30 · 4 min read

Google is a great source for obtaining data, either in your personal projects or in real business projects. Google has a Custom Search Engine (CSE) API which can be conveniently used in your Python code. It has a free version which is enough for studying purpose or even small projects which have less than 100 search queries every day.

To get started using Google Custom Search Engine, you need to have a valid API key, which can be obtained here. You need to log in with your Google account and create a project for it. You can create a project on the fly or create one in the Google Cloud Platform Console. A project is used to organize all your Google Cloud resources. All your API keys, virtual machines, storage, etc. reside in some projects. The projects and API keys are free to create. However, the resources in the projects and the usage of the API keys can incur costs. You can check the fees at this link. Therefore, keep your API keys secrete and never expose them in a public repository.

Then you need to create a Search Engine ID. Go to the Programmable Search Control Panel, click the “Add” button under “Edit search engines”, then enter the fields required, as shown below. Importantly, you can add some sites where your custom search engine will search for data. You can add multiple sites if needed. In the advanced settings, you can also specify which sites should be excluded from searching. In this demo, we just want to search in https://www.w3schools.com/.

Once the search engine is created, go to “Control Panel” to check the advanced settings. In practical usage, you would normally want to fine-tune the settings, especially the “Sites to search” and “Restrict Pages using Schema.org Types”. Check the relevant sections for the detailed introduction of them.

For a demo of the search results with your custom search engine, you can open the “Public URL” in the control panel, or just search directly in the embedded search window. For example, let’s search for “Python”. We indeed get the results from https://www.w3schools.com/. Cool, isn’t it?

The free version has Ads in the result. If you don’t want to have Ads in your search results, you can choose a paid version. However, you can also earn from the Ads in the search results if you integrate them in your front end. The JSON API we are going to use is a paid version and won’t have Ads. Additional API requests cost $5 per 1000 queries, up to 10k queries per day. Luckily, the JSON API also provides 100 search queries per day for free. Therefore, you don’t need to pay anything if you just use it for studying purposes or for your private project.

Now everything is ready, let’s implement the custom search engine in our Python code. We can use the generic requests library or the more specific Google API client to implement our searching code.

With requests. First, we need to install the requests library, which is the standard HTTP library for Python.

python -m pip install requests

The code for searching in the custom search engine can be found in the following code snippet. We need to manually construct the search URL with all the parameters needed. Importantly, we need to pass the API key for the key parameter and the search engine id for the cx parameter. You can check this link for all the parameters available, which can be helpful if you want to fine-tune your search results.

import requests
import urllib.parse as urlparseparams = {
    "key": "YOUR-API-KEY",
    "cx": "YOUR-SEARCH-ENGINE-ID",
    "q": "Python",
    "start": 0
}gcse_base_url = "https://www.googleapis.com/customsearch/v1"url_parts = list(urlparse.urlparse(gcse_base_url))url_parts[4] = urlparse.urlencode(params)gcse_url = urlparse.urlunparse(url_parts)data = requests.get(gcse_url).json()
items = data["items"]

data is a dictionary that contains the searching result for the search query. The important key is items, whose value is what we are interested in. You can implement your own business logic about how to deal with the data received. For example, if you scrape some shops you may want to return the product name, price, URL, etc.

With Google API client. Another more formal way is to use the Google API client in Python. First, we need to install it.

python -m pip install google-api-python-client

We need to use the build function to create a service object.

from googleapiclient.discovery import buildservice = build("customsearch", "v1", developerKey="YOUR-API-KEY")

Then we need to use the list method to get the results:

data = service.cse().list(q='Python', cx='YOUR-SEARCH-ENGINE-ID',).execute()

The data is the same as the one obtained from the requests library. However, the code is much clearer and simpler.

In this article, Google Custom Search Engine is introduced briefly and you should be able to use it in your work now. Google is a really good data source. Not only can it provides data for your business, but it also provides the option for you to earn money from the Google Adsense program if you implement Google Custom Search Engine in your front end.