How to Build a Custom Search Engine with the Google Search API (with Ruby)
Google is nowadays the go-to default search engine for almost everyone (Sorry Bing!). Not only that, but their service offering ranges from email to web hosting, including AI services, maps, among many other things.
You usually don’t think of Google Search as being available to developers as an API. In fact, it’s one of a plethora of APIs Google provides, and arguably one of the most powerful ones. In this article we’d like to leverage this service by creating a “custom” search engine, so we can integrate Google Search into our own website. This blog post is for educational purposes only, so always remember to check your API providers’ Terms of Service to make sure you’re allowed to use their services in the way you’re considering.
A little introduction
The idea is that we can build an “in-site” search box for our website using Google’s API (learn how to do this with IMDb here). To do this, we will provide a search form for our users, and in the background, we’ll prepend (or append, doesn’t really matter) a modifier to refine our search. So, for example, if our users enter a “white T-shirt” search term into our search field/address bar, we’ll query Google’s API for “site:ourwebsite.com white T-shirt” (aka site search). This tells Google to select only results for ourwebsite.com. You can try this right now if you go to Google and search for “site:rapidapi.com ruby” you’ll get specific results for RapidAPI’s website and blog.
There are other modifiers you can use on your daily searches, including prices (and price ranges), social media searches, among others. You can find them here for all types of searches.
What we’ll be building
Like we mentioned before, we’re going to build a custom search experience for our website. Seeing as we can’t build a whole website just for this article (and wait for it to be indexed by Google), we’ll be building a custom search site for RapidAPI. You can always just replace that with your own site, assuming it’s already been indexed by the search giant. This is an important caveat: you can’t really use this for internal websites (like an intranet) or use it locally for development since Google won’t have access to index these sites.
Let’s begin
You will need a couple of things to get started. Some knowledge of Ruby would be recommendable. Let’s start with signing up for RapidAPI and subscribing to the Google API. To do this, go to rapidapi.com and click on the Sign-Up button. After that, go to the Google API we’ll be using today, click on the “Pricing” tab on the top, and select the Basic Plan, which is more than enough for this tutorial. You’ll need a credit card, but you won’t need to pay for this plan in any case.
Next, download and install Ruby. The latest version will suffice, but since we’re not doing anything too advanced, an older version will do as well. Check out their official installation instructions for your platform. We’re all set now!
Setup our work environment
After installing Ruby, we can create our new project. Make a folder called custom-google-search and create a new Gemfile containing:
# Gemfile
source 'https://rubygems.org'
gem 'excon'
gem 'sinatra'After this, run bundle install in your terminal, to install both our dependencies. We'll need Sinatra to build our web server, and Excon to fetch results from our API. More on this later.
Create a new file called app.rb and add this to it:
# app.rb
require 'sinatra'
require 'excon'
require 'json'
get '/' do
erb :index
endThis is just a very basic app file for our Sinatra application. It receives a request on / and renders the index.erb template. Let's create this template now, in views/index.erb:
<!-- views/index.erb -->
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Custom Google Search</title>
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css">
</head>
<body>
Hello world!
</body>
</html>This is just a simple document that loads Bootstrap, for templating. We’ll be adding more things to this file in a moment. For now, run ruby app.rb in your terminal and go to http://localhost:4567 to check your work. You should be greeted by a "Hello world!".
The search form
Let’s now build the search form for our users. We’ll just do this in the index.erb template file. Change the contents of the body with this:
<div class="container">
<div class="row">
<div class="col-md-12">
<h1 class="text-center">Custom Search</h1>
</div>
</div>
<div class="row">
<div class="col-sm-12 text-center">
<form action="/" method="get">
<div class="form-group">
<input type="text" class="form-control" placeholder="Search" name="query">
</div>
<button type="submit" class="btn btn-primary">Search</button>
</form>
</div>
</div>
</div>This should render you a nice and simple search form. There are a couple of things to notice:
- We’re using a form that will send the user to the
/path, that is, the same path they're on now. This is just to simplify the search rendering. - The form is sent using a
GETrequest (notice the method). This is so that the search query ends up in the url, making the search results "shareable", that is, you can copy the search url and send it to someone, and they should get the same results.
Performing the search
Now that the user can type a search query and send the form, let’s process that query and fetch some results. First, modify the get '/' block in app.rb with this:
get '/' do
@query = params[:query]@results = fetch_results(@query) if @query && !@query.empty?erb :index, locals: { query: @query, results: @results }
end
We first get the query from the params (a Sinatra feature). If this is set and has content, we’ll fetch results from the API and assign them to a variable. Finally, we render the same template and pass some locals, which means these variables will be available in the template. Now, let's tweak the template a bit, change the input for this:
<input type="text" class="form-control" placeholder="Search" name="query" value="<%= query %>">Notice we added the value at the end. This is a convenience feature for our users, where the query they searched for is populated in the field after they submit the form. Without this, the field would be empty when searching, which is counterintuitive. Try it!
Now, let’s create the fetch_results method. For this, we'll need to grab our API Key and Host from the RapidAPI website. Go to the Google API page on RapidAPI and grab these two values from the Endpoints Tester:
With this, let’s create our method:
Read the rest of the tutorial at https://rapidapi.com/blog/custom-search-engine-google-api/Rendering the results
Read the rest of the tutorial at https://rapidapi.com/blog/custom-search-engine-google-api/
Final result
Restart your Sinatra server (CTRL-C and then run ruby app.rb again). You should now be able to perform searches, and see something like this:
Originally published at https://rapidapi.com on January 22, 2020.
