Make It Real Elite — Week 3: Search Engine Client — WEB application using Golang
Welcome back! This is the second publication of the WEB Search Engine series. Previously we built a search engine indexer to crawl the Internet from a starting given URL, while using ElasticSearch to store our information taking advantage of the full text search functionality. If you missed it, no biggie, here it is. It all sounds really interesting, almost magical, but there is something that we should keep an eye on to complete the puzzle. We need an application dedicated to consume our indexes and interact with our stored information. Are you ready to learn more? Perfect!
Today we’ll build a search engine client using Golang. It will consist in a WEB app application where we will be able to query our ElasticSearch database with different terms matching them with our indexed data. Let’s take a look to the diagram below:
As you can see in the diagram above, the WEB application client will be connected to the same database as our indexer. For this proof of concept, we are starting our crawlers manually from the console, running a single command in our indexer and filling the database with the indexed values.
WEB Client using Go
Here’s one first thing to consider before building our client. We’ll need to use the ElasticSearch client for Golang called elastic. Likewise than our indexer, for this client we will use similar methods to start it, and to check if the index already exists. A SearchContent function will be in charge to run our searches. Let’s take a look at this method:
// SearchContent returns the results for a given query
func SearchContent(input string) []Page {
pages := []Page{}ctx := context.Background()
// Search for a page in the database using multi match query
q := elastic.NewMultiMatchQuery(input, "title", "description", "body", "url").
Type("most_fields").
Fuzziness("2")
result, err := client.Search().
Index(indexName).
Query(q).
From(0).Size(20).
Sort("_score", false).
Do(ctx)
if err != nil {
log.Fatal(err)
}var ttyp Page
for _, page := range result.Each(reflect.TypeOf(ttyp)) {
p := page.(Page)
pages = append(pages, p)
}return pages
}
The SearchContent method uses the input string given by the user to run a multi match query against the title, description, body and url attributes. This method accepts a fuzziness of 2, and uses the most fields type.
Further, we’ll be running the query for the first 20 results. They’ll all be sorted in decreasing order using the score relevance measure given by ElasticSearch to each result. We’re now ready and we’re parsing the results with the Page struct.
Tip: as we don’t want to show the ID and the body to the user, we are not considering these attributes in the Page struct for the WEB client service.
// Page struct to store in database
type Page struct {
Title string `json:"title"`
Description string `json:"description"`
URL string `json:"url"`
}On the other hand let’s take a look how we can serve our WEB application:
func main() {
NewElasticSearchClient()
exists := ExistsIndex(indexName)if !exists {
CreateIndex(indexName)
}mux := mux.NewRouter()mux.HandleFunc("/", homeHandler).Methods("GET")
mux.HandleFunc("/search", searchHandler).Methods("GET")
http.ListenAndServe(":8080", mux)
}
For building our application router, we’ll be using the gorilla/mux package.
Our WEB application has two routes with the GET method. The first one is the root path in the \ route. This is the homepage of our application, and every time an user hits the server with this URL, we will call the homeHandler:
func homeHandler(w http.ResponseWriter, r *http.Request) {
t, err := template.ParseFiles("views/home.html")
if err != nil {
log.Print("Template parsing error: ", err)
}err = t.Execute(w, nil)
if err != nil {
log.Print("Template executing error: ", err)
}
}
The homeHandler uses the html/template package from Go, and writes our html layout in the http.ResponseWriter . The home page view uses Bootstrap 4.1 to show the search form and the search results to the user. After hitting this route, an user should see something like this:
The second route in our application is the \search path and it is handled by the searchHandler. This handler is a bit more complex than the one before, since we will query ElasticSearch from this point:
// SearchResult struct to handle search queries
type SearchResult struct {
Pages []Page `json:"pages"`
Input string `json:"input"`
}func searchHandler(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
searchInput := r.Form.Get("input")log.Print("Querying database for: ", searchInput)pages := SearchContent(searchInput)searchResult := SearchResult{
Input: searchInput,
Pages: pages,
}jsonData, err := json.Marshal(searchResult)
if err != nil {
log.Print("JSON executing error: ", err)
return
}w.Header().Set("Content-Type", "application/json")
w.Write(jsonData)
}
Further, we’ll use the net/http package from Go to parse the form and to get the search input data given by the user. By employing this input, we called the SearchContent method we commented before. The results from ElasticSearch are stored in the pages variable. Last, but not least we’ll use the SearchResult struct with the input from the user and the pages variable to return a JSON response to the request. This response will look similar than the one from below:
# JSON response after searching for Make it real{
"pages": [
{
"title": "Make it Real - Home | Facebook",
"description": "Make it Real. 12,740 likes · 91 talking about this. ¡Somos Make it Real! Programas virtuales, presenciales e híbridos (virtual + presencial) diseñados para que aprendas a crear tus propias...",
"url": "https://facebook.com/makeitrealcamp"
},
{
"title": " Descubre si Make it Real es para ti ",
"description": "En Make it Real buscamos entrenar a los desarrolladores Web que nosotros mismos quisiéramos contratar. Personas con autodisciplina que sean capaces de resolv...",
"url": "http://blog.makeitreal.camp/descubre-si-make-it-real-es-para-ti"
},
{
"title": "Make It Real (@makeitrealcamp) | Twitter",
"description": "The latest Tweets from Make It Real (@makeitrealcamp). Online, on site and hybrid programs designed to teach you how to create your own Web applications. #AtréveteaCrear. Colombia",
"url": "https://twitter.com/makeitrealcamp"
},
"input": "make it real"
}
At this point, we can use jQuery to run an asynchronous request every time the user makes a new search, and to update the results with the ones returned in the JSON response.
Regarding the code, I won’t go deeper as of now, since this a simple code built to handle manually the response from the search request depending on how we receive the results. However, you are still welcome to take a look at this version of the code, and let me know in the comments section, if you have any questions:
After making a search for the “make it real” term, the user should be able to see more or less as represented below (Note: I ran the indexer manually before starting from the http://www.makeitreal.camp url):
And we reached another successful story together! If you want to check more about the search engine client using Go, you can visit the repository. Don’t forget to check also the readme file. I would appreciate any comment, feedback or contribution. Till next time, cheers!
