FREE ARTICLE
Enabling Client-Side Search
From Hugo in Action by Atishay Jain
In this article we show how to build a search widget for a Jamstack website.
Take 40% off Hugo in Action by entering fccjain into the discount code box at checkout at manning.com.
Just like dynamic form submissions, a search widget with real-time results as you type requires JavaScript. Now that we have a skeleton structure of the JS code and JSON-based pseudo-API for the website ready, we can use them to provide client-side search.
Concept of client-side search
In traditional systems, search is server-based, where the keyword supplied by the client maps to values in a search index which provides the best ranking pages for the keyword. A client-side search is a concept where the server supplies this index to the client (or the client builds it dynamically), and this mapping happens on the client.
Client-side search has a bunch of advantages over server-based search:
- The search index is static like everything else in the Jamstack. It can be distributed over a CDN and provide all the advantages of caching and performance that a CDN has to offer.
- The search index is pushed to the client on-demand or even preloaded. Therefore, there is no roundtrip time lost in sending keystrokes to the server, and the search becomes faster.
- There is no additional server to maintain and keep in sync with the database. The user’s machine supplies the resources required to perform the search.
- The search can work even if the user goes offline after loading the initial web page.
The significant limitation of the client-side search is the size of the index. If we have an extensive index, the preload of the search index can prove to be too bandwidth-intensive to be of any practical use. We can split the index and load it in parts on demand but stretching that approach goes back to the world of the entire index maintained by the server.
For most websites, the textual content is not very huge from the eyes of the modern web. 2 MB of data can store 2 million characters. That number is considerable for a text-based search index but not a massive overhead for a web page where we do have images of this size often on websites. While we can create a more optimized and robust search index in Hugo, the amount of data in the Acme Corporation website is so tiny that we supply all of it via the JSON pseudo-API. We can even move the search index creation to JavaScript.
Showing the search box in the header
A search widget consists of an input box and a result dropdown to show results from partial queries. We will be adding it to the website header.
Listing 1. Search form to be added to the website header. (AcmeTheme/layouts/_default/baseof.html)
<header>
...
<span id="search"> ❶
<input type="search" placeholder="Search"> ❷
<div></div> ❸
</span>
{{ partialCached "menu.html" ... }}
...
</header>❶ Wrapper div to contain the search form and the result list.
❷ Actual search form for the website.
❸ Placeholder for search results.
That is all that is needed. JavaScript will be jumping in to make this search field active.
Loading the website data
To fill up the search results in JS, we need to load the website content from the Pseudo API and create a search index. We can use JavasScript’s fetch function to fetch the website data into a variable. (https://github.com/hugoinaction/hugoinaction/tree/chapter-10resources/03 ).
Listing 2. Loading the website data using window.fetch function in JavaScript (AcmeTheme/assets/search.js)
export default {
async init() {
try {
const response = await window.fetch( ❶
"/index.json");
if (!response.ok) {
this.removeSearch(); ❷
return;
}
let data = await response.json(); ❸
/ Just for now.
console.log(data);
} catch(e) {
this.removeSearch();
}
},
removeSearch() {
document.querySelector("#search")?.remove();
}
}❶ Using the fetch function to download the index file with all the website content.
❷ In case of error, remove the search box.
❸ Get the response data as an object from JSON.
The above code has one problem that will break if the website is hosted within a subfolder like in GitHub Pages, as this code assumes the root /index.json is where the JSON version of the code lives. We will be passing the site.BaseURL to the defines as another variable to fix this. This value needs to be surrounded by quotation marks to be valid JavaScript (we could use params instead of defines which does not have this limitation).
Listing 3. Adding BaseURL to defines parameter (AcmeTheme/layouts/default/baseof.html)
{{ $defines := dict
"REMOVE_FORM_ON_SUBMISSION" (default "false" (
site.Param "RemoveFormOnSubmission"))
"BASE_URL" (print "\"" site.BaseURL "\"") }} ❶❶ Surround by quotes to make this a valid JavaScript string.
We will also need to fix our JS code.
Listing 4. Adding BASE_URL to ensure the search always picks up from the right endpoint(AcmeTheme/assets/search.js)
const response = await window.fetch(
BASE_URL + "/index.json");We will be invoking the init method of the search form from the index. Even though the function is async, we can call it without using await if we do not need to wait for it to return a valid value.
Listing 5. Initializing the search query. (AcmeTheme/assets/index.js)
import Search from "./search"
function init() {
...
Search.init();
}The code above should log the entire contents of the website in the browser console.
Code Checkpoint. Live at https://chapter-10-05.hugoinaction.com.
Source https://github.com/hugoinaction/hugoinaction/tree/chapter-10-05
Importing a search library
When the data on the website is small, we can use regular expressions and loop through the content to find results. It may work, but an excellent full-text search library can be helpful when we need features like fuzzy matching (which allows for results with partial terms and autocompletes), properly weighted scoring of search results. The JavaScript ecosystem has many ready-to-use libraries, which are very well maintained and easy-to-use readily available.
Node.js (See https://nodejs.org) needs to be installed (we can use any version) on the machine for getting community modules. Once node.js is available, we can use the npm (Node package manager) command line.
Before installing a node.js dependency, we need to initialize node.js for our project. We have multiple projects on our website, the Acme Theme project and the Acme Corporation website project. Since the search code lives in Acme Theme and is shared, we need to initialize node.js in the Acme Theme project.
To do that, we will be running npm init and answering a small questionnaire to get a `package.json file that can list our JavaScript-based dependencies.
Listing 6. Initializing as a npm reposiory (In AcmeTheme/)
npm initNext, we need to search for and download a node.js module to help users with fuzzy search. To find a library using npm, you can use the npm search command.
Listing 7. Searching for fuzzy search library on npm
npm search fuzzy searchListing 8. Search results for fuzzy search using npm search fuzzy search.
❯ npm search fuzzy search
NAME | DESCRIPTION | AUTHOR | DATE
fuse.js | Lightweight… | =krisk | 2021-01-05|
fastest-levenshtein | Fastest Levenshtein… | =ka-weihe | 2020-08-07|
fuzzy-search | Simple fuzzy search | =wouter2203|2020-02-20|
feathers-mongodb-fuzzy-se | hook which adds… | =arve0 | 2020-09-13|
arch | | | |
minisearch | Tiny but powerful… | =lucaong |2021-06-25|
mongoose-fuzzy-searching|Mongoose fuzzy…| =vspallas|2020-11-03|
fuzzy-tools | Functions for fuzzy… | =axules | 2021-04-18|
fuzzy | small, standalone… | =mattyork | 2016-10-01|
leven-match | Return all word… | =eklem | 2021-06-11|
fuzzysearch | Tiny and… | =bevacqua | 2015-03-06|
mongoose-fuzzy | Mongoose fuzzy… | =pabloc | 2020-07-28|
scored-fuzzysearch | Tiny and… | =jhudson | 2020-07-31|
neofuzzy | Quick fuzzy search… | =jeanno | 2020-11-26|
fuzzy-search-mongoose | Fuzzy sarch | =piotreksl | 2020-09-28|
vue-fuse | A Vue.js pluggin… | =shayneosulli… | 2021-07-02|
liblevenshtein | Various utilities… | =dylon.edwards | 2015-07-04|
fuzzy-pop | Simple fuzzy search… | =yoshokatana | 2015-05-05|
fast-fuzzy | Fast and tiny… | =ethanrutherf… | 2021-05-19|
react-fuzzy-picker | Search through a… | =1egoman | 2019-09-29|Here the root command passed to npm is search, and we are searching for a library that provides fuzzy search. The top result from npm is fuse.js. A quick check over the internet shows us that `fuse.js` is Apache-licensed, reasonably small(< 50kB), has no other dependencies, and has been maintained regularly for almost a decade with regular releases along with having a lot of downloads and packages depending on it.
To add a dependency, we can use the npm install command. The --save-dev flag saves the development dependency in package.json so that is it is available for use if we do npm install on a new machine. A development dependency means that it is used only during development and not required in the released website. Since we compile our dependencies, we do not need them at runtime.
I would recommend using version 6 of fuse.js.
Listing 9. Adding fuse.js as a dependency (In AcmeTheme/)
npm install --save-dev fuse.js@6This command will generate a file called the package-lock.json file along with a node_modules folder. The package-lock.json is equivalent to go.sum and holds the checksums to ensure the integrity of our dependencies. The node_modules folder is similar to the _vendor folder, which stores our dependencies. npm does not create a hidden folder for the dependencies.
Updating our build systems to support npm
Unless archiving the node_modules folder does not make sense to be submitted to source control. It is not easy to keep it out either, as we will need to run npm install inside the
AcmeTheme module to get its contents. Running npm install inside the AcmeTheme module may not be possible since Hugo, by default, puts modules in a hidden folder.
Therefore we need a way to get the fuse.js dependency exposed to the top-level AcmeCorporationWebsite project. To perform this task, we need to rename package.json inside the AcmeTheme module to package.hugo.json. If there is a package.hugo.json file present in a Hugo module, Hugo understands that this module depends on npm, and Hugo is allowed to copy its dependencies to the top-level project.
To transfer our dependency to the top-level AcmeCorporationWebsite project, we can run the following command:
Listing 10. Generating the top-level package.json by packing all module packages (In website root folder /)
hugo mod npm packHugo will initialize the top-level AcmeCorporation website as an npm-based project and create a package.hugo.json and a package.json. Now we run npm install at the top-level AcmeCorporationWebsite project to get node_modules and package-lock.json in that folder. The ones in the AcmeTheme project are redundant, and we can delete them. If ever a new dependency is added to the AcmeTheme project, we need to add it to the package.hugo.json and run the hugo mod npm pack command again.
Next, we need to update our build script to install npm-based dependencies. For this to work, we need the npm install command installed on the build machines. Netlify’s build machines come pre-installed with npm, while for GitHub Actions, we need to add a step. Note that npm i' is a shorthand to `npm install. There is also an npm ci command which ensures the dependencies match the package-lock.json. But it does not delete the already installed node_modules may cause builds to take longer.
Note that since we have the exact version of our Hugo Modules-based dependencies via go.sum, the npm-based dependencies of the Hugo Modules can not change across builds. Therefore we can run hugo mod npm pack only when we change our modules and check in the generate package.json in source control.
Updating Netlify
To update the build command in Netlify, we can go to the Site settings > Build & deploy > Continuous deployment > Build command to update the build command. Since the Netlify UI takes only one test box for the build command, we can use the && operator to pipe commands and ensure both succeeded.
Listing 11. Build command to setup npm based dependencies and build Hugo.
npm i && hugo --minify --baseURL $DEPLOY_PRIME_URLUpdating GitHub Actions
For GitHub Pages, we need to add a set of build steps in gh-pages.yml to set up node.js and then run `npm i’.
Listing 12. Changes to GitHub Actions to install npm and npm based dependencies. (.github/workflows/.gh-pages.yml)
jobs:
deploy:
steps:
...
- name: Use Node.js
uses: actions/setup-node@v1
with:
node-version: '16.x'
- name: Install NPM Dependencies
run: npm iWith these changes, we have the fuse.js search library ready to use in our JavaScript code.
Creating a search index
We can import fuse.js by using the import statement in JavaScript. After fetching the website data, we need to pass this to fuse.js to create a search index. We will be making a weighted search index where the title will weigh 20, and a tag will score 5 while the content gets a weight of 1. This scoring allows for having the word in the title given a much higher value than being present in the web page content.
We store the index as a local variable of the module. This way, it can be used by all methods in the modules. Since search is not a class and we expect only one instance, a local variable of the module acts like a private variable not accessible outside of this file.
Listing 13. Importing The fuse.js library to perform a search within JSON-based content. fuse.js provides support for fuzzy matching, weighted search to have a great searching experience. Running this in JavaScript makes the search responsive and fast. (AcmeTheme/assets/search.js)
import Fuse from 'fuse.js'
let index = null; ❶
export default {
init() {
...
let data = await response.json(); ❷
index= new Fuse(data, {
keys: [{ ❸
name: 'title',
weight: 20
}, {
name: 'tag',
weight: 5
}, {
name: 'content' ❹
}]
});
/ Just to test. Do not leave in code.
console.log(index.search('acme')); ❺
}
}❶ Creating a module variable to store the index to be used in all functions.
❷ Creating a fuse.js index.
❸title is added with weight 20.
❹ If not provided, weight is treated as 1.
❺ While developing, leaving a test query can help. We log the search results to the browser’s console.
Code Checkpoint. Live at https://chapter-10-06.hugoinaction.com. Source code at https://github.com/hugoinaction/hugoinaction/tree/chapter-10-06
Getting search input and showing results
With the search input box and the search method ready, the next step is to link the two together. The first thing we need to do is listen to the input event on the search box. We will be running a search query as soon as the user enters a single character in the search box and displays the resultant page’s title in the result div. If the user presses the enter key, we will navigate to the first search result. We will also be limiting the number of search results to a reasonable number.
We also need to show the search result dropdown when the user focuses on the search box and remove it when the user clicks outside. The full file after this change is present in chapter resources (
https://github.com/hugoinaction/hugoinaction/tree/chapter-10resources/05)
Listing 14. Showing the search results inline via a dropdown is relatively straightforward. We use the input event to take keyboard and context menu entries. (AcmeTheme/assets/search.js)
import Fuse from 'fuse.js'
let index = null;
const MAX_SEARCH_RESULTS = 5;
export default {
init() {
...
document.addEventListener("input", this.showResults); ❶
}
showResults(event) {
const searchBox = document.querySelector(
"#search input");
if (event.target !== searchBox) {
return;
}
const result = document.querySelector(
"#search div");
result.style.display = "block";
if (searchBox.value.length > 0) {
const results = index.search(searchBox.value);
result.innerHTML = results ❷
.slice(0, MAX_SEARCH_RESULTS) ❸
.map(x => `<a href="${
x.item.url}">
<img src="${x.item.cover || ""}" width=
"40" height="40">
<h3>${x.item.title}</h3>
<span>${x.item.content.substr(
0,40)}</span>
</a>`) ❹
.join("");
} else {
result.innerHTML = '';
}
},
...
}❶ The input event is the best one for a text box as it handles uncommon cases like copy paste via mouse and regular keyboard presses.
❷ The innerHTML is used to replace the contents of the dropdown. Note that we can update existing DOM elements instead if performance is a big concern.
❸ Limit the number of search results to MAX_SEARCH_RESULTS
❹ Provide a rich dropdown experience with an image and accompanying text.
Note that the variable MAX_SEARCH_RESULTS could come from the Hugo config as define or a param.
With these changes, we have a working search box inside our website to help users navigate the entire content.
Code Checkpoint. Live at https://chapter-10-07.hugoinaction.com. Source code at https://github.com/hugoinaction/hugoinaction/tree/chapter-10-07
The GitHub Pages repository with the npm changes is present at https://github.com/ hugoinaction/GitHubPagesNpm.
Using Hugo modules with JavaScript
While npm is straightforward to use, we can continue to use Hugo modules to load dependencies. Hugo Modules allow dependencies to provide template code, bundled content, and other Hugo-specific data alongside JavaScript. The assets folder in a Hugo module acts as the node_modules folder in node.js.
We did not add any keyboard handling in our search handler. We will be importing a Hugo module AcmeSearchSupport (
https://github.com/hugoinaction/hugoinaction/tree/ chapter-10-resources/06) to perform this task.
We start by adding this as a dependency to AcmeTheme.
Listing 15. Adding AcmeSearchSupport as a dependency to AcmeTheme (AcmeTheme/config.yaml)
module:
...
imports:
...
- path: github.com/hugoinaction/AcmeSearchSupportNext, we load this module our search.js and call it during initialization.
Listing 16. Loading JS code from Hugo modules to be compiled by js.Build (AcmeTheme/assets/search.js)
import AcmeSearchSupport from "SearchSupport"
...
export default {
async init() {
...
try {
...
AcmeSearchSupport();
} catch (e) {
this.removeSearch();
}
},
...
}That’s all for this article. If you want to learn more about the book, check it out on Manning’s liveBook platform here.
