Paginate Your Search Result with Elasticsearch and Go
As a human, we have limitations in processing information. We separate the newspapers or books into several pages and read the content on each page individually. So it’s essential paginating the information to consume it effectively. And Elasticsearch provides some way to do pagination based on our needs. Let’s discuss more detail in the following sections!
Typical Pagination Types
In our digital devices, pagination commonly consists of two types below :
- Pagination by number
- Pagination by scroll
Pagination by number is a metaphor for our real-world activities applied to digital devices. Like we read books, newspapers, dictionaries, etc., it transformed into digital books, digital dictionaries, etc.
Moreover, for another example is the data table. The data table displays the data inside the table with the pagination feature. Usually, it’s applied in the admin dashboard for data processing and monitoring.
Next is pagination by scroll. This pagination is well-known as an infinite scroll. It’s popularly used in our social media feeds (or any similar platform). The feeds paginate the information using a scroll to consume information indefinitely. Aside, we can only move the information before and after. It differs from a number in which we can see future information by jumping into several pages above.
In Elasticsearch provides all the pagination mentioned above. On the following section we will discuss the detailed implementation for each use case.
Design a Bookstore Application
To grasp the concept quickly, let’s do some actual code by creating a simple bookstore application. Start with designing the model of our bookstore structure using struct in Go.
After that, we continued to design an API contract with the pagination features described below:
- Get Books
- Get infinite Books
Coding Time!
Before we start, we need to make sure all pre-requisite below is readily installed on our local machine:
Then you have to make sure your Elasticsearch is running using docker. You may visit my previous post about how to run Elasticsearch with docker for complete step-by-step instructions. Once all is successfully installed, we are ready to write a code.
Populate Books
We have to populate books in Elasticsearch to simulate our pagination. We are settling 50 books by executing the bulk API below :
# download populate-books bulk script
curl -O https://raw.githubusercontent.com/sog01/simple-bookstore/main/populate-books
# populating books from downloaded populate-books
curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/_bulk --data-binary "@populate-books"To ensure the books are stored in Elasticsearch properly, we can examine them by hitting API http://localhost:9200/books/_search.
Develop Get Books Endpoint
Let’s try the API from Elasticsearch to paginate the search result by number, with executing curl below :
curl --location --request GET 'http://localhost:9200/_search' \
--header 'Content-Type: application/json' \
--data-raw '{
"from": 1,
"size": 10
}'Then, the result will show :
To explore more, you can change the value of from and size params and try to examine the result. After you are satisfied, we can jump into our Go code.
Let’s begin with writing a GetBooks function in the repository layer, but before doing that, we have to write an Elasticsearch model first. The Elasticsearch model is the initial response we need to map into the Books model. Since we don’t need all attributes, we select the parts we care about. The code would be like this :
Then, we can write a GetBooks function followed by a repository contract and implementation:
Once we did, we continued to write the service layer like the following code below:
The service needs to embed into the handler object, so let’s do this by writing the code below:
Finally, all layers are done! One last step is writing our main.go file that injects all layers to become one union object of the handler. Now, the handler can listen and serve a request that comes into our API :
Let’s test our GetBooks API by running go run main.go, then make some requests:
If you got a similar response to the screenshot above, you successfully created your API.
Develop Get Infinite Books Endpoint
Once we have done with the Get Books API, let’s continue to develop the Get Infinite Books API. For simplicity, We don’t separate these endpoints code into two files. Therefore for the following code views, instead of gist, we use a diff changes to examine the code before and after.
But before coding, let’s try the API from Elasticsearch first. We are using search after API to perform an infinite scroll endpoint:
curl --location --request GET 'http://localhost:9200/books/_search' \
--header 'Content-Type: application/json' \
--data-raw '{
"search_after": [0],
"sort": [
{
"id": "asc"
}
]
}'To use the search after API, we must sort our search and utilize the sort values to move to another page:
After that, let’s start to write the repository layer code by modifying the SearchHits first :
After that, we can start to write the repository code inside the Books and BooksRepository objects:
Once we have done this, we can continue to create the GetInfiniteBooks service that calls the repository function, then maps the data into the service response :
Lastly, we can create the handler that calls the service GetInfiniteBooks function based on requests from the client:
Finally, we are ready to test our GetInfiniteBooks API by running go run main.go, then make some requests:
Congrats, we are successfully developing the pagination API! Let’s make some requests to get more results.
After you are satisfied, we can move to the bonus section.
Bonus: Bookstore Simple UI
This bonus section gives us an idea of how this API works in the real application. We create a simple UI that lists our books collection using pagination by number.
We start writing code by creating the handler for the web :
After that, register the router into the books handler :
One last step is writing an index.html for our web template:
Finally, to see the result, we can open http://localhost:8080. Try to look at our book collection by moving the page.
Wrapping Up!
Congratulations to us! We have successfully developed a simple pagination using Elasticsearch and Go.
We begin with developing endpoints that implement pagination of a book list using two following types :
- Pagination by Number
- Pagination by Scroll
And in the bonus section, we integrate the endpoints into a simple bookstore UI.
Thank you; hopefully, it will be helpful for you. You can access to complete source code in my GitHub repository.
Always stay tuned because we will be back with other advanced content.
