Handling Pagination using Selenium and Java: A Quick Guide
If you are looking forward to scraping data from websites using the Selenium web driver and don’t know how to manage data spread across multiple pages, this article has you covered.
Selenium, which is primarily used for Test Automation, can also perform dynamic scraping of web content due to its ability to interact with elements of web pages by clicking buttons or navigating between pages. Before delving into the details of using Selenium to automate pagination, let’s explore the concept of pagination.
What is Pagination?
Imagine testing a website presenting products, search outcomes, or various data. Rather than showcasing everything in one go, the content is distributed over several pages. Pagination /Paging facilitates users’ navigation through these pages, commonly employing options like “Next,” “Previous,” or numbered page links. For example — Amazon’s products can have multiple pages and to scrap all products successfully, one would need the concept of pagination.
How to handle pagination?
There are different approaches to handling pagination on web pages. The basic method involves identifying pagination elements (such as “Next” or page numbers) using appropriate locators (XPath, CSS selectors, etc.) and iterating through a loop to click on each pagination link. Inside the loop, perform the necessary actions (e.g., data extraction, validation) on the current page. But what if the pagination elements are not directly visible? There are more advanced approaches like using dynamic XPath locators of page elements or infinite scrolling using JavaScript executors to load more content.
Implementing Pagination using Selenium Java -An Example
Let me guide you through the code I’ve written to handle pagination for scraping recipes from a well-known website,“ https://www.tarladalal.com/”. I used Selenium web driver for dynamically scraping recipes for different health conditions (such as ‘Diabetes’, ‘PCOS’, ‘Hypertension’ etc.) and wrote Java code in Eclipse IDE for automating pagination.
Step 1: Initialize the web driver and access the website. Once you land on the website, click on the ‘Recipe A To Z’ link after locating the web element.
Step 2: Inspect the page and identify the web elements for the pagination given below s store them as a List of elements using the provided code below.
The total size of the pagination is extracted in this step using the above code.
Step 3: Define a loop to traverse through pages and click on the pages using dynamic XPath locators. In this scenario, the XPath locators for page numbers were used. One of the challenges was that all the page numbers were not visible. For example, the elements for pages 6 and 7 were not visible until clicking page 5. To address this, inside the loop, the element for the active web page is specified explicitly by using dynamic XPaths and Selenium uses this to navigate to the next page in the loop.
The recipe links from each page are stored in a web element list. The data to be scraped from the recipe cards on each page are accessed by defining another loop. The code snippet for the same is given below.
Place the block of code for scraping required data inside the loop by finding necessary web elements /locators. If you are looking to scrape recipes in one page that requires you to scroll through the page, you can handle the situation using the below JavaScript executor code:
The understanding of the implicit and explicit wait functions in selenium comes in handy, to wait until some elements are visible. Also, ensure that your code efficiently handles exceptions and failed page loads or interactions.
Now you have covered all basic steps to implement pagination in your project using the above example. The code varies depending on the pagination mechanism used by the websites.
Happy Learning!