Less Hassle, More Data
— Tips for scraping an infinite scrolling website
Quick Tips
- Go to your target website (infinite scrolling website)
- Select either LISTLY WHOLE or LISTLY PART
- Click Edit Setting on your Databoard
- Increase Repeat N Times as much as you want
- Click Refresh and get more data
Imagine yourself trying to do market research to start a phone case business. Well, you might first want to collect some information about your competitors’ best-selling items to analyze market trends.
Here are some tips to help you grab data out of your competitor websites in just a few clicks. We’ll take Casetify as an example.
As you can see, this is a long, infinite scrolling website where product information appears as you scroll down the page.
If you scrape this website in the first place, you’ll only see a limited number of products collected.
In order to load more data and collect them, we can simply tweak the settings by clicking add setting button. Go back to your Databoard and click add setting, as below.
Scroll down the page and increase Repeat N times to 5.
Ready to hit the refresh data button on your Databoard page?
You’ll probably be able to find more items that were collected after changing the setting.
Okay, then why not increase it up to 20?
Voila, now you can handle infinite scrolling and collect all the items out of the infinite scrolling website by automating this step!
Web scraping doesn’t have to be hard. Interested in learning more about automated web scraping? Check out here!
