Scraping an Infinite Scroll Table

Using Python / Selenium

At the start of the new year, I joined this insurance auto pricing competition: https://www.aicrowd.com/challenges/insurance-pricing-game. It’s quite similar to Kaggle, but the evaluation is way more complicated.

The competition has a model accuracy (RMSE) leaderboard that judges how close your model’s expected claims are to the actual claims in a test set (this part is similar to Kaggle), but the final evaluation is based on the profitability of your pricing model in “markets” against samplings of 9 “competitive” competitor models (there’s a game theory component here, I think).

After many nights and weekends of trying to improve my model but only achieving more and more unprofitability, I think it’s a fair question to ask if this game is actually winnable…

To get a better handle on this question, I wanted to scrape the RMSE and weekly profit leaderboards from the competition. Unfortunately, it was a bit of a pain because the leaderboard tables were displayed as infinite scroll tables.

Python and Selenium to the rescue! It took me a while to figure this out, so hope it can help someone else too :).