If you get really stuck, use Selenium

Tim Bray
Tim Bray
Sep 7, 2018 · 1 min read

I had a first in the last few months. A site which would not scrape. I thought I had matched everything. User Agent, time, load images …

They just knew I wasn’t a real browser.

This on a site that i’ve scraped for years and part of an audit process at work.

The answer was to use Selenium driven by Python. This runs in a virtual desktop.

    Tim Bray

    Written by

    Tim Bray