If you get really stuck, use Selenium
Sep 7, 2018 · 1 min read
I had a first in the last few months. A site which would not scrape. I thought I had matched everything. User Agent, time, load images …
They just knew I wasn’t a real browser.
This on a site that i’ve scraped for years and part of an audit process at work.
The answer was to use Selenium driven by Python. This runs in a virtual desktop.