I wrote a screen scraper
Michael Rossiter

Yea, that is dynamically generated dom by the browser. It is not in the html when you do a straight http request, I keep forgetting that once in a while and I have scraped for years.

Also, am glad someone else is using lxml since I find the use of beautiful soup 4 subpar. Lxml is faster and bs4 is based on lxml, plus I have gotten a few times where bs4 just crashed on me on malformed html.

Wait till you get to see some really bad coded html that the parser breaks(extremely rare on lxml) and then it will be a PITA just to find what went wrong.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.