Nice article about web scraping.
Mani Anis
41

Based on what you are describing, here is a quick solution for this is to use the time library & from there the function sleep. I think this would be the best course of action for what you are describing. The sleep function takes an integer that represents how many seconds the program should wait. This way you can have the program wait a reasonable amount of time that would allow the 1 liner to be replaced. The code would be like this:

from time import sleep

… your code that sets up the web browser …

browser.visit(“https://www.page_with_1_liner_body”)

sleep(15) # the program will wait 15 seconds

then scrape! I always use this but didn’t include it in the tutorial because it was so long. Great question! Thanks for the read.

Like what you read? Give Lauren Glass a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.