Using Selenium for a Web Scraper

Here’s all the things you can do with Selenium on your Web Scraper.

Photo by Farzad Nazifi on Unsplash

Finding Elements

This is the most common use for Selenium. You can find specific data on a website and the extract it.

Selenium can find elements using:

• Tag name
• IDs
• Class name
• X Path
• CSS Selectors

Here’s what it would look like on code for each case:

by_class = driver.find_element_by_class_name('class name')
by_id = driver.find_element_by_id('id')
by_tag = driver.find_elements_by_tag_name('tag name')
by_xpath = driver.find_element_by_xpath('x-path')
by_css_selector = driver.find_element_by_css_selector('css selector')

Running Javascript

You might need to make some actions on a web page, for example, scrolling. Here’s an example of how you would do that:

js = "window.scrollBy(0,1000);"
driver.execute_script(js)

Screenshots

You can also take a screenshot of a web page:

driver.save_screenshot('webpage.png')

Waiting for elements to show up

Web pages might take a while to load up. Sometimes it’s because of slow network speeds, sometimes the web application could be waiting for an API response.

In this case, we’d use WebDriverWait.

try:
    element = WebDriverWait(driver, 3).until(
        EC.presence_of_element_located((By.ID, "elementID"))
    )
finally:
    driver.quit()

Proxy with Selenim Wire

You can’t solely use Selenium for proxy authentication, you’ll need to install the Selenium Wire package.

Selenium Wire extends Selenium’s Python bindings to give you access to the underlying requests made by the browser.

I hope this was helpful! If you’d like to make a web scraper with Selenium & Beautiful Soup, read this post:

Make a Web Scraper with Python and Beautiful Soup