4 interesting to-dos with Selenium WebDriver

To put it briefly, Selenium WebDriver utilizes a programming language of your choice to design scripts for automated web browsing activities across different browsers. In another words, Selenium WebDriver helps to automate what a user would normally do on a web browser. That’s it!

But what you really do or automate with it, is purely up to you. Here’s four potential things that can be done with Selenium WebDriver using Python.

Before going into the specifics, it is a good idea to initialize and set up a WebDriver for the browsers you are intending to use. Here are 3 of the more common WebDrivers that are normally used:

• Mozilla — GeckoDriver
• Google — ChromeDriver
• Safari — SafariDriver (Be sure to “Allow Remote Automation” in Safari)

The WebDriver to download should be of the same version as your computer’s browser. (E.G.: Computer’s Firefox browser version is 70.0.1. Therefore Gecko driver to download is version 0.26.0 as this release is for Firefox version >60.0)

Initializing a WebDriver with your Python project can be achieved with:

from selenium import webdriver  # Main webdriver element
# The below 3 imports are utilities for the webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup  # For parsing of HTML if required
# setting up of WebDrivers - might be good to use 1 at a time
firefox = webdriver.Firefox(executable_path='path/to/gecko.exe')
chrome = webdriver.Chrome(executable_path='path/to/chrome.exe')
safari = webdriver.Safari()  # no exe_path needed for safari :)

Now let’s go through the fun stuffs!!

#1 Automated web testing

This is one of Selenium’s original intention. Utilizing web browsers to perform unit tests on your website’s functionalities, or checking of broken links and various other HTML assets.

The example below provides you with an idea of how to structure your unit tests. We can load the website, click into each article, and (maybe) parse the article with bs4 to ensure that it does not contain any malformed HTML tags.

# Loop to alternate Firefox, Chrome and Safari for website testing.
webbie_url = r'https://website_url.com/'
for i in [firefox, chrome, safari]:
    i.get(webbie_url)
    
    # Some awesome bs4 checks here!!
    # Or other selenium clicks and field inputs 
    # JavaScriptExecutor - for visualization of scrolling 
    i.execute_script("window.scrollTo(0, document.body.scrollHeight);")

While it may not be efficient to loop through browsers like this. It does provides you with the concept of how it can be implemented.

Fun fact — you can instruct Selenium to execute JavaScript actions!

#2 Web scraping dynamic content with Selenium

Some contents found on websites are generated dynamically, or have values changing because of certain conditions. If that’s the case, scraping content can be a little more tricky if you are using purely bs4. A different approach is to mimic a user’s interaction on the website before using bs4 to parse out the elements you are interested in.

Note: You can choose to run it headless (without the browser displaying) as well. A simple Google search will show you the way.

The below is a working example for fetching dynamic content with Selenium. It is for illustrative purpose only.

...
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
crypto_url = r'https://www.cryptocompare.com/'
firefox.get(crypto_url)
try:
    WebDriverWait(firefox, 10).until(EC.presence_of_element_located(
                                    (By.CLASS_NAME, "panel-body")))
    crypt_elements = firefox.find_element_by_class_name('table-coins')
    prices_html = BeautifulSoup(crypt_elements.get_attribute(
                           'innerHTML'), features='lxml').prettify()
    print(prices_html)
    # You can then parse prices_html to your heart's content. 
except Exception as e:  # Basic try catch to check for any errors.
    print(e)
finally:
    firefox.quit()  # closing webdriver

.get_attribute(‘innerHTML’) is used to obtain the HTML of the selected element. The idea is to wait for the elements to be loaded by the browser, find the elements you are interested in, and then parse it via bs4 to obtain the final result.

The above code fetches the prices once. To continuously fetch the prices, you can pass it through a recursive loop.

#3 Downloading pictures/videos and uploading them to cloud storage or to social media

The purpose of this is not to be creepy, but if you are performing image processing activities and require a large data set for your experiment, I’m pretty sure you wouldn’t want to download thousands of images manually.

An alternative is to use Selenium WebDriver to automate this task for you. For this example, this code retrieves all the img src endpoints from the website. It is possible to alter the code to save the images into a cloud storage, or directly into your machine learning algorithms for direct processing. Choice is yours!

firefox.get(r'https://www.reddit.com/r/wildlifephotography/')
response = BeautifulSoup(firefox.page_source, features='lxml')
image_divs = response.find('div', 
                           attrs={'class': 'rpBJOHq2PR60pnwJlUyP0'})
image_tags = image_divs.find_all('img')
for i in image_tags:
    print(i['src'])  # Various interesting methods to extract image!
# JavaScriptExecutor - for visualization of scrolling 
    i.execute_script("window.scrollTo(0, document.body.scrollHeight);")
for i in image_tags:
    print(i['src'])

One interesting thing about this is that the older contents will only be loaded into the HTML as they are scrolled into view. This event can be simulated in Selenium with the JavaScriptExecutor. It is used to scroll down to the end of the page for the next few sets of images to load.

Note: Some websites do offer users an API to consume their service. Be sure to check it out as well because it’ll be cleaner and a more recommended approach.

#4 Auto populate time sheets & trackers

Filling in trackers for Project Management or reporting of issues/bug (in JIRA for example) can be a real pain sometimes. But have no fear, Selenium is here!

Inspecting the field entries will allow you to target the forms precisely and populate it with the values and information you want to.

# Pseudocode for accessing and reporting on JIRA
firefox.get(r'https://url_to_jira)
# Log into JIRA
firefox.find_element_by_id('login').click()
firefox.find_element_by_class_name('user').sendKeys('me')
firefox.find_element_by_class_name('pass').sendKeys('w@nttol0gin')
# Navigate to your JIRA project
firefox.find_element_by_class_name('project_name')
# Report bugs / issues
firefox.find_element_by_id('report_type').sendKeys('bah')
firefox.find_element_by_id('issue_type').sendKeys('foo')
firefox.find_element_by_id('issue').sendKeys('This is the problem')
firefox.find_element_by_id('reporter').sendKeys('me')
firefox.find_element_by_id('date').sendKeys('10/11/2019')
firefox.find_elememt_by_id('submit').click()

This logic can be extended to other tracker web-based solutions as well. It is really up to your imagination on what can you automate away!

In short, Selenium Webdriver is a great tool for people to use in automating some of the monotonous web work that takes place on a regularly basis. Not everything can be automated away, or will be easy to automate. But at least it may be worthwhile to have a think about it to see how it can help lighten your workload.