Automate Web Scraping and Testing with Python | Selenium Recipies

Jonathan Turnock
Dec 22, 2019 · 4 min read
Photo by Franck V. on Unsplash

1. Introduction

Many automation libraries exist for python which help scrape sites and perform testing. Projects like Roboform, BeautifulSoup and requests all provide excellent features.

One slightly lesser known name to Python projects is Selenium. It’s an industry accepted automation and testing framework for UI’s. Unlike other web browsing/scraping libraries it actually plays out inside a browser as though the system was being interacted with.

This approach gives test teams the ability to automate a whole suite of regression tests for a site. It also helps to overcome some challenges with regards to JavaScript in sites.

A major hurdle in web scraping can be how JS causes the site to change after loading. We might find that a request response being parsed in something like BeautifulSoup is missing lots of parts. This is because sites that use JS need a browser such as chrome to actually run the JS code.

This is where Selenium comes into its own as it actually runs a browser window and sends our commands to it.

Here is a visual example of Selenium running through the JsonPlacerholder UI

Selenium Tests running on JsonPlacerholder (Try It Option)

An entire ecosystem exists for Selenium and its possible to make a career out of being good at this kind of automation and testing. There are UI toolkits, a bespoke IDE and even a custom language which can be used to write tests.

However, we are going to completely skip all this and get straight into the Python API.

2. Install Chrome and the Chrome Driver

The first step in automating our Python UI testing is getting a supported browser and its associated driver.

For our examples we will be using chrome so get chrome installed first.

Once in chrome we need to get the version by visiting chrome://settings/help

Chrome Version 79

As the installation instructions vary by OS we wont go into detail on individual process but download the driver, unzip the contents and add it to the system path.

For Windows it will involve unzipping into something like:

C:\Program Files (x86)\ChromeDriver

Then adding to the path via the System Properties > Environment Variables window.

For OSX and other Unix based operating systems simply copy the binary from the download to the /usr/local/bin folder.

For a more comprehensive guide on Selenium and the installation process the see the docs:

3. Hello World of Selenium

In the following recipe we see how to perform simple interactions with a basic site

The below recipe illustrates how to perform the actions from the GIF in the intro. We see how along the way we capture screenshots to create the GIF using the driver.save_screenshot(…) method.

  1. We first setup the Options for Chrome. This is just a way of passing arguments.

This page will be updated with additional recipes.

Jonathan Turnock

Written by

☕️Java 🍃 Spring 🐍Python 🐳Docker — Experienced Full Stack Java, Python, Angular and Electron developer.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade