Getting started with Selenium as a screenshotting tool

Selenium is a fantastic tool for web scraping and automation but it has some other use cases that sometimes are not that obvious. A client recently asked me if there would be a way to check in an automated manner whether an HTML newsletter contained broken links. The simplest but most laborious way would be to simply click each link and have a look at where it leads. But what if you have a large number of links? This is where Selenium comes to the rescue.

In this tutorial, we’ll be learning how to use Selenium with python for loading web pages and saving screenshots of them. As simple as this may be, when you have a large number of pages to check, this can be a life savior.

Installing selenium

The installation is as simple as typing the command below:

pip install selenium

After you are done, the next thing is to set up chromedriver. This is a small piece of software that makes it possible for Selenium to communicate with Chrome.

First, navigate to https://chromedriver.chromium.org/home and download the latest stable release for your operating system. Then, unpack the file and save the chromedriver executable somewhere that makes sense for your project (you’ll need the path to the executable in the next step).

A quick overview

Selenium works by connecting with your web browser (Chrome or Firefox) and then interacting with the web page in various manners, by clicking buttons, typing text on forms, and so on.

To see how this works, open a python shell and run the code below:

Let’s go over the code line by line:

• Lines 1 and 2: here we are simply importing some classes that are necessary for working with Selenium.
• Lines 4 and 5: here we are indicating where we saved the chromedriver executable and then starting Selenium. You’ll notice that at this point, Chrome will be launched (if you are on a Mac, see some extra notes below).
• Lines 7 and 8: here we are telling Selenium to navigate to Google. After you run this line, if you have a look at Chrome, you’ll see that it loads up Google as if you had navigated to it yourself. Pretty cool!
• Line 10: we are simply shutting things down. You’ll notice that after you run this, the Chrome window will be closed.

One special note for Mac users. You’ll notice that after you run line 5, a popup like the one below will show up on your screen:

Don’t despair! Click the Cancel button (you’ll see that an exception will be displayed in the python shell) and open up System Preferences. Then, go to Security & Privacy and in the General tab, you’ll see a message like the one below:

Now click the Allow Anyway button. Then, go back to the python shell and run line 5 again. Voilà! You should have Chrome popup as you’d expect.

Taking screenshots

After that quick intro, let’s get back to what we came here for: saving screenshots of multiple web pages with Selenium.

In order to save a screenshot, all you have to do is call the save_screenshot method on the WebDriver instance after the page has finished loading with the path where you’d like to save the screenshot. It sounds simple but let’s see how this works with an example.

Let’s say we’d like to stay informed and quickly have a look at the front page of the main news websites. We could just navigate to each one of them but that’s pretty boring. What we’ll do instead is write a script that navigates to our favorite news sites and save a screenshot of each.

Let’s have a look at the code:

Let’s go over it line by line:

• Lines 1 to 3: some imports we’ll need. These are basically the same as the previous example with the extra time import. More about this below.
• Lines 5 and 6: just like the previous example, we are specifying the location of the chromedriver executable and starting up Chrome.
• Lines 8 to 16: this is a simple list containing the URLs of the news sources we are interested in.
• Lines 18 and 19: we are looping through the URLs in the list and loading them on Chrome.
• Line 20: when we load the web page, it may take some time for the page to fully load, especially for websites that are more javascript intensive. For that, we wait 5 seconds before taking the screenshot just to make sure everything is ready.
• Line 21 and 22: we are cleaning up the URL to use it as the file name for our screenshot (so you get file names like www_nytimes_com.png) and then we’re telling Selenium that we’d like to take a screenshot.
• Line 24: after we’re done, we shut down Chrome.

And there you have it! After you run the code below, you’ll notice 7 PNG files with screenshots of the websites. However, you’ll notice that only the visible portion of the page was screenshotted. Hm, that’s no good for us. What if we need to see something that’s not right at the top of the page?

To fix that, we’ll need to interact a bit more deeply with the web page. We’ll also need to launch our browser in headless mode, which means it will not be visible to us. This makes debugging a bit more challenging so in general, you should work with headless mode turned off (the default) and only turn it on after you’re sure your code is working fine.

Let’s have a look at the updated code:

You’ll notice that the code is pretty much the same as the previous example but with a couple of new lines. These are lines 6 to 8 and lines 22 to 24. Let’s go over them:

• Lines 6 to 8: here we are passing options to the browser. We do that with a ChromeOptions object. On that object, we’re switching on the headless option. As mentioned, this will make Chrome work in the background so that it’s invisible to us.
• Lines 22 and 23: here we are injecting some javascript code to run inside the web page. We are using this code to measure the width and height of the web page.
• Line 24: with the width and height in hand, we change the dimensions of the browser window so that all the content is visible before we take the screenshot. Notice that even though the browser is running in headless mode, there is still a browser window running in the background.

And now we’re done! You’ll see that after running the code above, the screenshots will be full-sized and you are able to see everything.

I hope you enjoyed this tutorial! Hit me up if you have any questions.