Python + Selenium for Dummies like Me

honestly if I can do it, like so can you

I used to see Selenium on job applications and thought “wow what is that??? Like it just SOUNDS Hard!” Turns out Selenium is actually kind of EASY. Like, you don’t even really need to know Python to use Python Selenium because God loves the world.

What is Selenium and the DOM?

I don’t really know the smart person definition for this, but basically selenium lets you target different parts of the HTML DOM hierarchy. What is the DOM hierarchy? Basically all of this stuff:

The entire HTML Document is called the Document, which contains:
The Element which are anything like the div, text, nav, etc, which has:
An Attribute that adds further context to the Element. In this example:

<button type="submit" value = "some value" id = "bestButton">

The button Element has three Attributes: type, value, and id — each with their own property.

These Elements can also have Events which are various different interactions someone can have with the elements like clicking, scrolling, pressing a button, etc. Lastly the Elements can have style through HTML or CSS.

<form method="post" onsubmit="myFunction()" id="myform" style="background-color: #f787fe;">
{% csrf_token %}
{{ form.as_p }}
<button type="submit">
</form>

In this form Element, it has a three Attributes: method, action, and id. When the form is submitted via the button Element, it will activate the javascript attached through the onsubmit Event. This form has a Style that specifies the background color.

Why is this important?

Basically Selenium lets you target Elements of the Document through their Style or Attributes, and lets you trigger Events, set Attributes, and more.

First Steps: Installation and Test

To install Selenium type this in your terminal (assuming you have pip and python)

pip install selenium

We will be using Selenium with Chrome so we need to install the ChromeDriver. Place it in your /usr/local/bin folder once it is downloaded. Navigate to where you downloaded your chromedriver and:

mv chromedriver /usr/local/bin

Now lets try a simple Google search!

Launching Google

Line 1: Here we import selenium’s webdriver. This is what will be making calls to the browser.

Line 3: Set a variable to link back to where we placed the chromedriver earlier. Based on the browser you want to use, you’ll need a different driver.

Line 6: Set a variable that you’ll be accessing webdriver + chromedriver with. If this was for Firefox, it would be webdriver.Firefox().

Line 7: The .get function tells the webdriver to open that specific web address.

And with only 5 lines of code, we can automatically launch Google in chrome.

DOM navigation

Now that we know how to open a webpage, let’s tell selenium to interact with it.

Line 8: So now that the google page is up, we are telling selenium to find the element that has the id attribute of “lst-ib” and evoking the event send keys. “lst-ib” is the ID of the main page search box and we’re sending “Hello!” to the search box.

Line 8: Searchbox ID is “lst-ib”

Line 9: Now that “Hello” is in the search box, we want to press the search button to finish our search.

Line 9: Button’s name is “btnG”

Line 10: This calls the function to pause so we don’t get to Line 11’s quit() immediately. quit() causes the browser to close down and this pause interval is needed because the processing speed of the browser is much slower than the code. ***Setting intervals between page loads is usually needed. If the next page isn’t loaded but the next line is looking for an element that isn’t currently present, selenium will shutdown with a NoSuchElementException***

Just like that we had google automatically search the term “Hello!” for us and then shutdown.

Further Steps: Sailing with Selenium

It seems a little too simple right? It’s because it is simple! Honestly with the knowledge of a few more tools, you can automate anything you want.

Useful Element Selectors:

Just like the two .find_element_by’s we used in the Google example, there are many more ways of targeting elements on the page.

For Single Target Elements

  • find_element_by_id
  • find_element_by_name
  • find_element_by_xpath
  • find_element_by_link_text
  • find_element_by_partial_link_text
  • find_element_by_tag_name
  • find_element_by_class_name
  • find_element_by_css_selector

For Multiple Elements

  • find_elements_by_name
  • find_elements_by_xpath
  • find_elements_by_link_text
  • find_elements_by_partial_link_text
  • find_elements_by_tag_name
  • find_elements_by_class_name
  • find_elements_by_css_selector

Most of these are self explanatory just in their name. Link text is the literal Text in the href. Css Selector uses CSS syntax to find the element, here is a good write up explaining how it works. Xpath is used for locating via XML syntax. It’s a bit tricky but it can let you target elements really specifically. Useful if the site’s developers were terrible at assigning elements’ classes and IDs. Here is a good tutorial, but usually you don’t need to use it.

Interacting with Elements:

In the Google example, after we found the search box element, we sent the keys “Hello!”. Here are some other cool things you can tell selenium to do to elements.

  • clear()
  • send_keys(“some strings”)
  • send_keys(“some other string”, Keys.ARROW_DOWN)
  • select_by_value(“some value”)
  • select_by_index(index)
  • select_by_visible_text(“text”)
  • .deselect_all()
  • WebDriverWait(driver, “some time value”).until(some condition)

clear deletes whatever is value field if possible. 
Send_keys sends whatever you specify into the element, and the Key.ARROW_DOWN will press the down key. This can be replaced with UP, LEFT, and RIGHT. Good for filling out forms or using a site’s shortcuts.
The Selects are for selecting a value from radio buttons, option selects, and the like. 
WebDriverWait acts like the time.sleep we used in the Google example, but we can specify parameters for it to wait for. So we can call a 10 second wait until an element with a certain class is in the DOM. Good for navigating pages and load times.

Take Away Principal

Find Elements in the DOM using the selectors based on their attributes and interact with them by issuing events. With this basic principal, you can automate any web browsing.

Show your support

Clapping shows how much you appreciated Jude Gyeongbae Jung’s story.