Basics of Selenium in Python3

In today’s blog, we will be learning how to write a web scraper using selenium. Web scraping is searching through the web to find the information you need or automate some web task. You will require basic terminal knowledge, only commands’ cd’ and ‘mkdir’ are required. Pip and python3 should also be installed

One of the modules for web scraping in python3 is selenium.

To start, we will install a module called virtualenv, which creates a virtual environment in your computer, although it’s not necessary to do so, since you can simply:

pip3 install selenium

To work with selenium. But I prefer a virtual environment, so you can follow along if you like. Enter into your command prompt and type:

pip3 install virtualenv

After that change directory in the terminal/ command prompt to your desktop directory and make a new folder like so:

cd desktop
mkdir selenium_basics

Now change directory into selenium_basics and create a virtual environment like so:

cd selenium_basics
virtualenv -p python .

If you get some message on your terminal like this:

Great! Now activate the virtual environment by typing this in the terminal:

source Scripts/activate

And for windows, type in the command prompt:

Scripts/activate

If something like this comes:

Then Great! Your virtual environment is ready and activate! Now we can get to the main part! Type:

pip3 install selenium

And if you get something like this

Then you have successfully installed selenium in your virtual envrionment. Now create a folder named src, by typing this in the terminal/cmd:

mkdir src

Then you have successfully installed selenium in your virtual environment. Now create a folder named src, by typing this in the terminal/cmd:

mkdir src

And then change directory to src. We generate this src folder because if you might have noticed, the selenium_basics folder is now filled with some files. We don’t want our program to clash with that.

Create a new python file, and call it whatever you want, I am calling mine tut, short for tutorial. so ‘tut.py’. You can use any editor you want, but I prefer vim.

We need to install a software to run our selenium test. So for running tests on chrome, you have the chromedriver.exe. For firefox, its geckodriver.exe and for Microsoft Edge its the edge driver. Since I use chrome, I will install the chromedriver.exe.

You can install chromedriver.exe by clicking this link. Install the chromedriver.exe based on the version of your chrome. To check the version of your chrome. Simply type:

So my chrome version is 87:

So I’ll choose that link and install the win32.zip. Extract the file to your src folder. In today’s code, we will be opening youtube, searching on it automatically. Your account won’t be logged in and all. And if you see, if you try to log in, then exit the chromedriver.exe, you are logged out of your account, and all cache is deleted, this is because of safety purposes. Now lets some coding done.

First, you import the webdriver class from selenium:

from selenium import webdriver

Then you give the location of the chrome driver like this:

driver = webdriver.Chrome('/path/to/chromedriver.exe')

The ‘/chromedriver.exe’ must come in the end. After that we need to open youtube, so we write:

drive.get(‘https://www.youtube.com’)

if you run the code now, after a few seconds, a window of youtube will open:

If this happens till now, Congratulations! You just wrote your first automation program. Now, we need to type something in the search box. On you keyboard for windows click Ctrl+Shift+I. For Mac users it command+option+I. Note this is specific to chrome.

You can see the windows beside it that opened. Click on the button that I have encircled in red. Its right beside the sign in button

Now go ahead and hover your mouse over the search box, then click on it:

Now you’ll see that one section of the HTML will become blue. Right-click on that section select copy, and the copy XPath:

After that go to your code and type:

driver.find_element_by_xpath('#Paste the xPath here').send_keys('#Whatever you want to search')

.send_keys(), sends some text, and in some cases keyboard shortcuts. Now we also want it to hit the search button automatically, so we get the XPath for that too:

Note that you have to put the cursor, not on the icon, but the box in which the search icon is in. Now after this, type:

driver.find_element_by_xpath('#Put the xpath in here').click()

Pretty self-explanatory code here, find the element with this XPath and click on it. And when you run the program:

You’ll see that the search was automatically completed. Now all you have to do is click on the video you want. We can make this program even more fun! First, we import the time module and then ask for the user’s input on what they want to search. To make sure we have enough time to click back on the chromedriver window, we can tell the program to pause for two seconds before continuing. Something like this:

import time
ask_search = input(str('What do you want to search: '))
time.sleep(2)

And then we edit this line:

driver.find_element_by_xpath('//*[@id="search"]').send_keys("Whatever you want to search")

And change it to:

driver.find_element_by_xpath('//*[@id="search"]').send_keys(ask_search)

So we run this script and I enter, for example, Dani:

And then back in the chromedriver window:

Done! We can now search for whatever text we want, and then the search will happen automatically! Thank you for reading, and if you have any problems or if the programme worked for you, please put it as a comment. I’ll try to answer them all.

Here’s the entire code:

from selenium import webdriver
import time
driver = webdriver.Chrome('/path/to/chromedriver.exe')
ask_search = input(str('What do you want to search: '))
time.sleep(2)
driver.get('https://www.youtube.com')
driver.find_element_by_xpath('//*[@id="search"]').send_keys("Dani")
driver.find_element_by_xpath('//*[@id="search-icon-legacy"]').click()

Thanks once again! In the next tutorial, we’ll be learning how to send automated sms using python. Click this link to find out more.