Getting Real-time Stock Data with Python Selenium (2021)

Photo by Gilly on Unsplash

Selenium is a widely used portable framework for testing web applications.

Selenium Python bindings provides a simple API to write functional/acceptance tests using Selenium WebDriver. Through Selenium Python API you can access all functionalities of Selenium WebDriver in an intuitive way.

Selenium Python bindings provide a convenient API to access Selenium WebDrivers like Firefox, Ie, Chrome, Remote etc. The current supported Python versions are 3.5 and above.

1.1. Installing Python bindings for Selenium

Use pip to install the selenium package. Python 3 has pip available in the standard library. Using pip, you can install selenium like this:

pip install selenium

Then you should be able to import selenium modules into your script

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys

1.2. Set up drivers

Selenium requires a driver to interface with the chosen browser. Firefox, for example, requires geckodriver, which needs to be installed before the below examples can be run.

Other supported browsers will have their own drivers available. Links to some of the more popular browser drivers follow.

• Chrome:https://sites.google.com/a/chromium.org/chromedriver/downloads
• Edge:https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/
• Firefox:https://github.com/mozilla/geckodriver/releases
• Safari:https://webkit.org/blog/6900/webdriver-support-in-safari-10/

The downloaded driver executable file is to be placed in the same folder as the script.

For more information about driver installation, please refer the official documentation.

chrome_driver = os.getcwd() +"/chromedriver"
driver = webdriver.Chrome(chrome_options=chrome_options, executable_path=chrome_driver)
driver.delete_all_cookies()

For the chrome_options, because we want to run the script as headless/in the background, we added argument ‘headless’. Another tip here, is that the www.nasdaq.com site blocks the visits from selenium user agent, so we faked a user_agent here to bypass the checks.

chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--window-size=1920x1080")
user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.517 Safari/537.36'
chrome_options.add_argument('user-agent={0}'.format(user_agent))

1.3. Execution

Now we can use the driver to interact with the target website with the following lines:

# For nasdaq
driver.get("https://www.nasdaq.com/market-activity/stocks/screener?country=United%20States&exchange=nasdaq&letter=0&render=download")
button = driver.find_elements_by_class_name("nasdaq-screener__form-button--download")
button[0].click()

If you like the script to download multiple datasets at the same time, you can let the driver executer sleep between tasks, as below:

import time
time.sleep(2)
# For nyse
driver.get("https://www.nasdaq.com/market-activity/stocks/screener?country=United%20States&exchange=nyse&letter=0&render=download")
button = driver.find_elements_by_class_name("nasdaq-screener__form-button--download")
button[0].click()
time.sleep(2)
# For amex
driver.get("https://www.nasdaq.com/market-activity/stocks/screener?country=United%20States&exchange=amex&letter=0&render=download")
button = driver.find_elements_by_class_name("nasdaq-screener__form-button--download")
button[0].click()
# capture the screen
driver.get_screenshot_as_file("capture.png")

End of your run, don’t forget to close the driver, or you will end up with multiple web browsers open.

driver.close()

1.4. Results

The script running results are like:

(StockMonitor) ➜ MMM git:(main) ✗ python test.py
test.py:17: DeprecationWarning: use options instead of chrome_options
 driver = webdriver.Chrome(chrome_options=chrome_options, executable_path=chrome_driver)
Stock Screener | Nasdaq

1.5. Conclusions

The full executable script can be extracted from https://github.com/PennyQ/penny-medium/blob/main/SeleniumStocks/run.py