Create A Web-Crawler Notification Bot

Photo by Andrea De Santis on Unsplash

Motivation

With today´s shortage of technical products, every second counts and the notifications can be very helpful at this point. But the notifications are curse and blessing in my opinion. The problem is you get the ones you don´t want but the ones you are waiting for – arrive with delay or not at all.

So then let´s create own notification bot that will notify you exactly if it required. Can´t be that difficult, can it?

No — it´s even easier that expected!

Preparation

First step: let´s google a little bit about automation frameworks. After a few hours of reading I figure out there is a lot of different automation frameworks for different programming languages… To simplify the selection process, I closed all my tabs and chose Python as the programming language and Selenium as the framework.

If you ask why the two of all, because due to the light syntax of Python, the language is ideal for building fast and easy PoC and if you are talking about web tests or automation is always Selenium mentioned — I also heard a lot about this framework and finally I can try it out :)

Installation

At first you need to install a IDE of your chose (I use PyCharm) and Python (I use Anaconda). I will not go deeper into details because there are many tutorials and recommendations available in WWW.

Packages

Three packages are required:

1. selenium — web automation framework
2. webdriver-manager — browser drivers that required by selenium
3. smtplib (mostly already pre-installed) — Simple Mail Transfer Protocol

Installation via Anaconda

Open Anaconda3 CLI and run Conda-Packetmanager installation commands:

conda install -c conda-forge selenium
conda install -c conda-forge webdriver-manager

*stmplib is already pre-installed in Anaconda

Implementation

Let´s start with the interested part now. In the following example we will implement a simple notification bot for Amazon that will check for the provided URL the product availability and as soon the product is available, either an email or SMS should be sent.

Bot implementation

We begin with a new Python class bot.py and implement the method check_product_by_url with variables url for required product and email/sms where the notification should be forwarded.

class Bot:
    def __init__(self, url, email):
        self.url = url
        self.email = email
    def check_product_by_url(self):

Then it is required to initialize, configure and install driver for browser (I use Chrome) that will be used through the program.

options = Options()
options.headless = False
options.add_experimental_option("detach", True)
browser = webdriver.Chrome(ChromeDriverManager().install(),  
          options=options)
browser.maximize_window()

The main functionality is to check the availability of product. But how can we identify if the product is available or not?

With selenium framework and webdriver we can crawl the page and access or check the HTML-Elements by id, xpath, name, tag, … Ok then let´s go to Amazon and analyze the HTML-Elements of the page (press F12 on the page)

Great! There is a HTML-Element div id=”availability" present that contain information about on stock.

Let´s implement the function from Selemium to check the presence of required HTML-Element. In addition, I added a delay to ensure that the page was actually loaded. After checking the bot can assume one of the two states:

1. element is available -> on stock text available -> send notification | try again
2. element is not available -> reload & try again

while True:
    try:
        availability_div = WebDriverWait(browser, 10).until(
                           EC.presence_of_element_located((By.ID,                       
                           'availability')))

        if on_stock_text in availability_div.text.lower():
           print("Product is available")
           # send email notification
           Alert.email_alert(browser.title + " is available !!!",        
                             browser.current_url, self.email)
           browser.quit()
           break
        else:
           browser.execute_script("location.reload(true);")
    except:
        # reload browser and try again
        browser.execute_script("location.reload(true);")

*actual it is not really a delay, the webdriver will check within 10 seconds every half second by default if the element present or not

Notification implementation

For the notifications I use a gmail-email-server. To send the emails from other applications you have to activate Two-Factor-Authentication on you google account and then it is required to generate an app password. If you have no google account then you can quickly create one. The implementation is quite simple:

• initialize and open connection to gmail-email-server

server = SMTP("smtp.gmail.com", 587)
server.starttls()
server.login(<gmail account>, <generated app password>)

• initialize message information

msg = EmailMessage()
msg.set_content(body)
msg['subject'] = subject
msg['to'] = to
msg['from'] = user

• send message and close email-server connection

try:
    server.sendmail(user, to, msg.as_string())
finally:
    server.quit()

If you want to send a SMS instead of email you will find instructions here how to activate SMS notifications from email-server.

Summary

Incredible, these few lines are enough to write a notification bot. But let´s also check how the bot do his work.

At first I will try to run the bot with product URL that not available:

As expected, the bot runs in an endless loop. No let´s change the URL to a product that available:

The bot has finished the process and the notification is sent:

The code is available on GitHub. Try it out, explore it and have fun.