Automate your Python script using the “schedule” module

To put things in context, my task was to develop a python script that would scrape data from a webpage, model it, and render it to a csv format. As the data was refreshed daily, the script has to run once a day. Here is the full script (if interested), but the purpose of this blog is not the script itself but rather the automation.

After a few failed attempts to automate the script, I found the schedule python module. It’s a simple, one liner solution, to any automation problem!

Here is a simple example (taken from the modules’ official page):

import schedule
import time

def job():
    print("I'm working...")
schedule.every(10).seconds.do(job)
schedule.every(10).minutes.do(job)

It’s pretty self-explanatory. By adding this line: schedule.every(10).seconds.do(job), the job() function will be initialized to run every 10 seconds.

To go back to my project, and solution, I implemented the schedule module this way:

#Run the Script
#run_script being the function I want to run
sch.every().day.at('05:00').do(run_script)
while True:
    sch.run_pending()
    time.sleep(1)

I had to implement an infinite While loop that would run once every second to get the job done. This line: sch.every().day.at(‘05:00’).do(run_script), didn’t actually run my script. This line is the initialization of the time and frequency of extraction, but it needs to be “called” by: sch.run_pending().

And voila! My script, and data extraction, is ran and completed everyday at 5 AM.

Pros :

• Very simple solution for an automation task
• Practical if the script is to be implemented on a server (my case for this project) because it will run perpetually on it.

Con :

• If you’re running the script locally, the automation will not work if the script is not running.

Thank you for reading!