How to run multiple python scripts on AWS EC2 instance

supervisor

Problem: How to run multiple python scripts on a single AWS ec2 instance.

Once I was assigned to build multiple Data Scrapers which will run continuously to extract data from various sources. Another requirement was to scrape data every 15 seconds, and because of that lambda’s were out, as they don’t support second latency. So the only solution was to run scripts individually on AWS ec2 instance using crontab. But we can only run a single script using crontab, so it was out too. Then I came across supervisor which can be used to run multiple python scripts at the same time, and it also restarts the script, if there is an error.
I looked on the internet but there was not enough material on how to run supervisor on ec2. So here is a short blog post that can be helpful to you.

Required Steps:
1. Spin up ec2 instance(Amazon Linux 2 AMI).
2. Install Supervisor.
3. Send Python scripts to the server.
4. Configure the supervisor.
5. Run Supervisor.
6. Status check.

  1. Spin up ec2 instance(Amazon Linux 2 AMI).
    Login to your AWS web service account and spin up an Amazon Linux 2. I’m using the free tier eligible t2.micro with default configuration here for demo purposes.
Amazon Linux 2

To connect to the ec2 instance I’m using putty on windows. There are a couple of articles on the internet on how to ssh into the remote server using putty on windows/Linux. Once you ssh into the server we can proceed to the second step.

2. Install Supervisor
Once you log into the server before you install the supervisor you have to install pip. Install pip using the command below.

pip-installation

After pip is installed you can install supervisor using the command below.

supervisor-install-command
supervisord-setup

Once it is installed we can move to the next part of this project.

3. Send Python scripts to the server
To send the python scripts to the server I’m using WinSCP. I named my scripts googlescrap1.py and googlescrap2.py. You can have more than two scripts, depending on your requirements. These code snippets are taken from geeksforgeeks website.

googlescrap1.py
This script runs every 5 seconds.


# Import the beautifulsoup
# and request libraries of python.
import requests
import bs4
import time
def main():
# Make two strings with default google search URL
# 'https://google.com/search?q=' and
# our customized search keyword.
# Concatenate them
text= "geeksforgeeks"
url = 'https://google.com/search?q=' + text
# Fetch the URL data using requests.get(url),
# store it in a variable, request_result.
request_result=requests.get( url )
# Creating soup from the fetched request
soup = bs4.BeautifulSoup(request_result.text,
"html.parser")
print(soup)
if __name__ == "__main__":
while True:
main()
time.sleep(5)

googlescrap2.py
This script run every 10 seconds.

# Import the beautifulsoup
# and request libraries of python.
import requests
import bs4
import time
def main():
# Make two strings with default google search URL
# 'https://google.com/search?q=' and
# our customized search keyword.
# Concatenate them
text= "geeksforgeeks"
url = 'https://google.com/search?q=' + text
# Fetch the URL data using requests.get(url),
# store it in a variable, request_result.
request_result=requests.get( url )
# Creating soup from the fetched request
soup = bs4.BeautifulSoup(request_result.text,
"html.parser")
print(soup)
if __name__ == "__main__":
while True:
main()
time.sleep(10)

I then created a folder on the ec2 instance to store my scripts us thing the below command. cd into the scripts and store both the scripts there.

mkdir scripts

Make a python 3 environment and activate it using the commands below.

new-env
activate-env

To install dependencies you can use the regular pip command after activating the environment.
After that cd into the scripts folder and run python googlescrap1.py and googlescrap2.py. If they are running that means you are all set to move to the next step and set up the configuration file of the supervisor.

4. Configure the Supervisor

To configure the supervisord.conf file use any editor. I’m using nano.


sudo nano /etc/supervisord.conf

After that go to the end of the file and paste these lines of code there. This will run the googlescrap1.py script.

[program:scrapper1]
command=/home/ec2-user/env/bin/python3 googlescrap1.py
directory=/home/ec2-user/scripts
autostart=true
autorestart=true
stdout_logfile=/home/ec2-user/scripts/googlescrap1.log
redirect_stderr=true
stdout_logfile_backups=0
stdout_logfile_maxbytes=0

a. scrapper1 is the name of the program that you want to run. You can name it anything.
b. command is the python command to run the googlescrap1.py using the environment that we created above.
c. directory is the location where the supervisor will look for the scripts.
d. stdout_logfile is the location of the log file associated with the scripts.

To run the second script give a line break and paste the below code. Which is essentially the copy of the first one. The only thing that changed is the name of the program, script, and logfile.

[program:scrapper2]
command=/home/ec2-user/env/bin/python3 googlescrap2.py
directory=/home/ec2-user/scripts
autostart=true
autorestart=true
stdout_logfile=/home/ec2-user/scripts/googlescrap2.log
redirect_stderr=true
stdout_logfile_backups=0
stdout_logfile_maxbytes=0

After this setup save the supervisord.conf file and move to the next step.

5. Run Supervisor
To run supervisor we will use the command below and pass the above formed .conf file to it.

supervisord -c /etc/supervisord.conf

6. Status Check
To check the status use the below command


supervisorctl

You’ll see output like this:

supervisorctl-output

Voila, both the scripts are running. You can stop, restart, add multiple files using the supervisorctl.

I hope you enjoyed this blog. Thank you.

Cloud Data Engineer. Tech Enthusiast.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Tutorial: Hosting Slate Documentation On IPFS

Your Introductory Guide to the Kanban Board

Programming, Positivity, & Pain: How do they connect?

Will Fasted Cardio Help You Burn More Fat?

(Re) Building a Home Lab

Insights Of Getting Certified Ethical Hacker Training

Programming is boring art

Who should be able to create a SharePoint Site?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Yadvendar Singh

Yadvendar Singh

Cloud Data Engineer. Tech Enthusiast.

More from Medium

Building a Python Chatbot — Part I

Brief Overview of Python Mobile App Development

Web scraping with python

Check All The Links Of A Given Website Using Python