Set a Clock to Run Python Scripts on AWS EC2 with Lambda and CloudWatch & Upload Data to DynamoDB— PART A

Before this article starts, I will assume your EC2 instance is ready to go. If not, you can search for some tutorials to teach you how to set up an EC2 instance.

I am working with a startup to collect daily data from Reddit API. Since I am very lazy, I don’t want to manually run my scripts every day and store data on my local server (my MacBook has very limited flash storage :D). Then I was thinking if there is any way to automatically run my python scripts on a daily basis. I asked my friend who is really really really intelligent and works in AWS as a consultant right now. She gave me a great solution to solve this problem. The solution workflow is that CloudWacth triggers Lambda function (PART C)->Lambda invokes the EC2 instance (PART B)-> EC2 runs Python scripts as it starts (PART A)-> Upload data to DynamoDB from Python scripts (PART D). I will insert links to other parts in the future.

Let’s work on how to make python scripts start to run itself as EC2 starts. The idea here is to create a shell script called startup.sh, write command lines to run python scripts in startup.sh and some command lines to run the shell script in /etc/rc.local. The running process is that EC2 started -> run startup.sh -> run Python scripts.

A1. Hit Connect

A2. Select the red box

A3. Enter Python -V to check your python version. Type sudo yum install python37 to install Python3.

A4. Create a virtual environment for Python to install packages in the future — type python3 -m venv app/env to create the virtual env, enter source ./app/env/bin/activate to activate virtual env. Then you can type python -m pip install {package_name} to install packages you need.

A5. Create a test file by typing vim Hello.py and hit I to insert commands -print(“Hello”). To save and quit the file, hit esc and type :wq

A6. To check if the file exists, you can type ls

A7. Create startup.sh as below.

A8. You will activate the virtual env and run python3 scripts by entering the commands shown below. Then save and quit by hitting ESC and typing :wq

A9. Type sudo vim /etc/rc.local to access root file

A10. You will see this interface. Hit I to insert commands.

A11. Type exec 1>/tmp/rc.local.log 2>&1 to check logs. Type set -x and sh /home/ec2-user/startup.sh to run startup.sh when ec2 starts. Don’t forget to add exit 0 in the end. Then save and quit by hitting esc and typing :wq

A12. Type sudo chmod +x /etc/rc.d/rc.local to ensure script will be executed during boot. Then type sudo reboot to reboot the ec2.

A13. Close the terminal and connect the terminal again. Type cat /tmp/rc.local.log to check if we have hello in the log.

You had me at hello :)

I will write more tutorials in the future. If you have alternative ways to make the process better, leave comments below.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store