Schedule a Python Script to Run Periodically on a Cloud Server

Cloud servers have a significant advantage over local setups: they’re always on. This means that your Python scripts can run uninterrupted 24/7. In this post, we’ll explore setting up a Python script to run periodically on 8 different cloud services.

Table of Contents

• Introduction
  - Python script
  - Requirements
  - Environment Variables
• Cloud Services
  - GCP Cloud Functions
  - AWS Lambda
  - AWS EC2
  - DigitalOcean
  - Heroku
  - Render
  - Railway
  - Vercel

Introduction

When scheduling a Python script to run periodically on a cloud server, there are three main components that we need to consider:

1. The Python script itself
2. Requirements (Python packages used in the script)
3. Environment variables (API keys, etc.)

To demonstrate a complete example, we will use a script that uses the requests package and one environment variable. This way, when we schedule the script on various cloud services, we can see how to handle the requirements and environment variables.

Python Script

import os
import requests


def main():
    """
    Gets detailed information about a specified cryptocurrency. 
    The data includes the coin's ranking, supply details, and current price.
    """

    coin_id = os.getenv("COIN_ID")

    response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
    coin_data = response.json()
    
    print(coin_data)
    return coin_data


if __name__ == "__main__":
    main()

Requirements

requests

Environment Variables

COIN_ID=bitcoin

Cloud Services

Cron expressions are used to specify schedules on the cloud services presented below. You don’t have to know anything about cron expressions to follow along. However, if you want to learn more about them, check out crontab.guru.

We will use * * * * * as the cron expression for most cloud services. This expression will cause the script to run every minute. Some cloud services don't allow running jobs as frequently as every minute. In those cases, we will use a different cron expression.

GCP Cloud Functions

First, we are going to create a Cloud Function that will run our script. Then we will create a Cloud Scheduler job that will trigger the Cloud Function every minute.

Create a Cloud Function​

Go to Cloud Functions and click “CREATE FUNCTION”.

Give it a name, and select “HTTPS” as the trigger type.

Open the “Runtime, build, connections and security settings” dropdown and under the “RUNTIME” tab locate “Runtime environment variables”. Add COIN_ID variable with value bitcoin. You can leave everything else as default and click "NEXT".

Set runtime to “Python 3.12” (or whichever Python version you want) and entry point to “main”. Paste the following code in main.py:

import functions_framework
import os
import requests

@functions_framework.http
def main(request):
    coin_id = os.getenv("COIN_ID")

    response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
    coin_data = response.json()
    
    print(coin_data)
    return coin_data

Add requests to requirements.txt so that it looks like this:

functions-framework==3.*
requests

We are done with the configuration. Click “DEPLOY”.

It will take some time for deployment to complete. Once it is done, we can schedule our Cloud Function to run every minute via Cloud Scheduler. Copy the URL of the Cloud Function. We will need it in the next step.

Create a Cloud Scheduler Job​

Go to Cloud Scheduler and click “CREATE JOB”.

Give it a name and set frequency to “* * * * *” (every minute). Set timezone to whatever you want. Click “CONTINUE”.

Set target type to “HTTP”. Enter the URL of the Cloud Function that you copied in the previous step. For “Auth header” select “Add OIDC token”. For “Service account” select “Default compute service account”. Click “CREATE”.

That’s it. The Cloud Function will run every minute. To monitor runs, check the “Logs” tab of your Cloud Function.

AWS Lambda

First, we are going to create a Lambda that will run our script. Then we will create an EventBridge schedule that will trigger Lambda every minute.

Create a Lambda​

Go to AWS Lambda and click “Create function”.

Give it a name, set runtime to “Python 3.12” (or whichever Python version you want) and click “Create function”.

Under the “Configuration” tab, locate “Environment variables” and add COIN_ID variable with value bitcoin.

Even though Lambda has an inline code editor where we can write our script, it doesn’t allow us to indicate our requirements (in our case the requests package). Therefore, we will have to create a zip file with our script and installed Python packages and upload it to Lambda.

Create a new directory. Open your terminal and cd into it. Execute the following:

pip install requests --target .

This will install the requests package into the root of the directory. Then, create a file called "lambda_function.py" and paste the following code into it:

import os
import requests


def lambda_handler(event, context):
    coin_id = os.getenv("COIN_ID")

    response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
    coin_data = response.json()

    print(coin_data)

Next, we need to create a zip file of the contents of the folder. If you are on Linux or Mac, you can do:

zip -r package.zip *

The name of the zip file is not important. However, the structure of the zip file is important. It should have a flat directory structure, with “lambda_function”.py and installed package folders installed at the root:

package.zip
|- lambda_function.py
|- requests
|- urllib3
|- certifi
...

Go back to Lambda and upload the zip file.

Lambda is now ready to run our script. Next, we need to create an EventBridge schedule.

Create an EventBridge Schedule​

Go to AWS EventBridge Scheduler and click “Create schedule”.

Give it a name.

Under “Schedule pattern”, select “Recurring schedule” for “Occurrence”. Enter “* * * * ? *” (every minute) for “Cron expression”. Select “Off” for “Flexible time window” and click “Next”.

Under “Target detail”, select “AWS Lambda” and under “Invoke” select your Lambda function. Click “Next”.

Click “Next” on the next page and then “Create schedule” on the last page.

That’s it. Lambda will run every minute. To monitor runs, check the “Monitor” tab of your Lambda and click “View CloudWatch logs”.

AWS EC2

We need to start by launching an EC2 instance. Go to AWS EC2 and click “Launch instance”.

Give it a name and select “Ubuntu” as the AMI.

Scroll down to create a new key pair. We will use this key pair to ssh into the instance. Give it a name and click “Create key pair”. The key pair will be downloaded to your computer.

Click “Launch instance”.

Now we need to ssh into the instance. For that, we need to know the public IP address of the instance. Go to the new instance’s page and copy the public IP.

Open your terminal and cd into the directory where the the key pair was downloaded. If you are on Linux or Mac, make sure that the key pair is not publicly accessible by executing:

chmod 400 <key-pair-name>

Then, execute the following command to ssh into the instance:

ssh -i <key-pair-name> ubuntu@<public-ip>

Replace <key-pair-name> with the the name of your key pair and <public-ip> with public IP of your instance. For example:

ssh -i scheduler.pem ubuntu@54.234.83.151

Now we need to create script.py, .env for environment variables, and requirements.txt for requirements. We also need to create a virtual environment and install the requirements in it. Virtual environment allows us to install Python packages specific to our script without affecting the global Python installation.

Let’s start by creating the virtual environment. First, we need to install the python3-venv package:

sudo apt-get update
sudo apt-get install python3-venv -y

Then, we need to create the virtual environment and activate it:

python3 -m venv venv
source venv/bin/activate

Next, let’s create requirements.txt so that we can install the requirements in the virtual environment. We will use Vim editor to create the file:

vi requirements.txt

This will open the editor. Press i to enter insert mode. Then, paste the following:

requests
python-dotenv

Press esc to exit insert mode. Then, type :wq and press enter to save and exit Vim.

python-dotenv is a package that allows us to read environment variables from a .env file in our Python script.

Let’s install the requirements in the virtual environment:

pip install -r requirements.txt

Now we need to create the .env file.

vi .env

Press i to enter insert mode. Then, paste the following:

COIN_ID=bitcoin

Press esc to exit insert mode. Then, type :wq and press enter to save and exit Vim.

Finally, let’s create the Python script:

vi script.py

Press i to enter insert mode and paste the following:

import os
import requests
from dotenv import load_dotenv


load_dotenv()

def main():
    coin_id = os.getenv("COIN_ID")

    response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
    coin_data = response.json()
    print(coin_data)


if __name__ == "__main__":
    main()

Run the script manually to make sure that everything is set up correctly:

python script.py

It should print the data for Bitcoin.

We are going to schedule the script to run every minute. We will use crontab for that. crontab is a config file that describes the commands to run periodically. Let’s open the crontab editor:

crontab -e

You will be asked to select an editor. Select 2 for Vim. This will open the editor. Press i to enter insert mode. Use the arrow keys to go to the last line and paste the following:

* * * * * . /home/ubuntu/venv/bin/activate && python3 /home/ubuntu/script.py >> /home/ubuntu/logs.txt

Press esc to exit insert mode. Then, type :wq and press enter to save and exit Vim.

This will run script.py every minute and log the output to logs.txt.

use journalctl -u cron to check each run of the script.

After the first run, logs.txt will be created and you will be able to see the output of the script. To view the logs, execute cat logs.txt.

We’ve successfully scheduled our script to run every minute. You can safely exit the ssh session by executing exit. The script will continue to run in the background.

DigitalOcean

We will create a Function on DigitalOcean by creating Function files locally and pushing them to DigitalOcean via doctl. We will create the following files: a Python script (__main__.py), a requirements file (requirements.txt), a build script (build.sh), and a DigitalOcean config file (project.yml). Only the config file will be in the root directory. The rest of the files will be in a packages/core/cryptocurrency_checker subdirectory.

The directory structure must look like this:

|- packages/
|  |- core/
|  |  |- cryptocurrency_checker/
|  |  |  |- __main__.py
|  |  |  |- requirements.txt
|  |  |  |- build.sh
|- project.yml

Start by creating the root directory.

Then, create project.yml and paste the following:

packages:
  - name: core
    functions:
      - name: cryptocurrency_checker
        runtime: 'python:default'
environment:
  COIN_ID: bitcoin

Next, create packages directory. Within packages, create core directory. Within core, create cryptocurrency_checker directory. Inside the cryptocurrency_checker directory, create __main__.py and paste the following:

import json
import os
import requests

def main(args):
  coin_id = os.getenv("COIN_ID")

  response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
  coin_data = response.json()
  print(coin_data)
  
  return {
    'body': {
      'response_type': 'in_channel',
      'text': json.dumps(coin_data)
    }
  }

In the same directory, create requirements.txt and paste the following:

requests

Finally, create build.sh in the same directory and paste the following:

#!/bin/bash

set -e

virtualenv virtualenv
source virtualenv/bin/activate
pip install -r requirements.txt
deactivate

If you are on Linux or Mac, build.sh must be executable. To make it executable, run the following:

chmod +x build.sh

The files are ready. But, we need to install doctl and set it up. Install doctl by following the instructions here: https://docs.digitalocean.com/reference/doctl/how-to/install/.

Then go to Applications & API on DigitalOcean and generate a new token.

Make sure to check the “Write” box.

To authenticate, copy the token and execute doctl auth init. Paste the token when prompted.

Next, we have to create a serverless namespace. Namespaces are used to group resources together.

doctl serverless namespaces create --label my_namespace --region nyc1

After creating the namespace, we need to connect to it:

doctl serverless connect

At this point, we are ready to deploy our files. Go back to the root directory (where project.yml is located) and execute the following:

doctl serverless deploy . --remote-build

Once the deployment is complete, we can schedule the function to run periodically.

To create a schedule, go to DigitalOcean Functions, open the namespace that we created, and click on your function.

Click on the “Triggers” tab and then “Create Trigger”.

Give it a name, set the cron expression to “* * * * *” (every minute), and save.

That’s it. The function will run every minute.

Heroku

Start by creating an new app on Heroku.

Under the Settings tab, locate and click on “Reveal Config Vars”. Add COIN_ID variable with value bitcoin.

We will be doing deployment via Heroku Git using the Heroku CLI. To install Heroku CLI, follow the instructions here: https://devcenter.heroku.com/articles/heroku-cli#install-the-heroku-cli

Once you have Heroku CLI installed, log in by executing heroku login.

We are ready to create and push files to Heroku Git. We will be pushing a Python script (script.py), a requirements file (requirements.txt) and a Heroku config file (Procfile).

The directory structure must look like this:

|- script.py
|- requirements.txt
|- Procfile

Start by creating the root directory.

Then, create script.py and paste the following:

import os
import requests


def main():
    coin_id = os.getenv("COIN_ID")

    response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
    coin_data = response.json()

    print(coin_data)
    return coin_data


if __name__ == "__main__":
    main()

Next, create requirements.txt and paste the following:

requests

Finally, create Procfile and paste the following:

script: python script.py

To push the files to Heroku, execute the following commands:

git init
heroku git:remote -a function-app
git add .
git commit -am "first commit"
git push heroku master

The app is now deployed. We need to schedule it to run periodically. To do that, we need to install the Cron To Go Scheduler add-on. Execute the following command:

heroku addons:create crontogo:free-trial

To set the schedule, go to your app on Heroku, and open the addon:

Click on “Add Job”.

Set the schedule to “* * * * *” (every minute). Set the command to python script.py and click "Add job".

That’s it. The script will run every minute. To check the logs, execute the following in the terminal:

heroku logs --tail

Render

We will create a Cron Job on Render by connecting a GitHub repository. This repository will contain a Python script (script.py) and a requirements file (requirements.txt).

The directory structure must look like this:

|- script.py
|- requirements.txt

Start by creating the root directory.

Then, create script.py and paste the following:

import os
import requests


def main():
    coin_id = os.getenv("COIN_ID")

    response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
    coin_data = response.json()
    
    print(coin_data)
    return coin_data


if __name__ == "__main__":
    main()

Next, create requirements.txt and paste the following:

requests

We are ready to push the files to GitHub. Create a GitHub repository and follow the instructions on GitHub to push the files to it.

Now we need to create a Cron Job on Render by connecting to the GitHub repository that was just created. Go to Create a new Cron Job on Render. Select Build and deploy from a Git repository and click “Next”.

Select the GitHub repository that you created and click “Connect”.

Give your Cron Job a name, set the schedule to “* * * * *” (every minute), and set the command to python script.py. Scroll down to set an environment variable. Add COIN_ID variable with value bitcoin. Click "Create Cron Job".

That’s it. The Cron Job will run every minute.

Railway

We will create a project on Railway by connecting a GitHub repository. This repository will contain a Python script (script.py), a requirements file (requirements.txt), and a Railway config file (Procfile).

The directory structure must look like this:

|- script.py
|- requirements.txt
|- Procfile

Start by creating the root directory.

Then, create script.py and paste the following:

import os
import requests


def main():
    coin_id = os.getenv("COIN_ID")

    response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
    coin_data = response.json()
    
    print(coin_data)
    return coin_data


if __name__ == "__main__":
    main()

Next, create requirements.txt and paste the following:

requests

Finally, create Procfile and paste the following:

script: python script.py

We are ready to push the files to GitHub. Create a GitHub repository and follow the instructions on GitHub to push the files to it.

Now we need to create a project on Railway by connecting to the GitHub repository that was just created. Go to railway.app and create a new project.

Select the GitHub repository that you created and click “Deploy now”.

Now that the project is deployed, we need to add the COIN_ID environment variable. Click on the Variables tab and add COIN_ID variable with value bitcoin.

Next, we have to create a schedule. Click on the Settings tab and scroll down to “Deploy” section. Click on “+ Cron Schedule”.

Railway allows for scheduling scripts to run every 15 minutes at most — not every minute. Therefore, we will use */15 * * * * (every 15 minutes) as the schedule.

That’s it. The script will run every 15 minutes.

Vercel

We will create a project on Vercel by connecting a GitHub repository. This repository will contain a Python script (index.py), a requirements file (requirements.txt), and a Vercel config file (vercel.json). The Python script will be in an api directory. The requirements file and the Vercel config file will be in the root directory.

The directory structure must look like this:

|- api/
|  |- index.py
|- requirements.txt
|- vercel.json

Start by creating the root directory.

Then, create vercel.json in the root directory and paste the following:

{
    "redirects": [{ "source": "/", "destination": "/api" }],
    "crons": [{ "path": "/api", "schedule": "0 0 * * *" }]
}

Notice that we are defining a cron job within the crons array. Vercel’s Hobby plan allows for scheduling scripts once per day at most — not every minute. Therefore, we are using 0 0 * * * (every day at midnight) as the cron expression.

Next, create requirements.txt in the root directory and paste the following:

requests

Finally, create api directory inside the root directory and create index.py inside the api directory. Paste the following into index.py:

from http.server import BaseHTTPRequestHandler
import os
import requests

class handler(BaseHTTPRequestHandler):

    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-type','text/plain')
        self.end_headers()
        
        coin_id = os.getenv("COIN_ID")

        response = requests.get(f'https://api.coincap.io/v2/assets/{coin_id}')
        coin_data = response.json()

        print(coin_data)
        return coin_data

We are ready to push the files to GitHub. Create a GitHub repository and follow the instructions on GitHub to push the files to it. Make sure to initialize the repository in the root directory.

Now we need to create a project on Vercel by connecting to the GitHub repository that was just created. Go to vercel.com and create a new project by importing the repository.

When the repository is imported, you will be able to configure the project. The only thing that we need to do is add the COIN_ID environment variable with value bitcoin and deploy.

That’s it. The script will run every day at midnight.