Prefect Server 101: Deploying to Google Cloud Platform

Last month, Prefect open-sourced a piece of its orchestration backend, making available an open source version of our persistence layer and management UI. This has made possible a host of great features for iterating during local development and has helped foster a better understanding of the inner workings of the Prefect workflow engine.

Naturally, the next question many of us have is “how do I deploy this”? The latest Prefect release exposes the necessary configuration options for easily getting the full Prefect orchestration layer running and accessible on any hosted Cloud provider. In this tutorial we will walk through all the necessary steps from A to Z to get the Prefect server up and running on Google Cloud Platform (GCP).

Provisioning resources

To begin with, we need to provision the necessary compute resources. If you don’t have one already, let’s start by creating a Google Cloud Platform project.

Navigate to Compute Engine > VM instances and then click “Create” (you can skip ahead if you’d prefer to use thegsutil CLI to do this):

Create a VM instance using the Google GUI

These settings are up to you. I’m running on Ubuntu 18.04 LTS:

Instance specs will depend on the workload you expect Server to handle — regardless, since the Server stack consists of several applications, including a database and multiple servers, you’ll want to provision enough memory and CPU to make sure these can run at a base level (the micro-tier will not be enough for this).

Note that once you click Create, it can take a few minutes for the machine to be provisioned.

Accessing the machine

To use Prefect server, there are two endpoints that we will need to access from our local machine:

• GraphQL API: this endpoint is the one that workflows are registered through, and that state changes will be communicated to. This is also the endpoint that the UI queries in order to display workflow information (more on this later). When running locally, this is accessible at http://localhost:4200/graphql.
• UI: and of course, we will want an accessible web address for seeing the UI! When running server locally, this is accessible at http://localhost:8080 but when running the UI on a remote machine we will need to expose a new endpoint.

Once your machine is ready, you can view it on the VM Instances page. Navigate to the network details to make sure port 22 is enabled so we can SSH into the machine (it should be enabled by default):

Navigate to the VMs network details page

Check that SSH is enabled in the firewall rules (tcp:22)

If it’s not present in the firewall rules, you’ll need to enable it.

You’ll also want to add firewall rules for the GraphQL server (port 4200) and the UI (port 8080), so your firewall rules page will look something like this:

Next, you can connect to the machine using the GCP SSH client:

Opening the GCP SSH Client

Provisioning Resources with GSUTIL

For the CLI-minded folks, let’s repeat the above steps using gsutil(replacing the bolded portions with your own info and modifying resources as you see fit):

1. Provision the VM

gcloud beta compute --project=YOUR-PROJECT instances create YOUR-INSTANCE-NAME --zone=us-central1-a --machine-type=f1-micro --subnet=default --network-tier=PREMIUM --maintenance-policy=MIGRATE --service-account=YOUR-SERVICE-ACCOUNT@developer.gserviceaccount.com --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append --tags=http-server --image=ubuntu-1804-bionic-v20200414 --image-project=ubuntu-os-cloud --boot-disk-size=10GB --boot-disk-type=pd-standard --boot-disk-device-name=prefect-server --no-shielded-secure-boot --shielded-vtpm --shielded-integrity-monitoring --reservation-affinity=any

2. Allow HTTP traffic (we’ll open other ports later in the next step)

gcloud compute --project=YOUR-PROJECT firewall-rules create default-allow-http --direction=INGRESS --priority=1000 --network=default --action=ALLOW --rules=tcp:80 --source-ranges=0.0.0.0/0 --target-tags=http-server

3. Open the GraphQL port

gcloud compute --project=YOUR-PROJECT firewall-rules create graphql --direction=INGRESS --priority=1000 --network=default --action=ALLOW --rules=tcp:4200,udp:4200 --source-ranges=0.0.0.0/0

4. Open the UI port

gcloud compute --project=YOUR-PROJECT firewall-rules create graphql-1 --direction=INGRESS --priority=1000 --network=default --action=ALLOW --rules=tcp:8080,udp:8080 --source-ranges=0.0.0.0/0

Installing dependencies

Installing Prefect Server requires three dependencies: Python, Docker, and Docker Compose; let’s install these on our VM.

We’ll start with Python.

Installing Python (and Pip)

Update your package information:
sudo apt-get update

Install Python:
sudo apt-get install python3.6

Check that Python was installed correctly:
python3 --version

Checking that Python was installed correctly

Install Pip:

python3 -m pip install --user --upgrade pip

Verify Pip installation:

python3 -m pip --version

Checking that Pip was installed correctly

We’ll also install the Python 3 virtual environment manager:

apt-get install python3-venv

Installing Docker

Update package information:

sudo apt update

Install Docker from the Ubuntu registry:

sudo apt install docker.io

Check that Docker was installed correctly:

docker --version

Checking that Docker was installed correctly

Start Docker:

sudo systemctl start docker

Automate Docker on startup:

sudo systemctl enable docker

You may need to add your user to the docker group to run docker commands without sudo; you can read more about how do that here

Get the Docker Compose repository:

sudo curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

Apply executable permissions:

sudo chmod +x /usr/local/bin/docker-compose

Check that Docker compose was installed correctly:

docker-compose version

Checking that Docker compose was installed correctly

Installing Prefect

Now it’s finally time to install Prefect. You can do this using your favorite package manager or by cloning the repository and installing with pip; we’ll do the latter to get the bleeding edge version of Prefect.

git clone https://github.com/PrefectHQ/prefect.git

Cloning the Prefect repository

Create a virtual environment where we can install Prefect…

python3 -m venv prefect-env

… and activate the virtual environment:

source prefect-env/bin/activate

Change to the Prefect directory and install using pip:

cd prefect && python3 -m pip install .

Verify the Prefect was installed correctly:

prefect version

Checking that Prefect was installed correctly

Starting Prefect Server

Before we start the server infrastructure, we need to modify the UI endpoint so it knows how to access the GraphQL server correctly (note that this requires Prefect version 0.10.3+). We can do this by creating a user config.toml and adding the endpoint of our VM:

mkdir ~/.prefect
touch ~/.prefect/config.toml

Then we’ll add the following to the file we just created (replacing the text in bold with your own IP):

[server]
  
  [server.ui]
    
    apollo_url = "http://YOUR_MACHINES_PUBLIC_IP:4200/graphql"

This tells the Prefect Server UI to access the GraphQL API at port 4200 on your machine.

Save config.toml and exit your VM’s text editor.

You can read more about configuring Prefect in the Prefect docs.

Now we’re ready to spin up Prefect Server! You can do so by running:

prefect server start

Once the images are pulled and spun up, you’ll have output that looks something like this:

Prefect Server Start!

Since we didn’t set up HTTP rules for the UI, you’ll access it at http://YOUR_MACHINES_PUBLIC_IP:8080 , again replacing the bolded portion with the public IP address of your VM.

If all went well, you should see the UI!

Lastly, before you register any Flows with your remote Prefect server, you’ll need to edit your machine’s ~/.prefect/config.toml to point to the remote GraphQL API:

[server]
endpoint = "YOUR_MACHINES_PUBLIC_IP:4200/graphql"

Please continue reaching out to us with your questions and feedback — we appreciate the opportunity to work with all of you!

• join our Slack community for ad-hoc questions
• follow us on Twitter for updates
• visit us on GitHub to open issues and pull requests

Happy Engineering!

— The Prefect Team