Introducing: Prefect Universal Deploy

We’re excited to announce our best release yet: Prefect 0.8.0!

We’ve completely redesigned how workflows are registered and orchestrated with Prefect Cloud. We call the new design “Universal Deploy” because from now on, anywhere you can run Python, you can take advantage of Cloud.

In the past, Prefect workflows had to be “built” for a specific environment. The most common way to do so was by using Prefect’s built-in Docker facilities to build, push, and later recover a container that represented the flow artifact. The process works well for packaging workflows (and their dependencies) for remote execution, but after observing many Prefect Cloud onboards, we found that the lengthy process could be unpleasant for users who didn’t want or didn’t know how to work with Docker.

We challenged ourselves to design a completely new deploy process that was 1) lightning fast and 2) didn’t require Docker. (You can read more about our thought process in Prefect Improvement Notice #13.)

The result is Universal Deploy. With Prefect 0.8.0, the moment you finish creating a Flow object, you can instantly register it with Prefect Cloud and immediately begin orchestrating its execution, even right there in your running Python process.

Registering a Flow with Prefect Cloud and running a Local Agent in Process

Universal Deploy works with local storage, S3, GCS, Azure, Docker containers, and more. As long as your flow is accessible in one of those locations, you can instantly spin up a Prefect Agent that automatically interfaces with Cloud to decide how and when the flow should run.

This means your flows are as portable as your imagination, and achieves our goal of fully abstracting the flow from its execution environment. At Prefect, we always strive to offer a workflow system that is lightweight, with sensible defaults, while simultaneously allowing deep customization. Our Hybrid Engine allowed us to successfully separate workflow execution from orchestration, and Universal Deploy builds on that as one of the final pieces of the puzzle.

Interested in kicking the tires? Sign up for a free Scheduler account and check out our new Hello World tutorial!

Note: there are breaking API changes introduced in this release. If you upgrade to 0.8.0 you may need to change your Flow code accordingly (see below for details).

New Features

A New Local Agent

Prefect 0.8.0 comes with a brand new default agent, capable of handling all of our new flow storage types (described below). This agent runs your workflows in a subprocess, and can be provided any number of import paths to accommodate your Flow’s script dependencies:

prefect agent start -p /root/scripts/ -p ~/Developer/utilities

The new Local Agent will also auto-label itself with the hostname of the machine on which its running, along with all of the various storage labels described below.

New Cloud-compatible Storage Classes

Prefect 0.8.0 also comes with four brand new storage options for your Flows:

• Local: Local storage is the new default storage class, and is designed to save your flows in a configurable directory location on your local filesystem (the default location is ~/.prefect/flows). Note that Flows registered with this storage will be auto-labeled with the hostname of the machine where the Flow is stored.
• S3: S3 storage is designed to save your Flow to a configurable S3 bucket. Note that Flows registered with this storage will be auto-labeled with the s3-flow-storage label.
• GCS: GCS storage is designed to save your Flow to a configurable Google Cloud Storage Blob. Note that Flows registered with this storage will be auto-labeled with the gcs-flow-storage label.
• Azure: Azure storage is designed to save your Flow to a configurable Azure Cloud Storage Blob. Note that Flows registered with this storage will be auto-labeled with the azure-flow-storage label.

Both S3 and GCS allow you to share and run your Flow on any machine which is properly authenticated.

Breaking Changes

Prefect 0.8.0 also ships with a few breaking changes, all designed to make registration of flows with Cloud easier; within each section we describe how to preserve the behavior you were accustomed to in previous versions.

Agent Changes

The old LocalAgent has been renamed to the DockerAgent, and is no longer the default agent when running prefect agent start. To run the old default agent:

prefect agent start docker

Default Storage

The old default storage class of Docker has been changed to the new Local storage class. This means that deployment code which relied on default settings like

flow.deploy("My Project", registry_url="my/docker/registry")

will no longer work unless you update your defaults. To reset your default storage class back to Docker, include the following in your Prefect user configuration file:

[flows]
    [flows.defaults]
    [flows.defaults.storage]
    # the default storage class, specified using a full path
    default_class = "prefect.environments.storage.Docker"

Alternatively, you can import Docker directly:

from prefect.environments.storage import Docker
flow.storage = Docker(registry_url="my/docker/registry")

Deploy -> Register

Most users prefer to think of their Prefect agents as the actual deployment, while sending the flow metadata to Cloud is more of a registration step. Consequently, to better match user expectations we have deprecated flow.deploy in favor of the more descriptive flow.register.

Please continue reaching out to us with your questions and feedback — we appreciate the opportunity to work with all of you!

• join our Slack community for ad-hoc questions
• follow us on Twitter for updates
• visit us on GitHub to open issues and pull requests

Happy Engineering!

— The Prefect Team