Getting Started with Prefect Flows

Welcome to using Prefect! This guide is here to help you step through 7 simple steps to get started running a simple flow using Prefect Cloud and your local machine. By the end of this guide, you will have downloaded and installed Prefect and its dependencies, have a basic understanding of Prefect core concepts (tasks, flows, agents, cloud), and used some light navigation with terminal and Prefect documentation.

If you’re thinking whoa…that’s a lot of technical terms, let’s break this down a little starting with some basic terminology.

• Your local machine is your laptop. (For the purpose of this blog, we’ll assume you’re using a MacBook.)
• Flows consist of tasks. A task is a Python function that does stuff.
• Flows are workflows. They are defined in Python files on your local machine. Workflows describe the order in which the tasks are executed.
• Agents run on your local machine and dynamically execute flows when there is a flow scheduled using Prefect Cloud to run.
• Prefect Cloud is a hosted UI to orchestrate workflows and have visibility into the status of your workflows.

That’s really all you need as a foundation to write Prefect flows.

Now let’s get started:

Step 1: Installing Prefect

There are multiple ways to install Prefect listed in the Prefect documentation.

To open your terminal, use the keyboard shortcut Command-Space Bar and type Terminal to search for the Terminal application on your MacBook.

The terminal is an application on your MacBook for using a command-line interface with your machine. Sometimes it is referred to as the command line or console.

Now we can install Prefect by typing one of the install commands into our terminal and hitting Enter to run the command. The install commands use either Pip, Conda, or Pipenv.

To check that the install was successful run this command in the terminal:

prefect version

You should see an output on the terminal that shows which version of prefect you installed successfully

Lastly, let’s check which version of Python our MacBook is using. Prefect requires a Python version of 3.6 or higher. On your system you may need to run scripts with python3 instead of just python.

python --version
python3 --version

Step 2: Install a code editor

Prefect flows are built using Python files and a code editor is an easy way to access and edit those files. You can install a code editor of your choice. My favorite is VSCode.

Step 3: Create a folder to place demo flows

Now we need to create a location on our machine where our flows will live. Let’s use the terminal and some terminal basics to make a folder.

Inside your terminal navigate to your root (home) directory:

cd ~

To see the files or folders inside the root directory type the command for list:

ls

Now let’s create a new folder called flows:

mkdir flows

Let’s check that your new folder is now in the root directory by listing the files or folders again:

ls

Now navigate into that folder by typing:

cd flows

Lastly, let’s also open our new flows folder in the VSCode.

Step 4: Create a flow and run it locally

We should have two main screens open on our desktop (terminal and vscode) both open to the flows folder. Using VSCode, create a new file named first_flow.py

Ending a file with .py means it’s a Python file. Prefect flows are built with Python files.

Next, let’s copy and paste this code snippet into your new Python file (first_flow.py). Make sure you save your file.

What does this snippet do? This snippet is a basic flow named “hello-flow”. It first imports the Prefect library to use. Then it declares a task. A task is a Python function. Functions do stuff. We use the functional task API decorator @task above the function to declare it is a task. Then we declare our flow and name it "hello-flow". Our flow will call the task (function). Lastly, we call our flow with flow.run()

To run a Python file, you type Python before the file name in the terminal. Let’s give this a try in your terminal. (Make sure you are in the flow folder.)

python first_flow.py

The output inside your terminal should look like this:

🥳 You just ran your first flow on your local machine!

Next Up: Running your flow with Prefect Cloud

Before we start running our flow with Prefect Cloud, let’s quickly understand what that means. Right now we have our flow running locally on our local machine (MacBook).

Running the flow with Prefect Cloud uses what we call the hybrid model. Flows will still be built using Python files on your local machine and still consist of tasks (functions). But instead of running the flow using the terminal, we will register the flow to Prefect Cloud and send metadata about the flow to our cloud project. Then we will trigger flow runs from Prefect Cloud and use agents that run on our machine to look at Prefect Cloud. When an agent sees a scheduled flow run, they will dynamically spin up FlowRuns that send the metadata status back to Prefect Cloud.

Step 5: Hook up Prefect Cloud account to local machine

Start by logging in or creating a Prefect Cloud account. It’s free.

Next, let’s connect our local machine to use the cloud by running this command in our terminal:

prefect backend cloud

We need to create an API key associated with our account to use for authentication and connecting our local machine to the cloud. Create an API Key by navigating to the user menu in the top right corner and going to Account Settings → API Keys → Create An API Key. Make sure to copy the newly created key. Inside the terminal, login by replacing <YOUR-KEY> with the one you just copied:

prefect auth login --key <YOUR-KEY>

Step 6: Register your flow to your cloud account

Create a project in your cloud account either through the UI or by typing in the terminal:

prefect create project "tutorial"

Inside your code editor, change line 12 from flow.run() to flow.register(project_name=”tutorial”) and save your file.

Run your flow:

python first_flow.py

The output should look different this time and instead show the URL where the flow is registered to. If you copy and paste the URL into your browser you will see your flow on your Prefect Cloud account.

Prefect Cloud now knows about your flow!

Step 7: Run your flow

At the top of the flow page, click Quick Run to run your flow.

Click on the pop-up message in the lower right to visit your running flow.

Wait.. why is the flow not running? Notice how your flow is scheduled but isn’t actually running yet. This is because you don’t have an agent running that looks for scheduled flows.

Agents will dynamically spin up flow runs when flows are scheduled in Prefect Cloud and send the metadata status to the cloud.

Start an agent by typing the following into your terminal:

prefect agent local start

In your terminal window, you will see a local Prefect agent spin up.

Now when you go back to your flow run, you should see the status is Successful! You’ve just run a flow on Prefect Cloud!

To stop the local agent when you’re all done, type Ctrl-C. But, if you want to keep experimenting, or schedule the flow to run on a regular basis, just keep the agent running.

Next Steps: Explore Prefect

Now that you’ve run a simple flow on your laptop using Prefect Cloud, you’re ready for more. Here are a few things to try:

• Run the ETL example
• Creating a schedule for your flow
• Storing your flow (code) in a repository
• Using a different type of agent
• Explore the cloud UI

Happy engineering!