Getting started with operationalizing your Python code
In the first part of this blog series we saw how quickly Prefect can help you run parallelized Python code from a fresh OS install. This is great for development and running code locally but as you think about operationalizing your flows, you’ll need to think about schedules, remote environments and monitoring the execution of your code.
In this second blog post, we’re going stay on the local machine but I will demonstrate two options Prefect provides for scheduling, executing and monitoring your code by following some very simple steps. In addition, I’m going to show the steps that you can take to rerun parts of your flow.
I will be continuing with the same snippet of code used in the previous blog post and I will use this to demonstrate Prefect Cloud and Prefect Server features. You may wish to choose one or both, either way, I will show you how to simply switch between the two.
Starting with Prefect Cloud. (If you prefer not to set up an account, skip ahead to Prefect Server)
- Visit cloud.prefect.io. If you don’t already have an account, follow the sign-up process, create your team and then click the button to go directly to the dashboard.
2. Once logged in, click the hamburger menu in the top left, select User > Personal Access Tokens to Create a Token. Provide a name for your token and press the create button. No need to set an expiry in this example.
Once it’s created, copy the token to your clipboard.
3. With the token copied to your clipboard, open the terminal on your local machine and run the following command to login to Prefect Cloud using the Prefect Client.
But first…
# Following directly on from the first blog post, assuming you are
# using Conda and you created a Conda environment with the name
# "prefectenv" make sure you activate your Conda environment first. >> conda activate prefectenv
then
>> prefect auth login -t <copied personal access token>
Login SuccessfulOnce logged in, the following steps can be done either in the terminal window or the UI. For the purpose of this blog, I will show the command line instructions. Feel free to explore the UI to see how you would perform these actions in the UI.
4. Create an API token called a RUNNER token so that the Prefect agent can communicate with the Prefect Cloud API to run your flows. (Hint: in the UI, click the hamburger menu > Team > API Tokens)
>> prefect auth create-token -n my-runner-token -s RUNNER5. Save the RUNNER token to your local configuration
>> export PREFECT__CLOUD__AGENT__AUTH_TOKEN=<COPIED_RUNNER_TOKEN>6. Set the backend to talk with Prefect Cloud
>> prefect backend cloud7. Create a project in Prefect Cloud
>> prefect create project “Prefect Demo”8. Modify the code (example_flow.py) from the last blog to work with Prefect Cloud.
As a reminder this is the last few lines of code from the last blog
with Flow(“getting-started-example”) as flow:
incs = inc.map(x=range(10))
decs = dec.map(x=range(10))
adds = add.map(incs, decs)
total = list_sum(adds)executor = LocalDaskExecutor()
flow.run(executor=executor)
This now needs to be changed to the following:
# import the line below at the top of your file
from prefect.run_configs import LocalRun...with Flow(“getting-started-example”,
executor=LocalDaskExecutor(),
run_config=LocalRun()) as flow: incs = inc.map(x=range(10))
decs = dec.map(x=range(10))
adds = add.map(incs, decs)
total = list_sum(adds)flow.register(project_name = "Prefect Demo")
The only difference is that we now specify an executor and a run config on the flow to configure our desired runtime behavior, and we end by registering the flow’s metadata with the API.
Next, we run this code to perform the registration.
>> python example_flow.pyFinally, we need to run an agent. In this case, we’ll be running the agent on our local machine. The agent will poll the Prefect API for any flows and execute them locally.
>> prefect agent start Once the flow is registered with Prefect Cloud and the agent is running we can now navigate to the Prefect Cloud UI to learn more about the ways Prefect Cloud can help orchestrate your code.
9. Create a schedule for your flow run and turn the schedule on. The following animation show the steps required.
Note: you can create and set your schedule in your Python code also. See this simple example.
10. Watch the flow run with the timeline view.
We just set the schedule to run every three minutes, but for the purpose of the following animation, I clicked into a scheduled flow run and used the “Start Now” button to get things started. In this particular case, we’re running fairly simple code on a local machine, so little can go wrong, but this minimal design allows you to identify potential issues in your flows or your infrastructure much more readily, how long tasks are taking and just generally more granular detail on latencies between the various task and flow run states.
Unless you skipped directly to this section, now that you have seen this run in Prefect Cloud, I will demonstrate a similar experience in Prefect Server. Prefect Server is a self-hosted open source version of the Prefect API and UI.
- Firstly, Prefect Server requires Docker and Docker compose. Install Docker Desktop for your operating system from the following link. Once installed, start Docker Desktop and make sure it is running.
On the mac, I can see this in the menu bar at the top of my desktop.
2. Once Docker is running, we can open up our terminal window to start Prefect Server. First though, you’ll want to switch your Prefect Client to talk with the Prefect Server that will soon to be running on your local machine.
But first…
# Following directly on from the first blog post, assuming you are
# using Conda and you created an environment with the name
# "prefectenv" make sure you activate your Conda environment first.>> conda activate prefectenv
then
# Note: you can use prefect backend cloud to switch to Cloud
>> prefect backend server3. Start Prefect Server with the following command
>> prefect server start
# the UI will be available at localhost:80804. Open a new tab in your terminal to follow the steps from the Prefect Cloud example, starting with step 6 of creating your project. Remember, as you open up new terminal windows you’ll need to make sure your conda environment is activated.
In the intro, I mentioned that I was going to show you how to restart part of your flow. Simply choose a task that you want to rerun, set the state back into “Pending” and click Restart. Easy as that!
From these first two Getting Started blog posts, you should have been able to get a good sense of how you can decorate your Python code with Prefect tasks, allowing you to build a workflow, and manage and visualize your running flow in the Prefect UI either via Prefect Server or Prefect Cloud.
As we progress, we will explore more advanced concepts in addition to setting up and running in remote environments.
Please continue reaching out to us with your questions and feedback — we appreciate the opportunity to work with all of you!
- join our Slack community for ad-hoc questions
- attend our meetup events for contributors, focused on the internals of Prefect
- follow us on Twitter for updates
- visit us on GitHub to open issues and pull requests
Happy Engineering!
— The Prefect Team
