Prefect 0.9.0: Unlocking Cloud
Prefect 0.9.0 is here and makes it even easier for users to take advantage of Prefect Cloud, including its free Scheduler tier. As we continue working with early partners to prepare for Cloud’s upcoming public launch, we’ve focused on reducing frictions around third-party authentication and task result management.
Prefect Cloud’s unique Hybrid architecture allows our managed workflow orchestration service to operate without access to users’ infrastructure, code or data (more on this in a future post). In order to facilitate working with data, Prefect introduced result handlers to provide a mechanism for tracking, serializing, and protecting task data without sending it to Prefect Cloud servers. Result handlers can be configured on a per-task basis to specify how individual task results should be stored. In practice, they are used primarily in situations where users need to recover from failure or rerun / retry specific tasks that rely on upstream data.
In previous versions of Prefect, result handlers were an opt-in feature. We learned that this was ultimately frustrating for our users, who expected that any task could be rerun at any time, for any reason. As a consequence, we’ve enabled result handling by default where possible: starting in 0.9.0, any flows registered to Prefect Cloud that use Local, S3, GCS, or Azure Blob Storage options will automatically be configured to use an appropriate result handler (you can still customize the default however you want). Moreover, all tasks which have access to a result handler will now checkpoint their results. This makes it even easier for users to track and manage their task’s results, and additionally ensures that users can always recover from any failure mode even if their tasks exchange data. Please note: because Docker has no natural default option for permanent storage, a warning will be raised if users register Docker-based flows without providing a result handler.
In addition to default result handlers, there are many other notable new features and enhancements that are in this release! We’ve highlighted some here and the full changelog is available on GitHub.
New Features
Enhanced Fargate Agent
Being an early partner of Prefect gives individuals and organizations a chance to meaningfully steer the direction of our project. As the latest example, Braun Reyes from Clearcover developed two new features for Prefect’s Fargate Agent that allow for cleaner configuration and task management:
- Instead of providing the Fargate Agent with the same Fargate Task keyword arguments for all submitted flow runs, users are now given the option to load these Task keyword arguments directly from S3 for each flow that is deployed.
- Users can now make use of custom ECS-native task revisions for each newly-registered flow version.
Clearcover’s early adoption of Prefect gave them the capability to roadmap and design these features. We know this work directly supports their internal use of Prefect and we are grateful that they elected to contribute them back to Core. We are sure the community will benefit greatly.
Agent flow configuration through environment variables
All Prefect Agents now accept an optional list of environment variables that will be set on each submitted flow run from that agent. This gives users the ability to dynamically configure their flows and securely pass sensitive information on a per-agent basis (recall that flows can be directed to run through certain agents via flow labels).
For example, users who rely on Prefect Secrets can provide them to their agents using the following pattern:
prefect agent start -e PREFECT__CONTEXT__SECRETS__AUTH_KEY=VALUENote that arbitrarily many environment variables can be set by providing the -e flag multiple times.
Similarly, users can use agent environment variables to provide credentials for various third-party systems such as AWS, PostgreSQL, and any other service that relies on environment variables for authentication.
Looping can now exceed recursion limits
Some users have been running large flows in Prefect by taking advantage of dynamic Task Looping. Some of those flows have actually run into Python’s native maximum recursion depth! We’re happy to say Prefect now supports task looping for arbitrarily deep loops.
Please continue reaching out to us with your questions and feedback — we appreciate the opportunity to work with all of you!
- join our Slack community for ad-hoc questions
- follow us on Twitter for updates
- visit us on GitHub to open issues and pull requests
Happy Engineering!
— The Prefect Team
