Announcing Prefect 0.6.0: Cloud Ready
Our team is excited to announce the release of Prefect Core 0.6.0!
To install, just run: pip install prefect -U
This release is the culmination of feedback from both our open-source community and Cloud Lighthouse Partners over the last three months. It coincides with the graduation of Prefect Cloud to a beta product by bringing Core in line with the latest version of the Cloud API. Consequently, all Cloud users must update to Core 0.6.0 to deploy new flows.
In addition, Core has become a stronger standalone workflow engine. The feedback we’ve received from users deploying Core into production has been invaluable and directly contributed to many of the changes in this release.
New Features
Prefect CLI
Prefect 0.6.0 ships with our first version of the Prefect CLI for interacting with Flows deployed to Prefect Cloud. Josh Meek built this in his spare time and surprised us with it during one of Prefect’s internal demo days! We’re confident you’ll love it as much as we do… especially the streaming logs functionality. We’re positioning this CLI as a full-fledged Prefect Cloud client, alongside our UI.
Long-Running Dask Clusters
Most of Prefect’s users prefer executing their Flows using Dask (although this is not a requirement for using Prefect).
Cloud users can now enjoy the ability to run multiple flows against a fixed, long-standing Dask Cluster with the new RemoteEnvironment execution environment.
Task Affinity via Dask Resource Management
Prefect Tasks can now hook directly into Dask’s Worker Resources mechanism via specially formatted Task tags. This allows users to specify both resource affinity for Tasks that need to run on certain machines (e.g., if they require special hardware) as well as resource constraints to limit the number of running Tasks which require a similar scarce resource.
Improved Dask UI Integration
Prefect’s DaskExecutor now explicitly sets future keys with Task names for a more intuitive visualization experience with the Dask Scheduler UI.
Caching
Caching is now supported across flows and tasks! This was a very popular request and we are happy to say that 0.6.0 supports this feature for both Prefect Core and Cloud. To use it, simply specify a common cache_key on the relevant Tasks that should share a cache — in Core, the cache will persist in memory across any and all Flow runs. In Cloud, the CachedState will persist in the Cloud database, whereas the data itself will persist in the location of your choice (specified by a ResultHandler).
Task Library
The task library now includes tasks for AWS step functions, Dropbox, and enhanced Google Cloud Storage interaction.
Slack Community
We heard you loud and clear: we’ve set up a Slack community for users to discuss Prefect, get support, and help shape its future direction.
Please continue reaching out to us with your questions and feedback — we appreciate the opportunity to work with all of you!
- reach out for early Cloud access
- join our Slack community for ad-hoc questions
- follow us on Twitter for updates and to win free coffee!
- visit us on GitHub to open issues and pull requests
Happy Engineering!
— The Prefect Team
