Custom Schedules in Prefect
Prefect supports fully-customizable workflow schedules
One of Prefect’s core ideas is that workflows should be able to run at any time, for any reason, with any concurrency — but that doesn’t mean they can’t run on schedule, too!
We designed Prefect’s new Schedule classes with our users’ requirements in mind. It turns out almost every company has at least one workflow that doesn’t run on a simple interval like “hourly” or “daily.” Examples we’ve worked with include:
- trading days
- month-ends
- quarter-ends
- alternate Tuesdays
- next business day
- professional sports team schedules (stay tuned for that blog post!)
Therefore, to Prefect, a schedule is simply a flexible way of deciding when flows should run. We’ve written extensively about the issues caused by Airflow’s hard dependency on interval-based scheduling, but another major consequence is that Airflow can’t represent any of the unusual scheduling patterns we just described. On the one hand, we’re extremely impressed with users’ ingenuity as they hack around Airflow’s limitations; on the other hand — shouldn’t this be easy?
Thanks to a new Prefect feature, it is. Our user research led us to design Prefect’s schedules with three components: clocks, filters, and adjustments (you can read more about them in the documentation).
To see how lightweight this can be, here’s a schedule that runs on the first weekday after each month-end:
from prefect import schedules
from datetime import timedeltaschedules.Schedule( # fire every day
clocks=[schedules.clocks.IntervalClock(timedelta(days=1))],
# ...but only include month-ends
filters=[schedules.filters.is_month_end],
# ...and run on the next weekday
adjustments=[schedules.adjustments.next_weekday] )
Clocks
Each schedule has one or more clocks that emit events. Prefect currently has three clocks — one which emits events on a regular interval, one that parses a cron string, and one that takes a preset list of datetimes.
All Prefect clocks are timezone-aware, so if you specify a start_date with a particular timezone, Prefect will respect all of its conventions, including adjustments like daylight savings time.
Filters
Once an event has been emitted by a clock, it is run through a series of optional filters. Filters are used to decide whether the candidate event should be included as part of the flow’s schedule. For example, you might have an hourly clock paired with a business-day or business-hours filter, to only include events during the appropriate window. You could also filter for weekdays, or use date windows to only run your workflow in the summer. Filters can be combined with and, or, or not logic for flexibility.
Adjustments
If a candidate event passes the filters, it finally goes through a series of optional adjustments. This is an opportunity to modify the event. This is less common, but can be used (for example) to select the next business day if the candidate event falls on a weekend.
By combining clocks, filters, and adjustments, Prefect schedules can represent arbitrarily complex scheduling logic. Familiar convenience classes still exist if you simply want to generate schedules from an interval or cron string. In addition, the new interface is fully backwards-compatible with old-style schedules, including serialization.
This functionality is now available on the master branch of the open-source repo and will be widely available in the next release of Prefect Core.
Happy engineering!
