Open-Sourcing the Prefect Platform
One year ago, we open-sourced the Prefect engine, and the response has been incredible. Our lightweight Python framework for combining tasks into workflows has been embraced by users otherwise forced to contend with cumbersome and disappointing tools. Today, we’re taking another huge step forward and open-sourcing an orchestration layer for Prefect — including the Prefect Cloud UI.
You can get the code and fire it up right now: just prefect server start.
The response to Prefect has exceeded all of our expectations. The vibrancy of our community is wonderful, and the breadth of use cases we’ve seen has been incredible. We’ve gotten to work with everyone from climate scientists to astronauts; online educators to World Series champions; single-person data teams to the largest banks in the world. One of the benefits of working directly with such great partners is that we’ve had the opportunity to hone our product with a gauntlet of real-world use cases. Despite the variety of applications, we see the very same negative engineering frustrations repeated across every environment, rediscovering our original thesis over and over:
Negative engineering problems are not always complex, or sophisticated, or difficult. On the contrary: they are often minor, annoying, and repetitive.
This has translated into Prefect’s insurance-like design goal to be minimally invasive when things run smoothly and maximally helpful when things go wrong.
COVID-19 is certainly an extreme example of things going wrong. At Prefect, we’ve tried to figure out how we, as a company, can help. The clear solution is to take features that were formerly part of our proprietary platform and release them as open-source. Our major motivation is to ensure that individuals and businesses can start taking advantage of our software with the least friction possible. As an open-source community, this furthers our objective of delivering a best-in-class workflow management experience without compromise; as a business, allying ourselves with our users at the earliest possible moment — for free — is the ultimate expression of our commercial directive: deliver value, don’t extract it.
Prefect was not started to be a business; Prefect was started to solve a problem. The successful company that followed is merely evidence of the size and scope of that problem, as well as the value and efficacy of our solution. By open-sourcing more of the Prefect Cloud stack, we hope to deliver a valuable workflow management system to our users at a moment when saving time and money is especially important.
Getting Started
In true Prefect fashion, we’ve worked to make this as simple as possible, and our team has worked hard to ensure that the open-source user experience is on par with Prefect Cloud.
If you have the latest version of Prefect installed (we released 0.10 last night) and Docker running, simply enter the following in your terminal:
# set the local backend (you only have to do this once)
prefect backend server# go!
prefect server start
Now you can visit http://localhost:8080 to launch the UI.
It’s really that easy.
We’ve expanded the Cloud deployment tutorial to include the local server, and will be releasing more tutorials soon.
Prefect Cloud
As a business, we’re keenly aware of the potential for tension between our open-source and proprietary products. Historically, we’ve tried to maintain a strict rationale for what we could open-source and what we couldn’t, and today we’ll be updating that mandate in favor of our open-source community.
The Prefect engine has always been licensed under a permissive Apache 2.0 license and will remain that way, forever. Following the model of open-core companies we admire, we are open-sourcing the new server backend and UI under a new Prefect Community License that allows their use for any purpose, including modification or distribution, other than directly competing with Prefect’s own products. This model allows us to continue to invest in our commercial relationships while offering elements of our powerful Prefect Cloud service to our open-source community.
We will continue to offer Prefect Cloud, our managed orchestration service, to organizations that require more than basic workflow management. Cloud includes features like our patented Hybrid Model that allows customer code to remain completely secure and on-premise, as well as authorization and security settings, secrets, third-party integrations, HA infrastructure, and services that ensure workflow semantics are respected even in cases of total failure.
Happy Engineering
Last week, we informed each of our current customers that the UI, a feature previously exclusive to their Cloud licenses, would be made open-source. Every single reply was in favor of this decision — in fact, the most popular response was that if we had done this sooner, the decision to evaluate and purchase a Cloud license would have been much simpler! We are grateful to have formed such amazing relationships, and we are confident in our ability to deliver a best-in-class experience for both our open-source and commercial products.
We’ve collected enough datapoints to know that Prefect can immediately drive major cost savings and efficiencies for businesses that adopt it; we’re also aware that the lack of an open-source UI historically introduced some friction to Prefect’s adoption. In fact, a UI is the most-requested feature we’ve ever had on the open-source repo. By releasing these key pieces of the Prefect Cloud stack, we hope to make it as easy as possible for you, your team, or your company to get started with a complete workflow management solution. The new, open-source Prefect platform encompasses every stage of the workflow pipeline: building, testing, deploying, orchestration, monitoring, and intervention.
We’re incredibly excited to hit this milestone for our product, and grateful to the enormous and enthusiastic community that’s made it possible.
Happy Engineering!
