How to Self-Host Prefect with Postgres Using Docker Compose
Prefect is a great way to make your Python dataflows observable and reliable. It’s like air traffic control for your data.
As an open-source tool, Prefect is flexible. You can run it how, when, and where you want. That flexibility means Prefect works well in a variety of Python environments. So whether you prefer Anaconda, venv, or Docker, Prefect has you covered.
Prefect Orion is Prefect’s open-source API and UI. It lets you create blocks, view past and scheduled flow runs, and view and edit deployments and work queues.
Orion contains a significant subset of Prefect Cloud’s features, so it is a good starting point if you’re getting started with Prefect and want to explore it. Orion is ideal if you are willing to self-host and don’t yet need cloud-only features like authentication and workspaces.
This post demonstrates how to use Docker Compose to run Prefect Orion.
Why Run Prefect with Docker Compose?
Docker Compose is like easy mode for Docker.
In fairness, running Docker containers using the CLI is already relatively easy. But when you start adding things like networking, volumes, resource constraints, and health checks, you begin to feel like Gandalf trying to cast a tricky spell.
Although many engineers wouldn’t mind being Gandalf, we shouldn’t need to be wizards just to run a container. Docker Compose lets us move our CLI wizardry to a YAML file so we can reuse it, share it, and add it to a Git repository.
Prefect Docker Images
To make engineers’ lives easier, Prefect provides a variety of Docker images on Docker Hub. For every Prefect release, there are images available for several Python versions, both with and without Anaconda.
We will use prefecthq/prefect:2-python3.10
in the examples below, but feel free to browse Prefect’s images on Docker Hub if you know you need a specific Prefect or Python version.
Running Prefect Orion with Docker Compose
Now, let’s look at how to run Prefect Orion using Docker Compose.
Start by creating a new directory and navigating to it. In this directory, create a file named docker-compose.yml
and add the following:
Fortunately, this YAML is straightforward, especially if you’ve run a container using the Docker CLI. However, if you haven’t used a Docker volume before, you might wonder why we need one to run Orion.
Docker images are immutable. So, if you run a container, write to its filesystem, and then shut it down, your changes will be erased. Volumes are Docker’s way of providing persistent storage that saves changes across container restarts.
Since we don’t want information about our flow runs, deployments, blocks, and work queues to disappear any time we restart Orion, our docker-compose.yml
file adds a volume named prefect
and mounts it at ~/.prefect
, which is where Prefect Orion stores its SQLite database and cached flow runs.
We’ve also set the entrypoint
property to run the prefect orion start
CLI command when the container loads.
Finally, note the ports
property. This means we’re sharing port 4200 with the Docker host machine, so you’ll be able to interact with Prefect Orion as if it is running in a local process.
You wouldn’t typically want to run Prefect Orion in Docker Compose this way in production, but it makes testing and experimenting easy. We’ll cover production deployments with Docker Compose in a separate post.
Now that you understand docker-compose.yml
, let’s run Orion! First, open a terminal, then change to the directory where you saved your YAML file and run:
docker compose up
You’ll see something like this:
Attaching to orion
orion |
orion | ___ ___ ___ ___ ___ ___ _____ ___ ___ ___ ___ _ _
orion | | _ \ _ \ __| __| __/ __|_ _| / _ \| _ \_ _/ _ \| \| |
orion | | _/ / _|| _|| _| (__ | | | (_) | /| | (_) | .` |
orion | |_| |_|_\___|_| |___\___| |_| \___/|_|_\___\___/|_|\_|
orion |
orion | Configure Prefect to communicate with the server with:
orion |
orion | prefect config set PREFECT_API_URL=http://0.0.0.0:4200/api
orion |
orion | View the API documentation at http://0.0.0.0:4200/docs
orion |
orion | Check out the dashboard at http://0.0.0.0:4200
orion |
orion |
orion | INFO: Started server process [8]
orion | INFO: Waiting for application startup.
orion | INFO: Application startup complete.
orion | INFO: Uvicorn running on http://0.0.0.0:4200
If you open a web browser and navigate to http://localhost:4200, you’ll see the Prefect Orion UI:
Running Prefect Orion and PostgreSQL with Docker Compose
Prefect Orion also supports PostgreSQL if you’d prefer to use it instead of SQLite. You can run both Orion and Postgres at the same time using a single Docker Compose file:
Note the new service named database
. It pulls a minimal Postgres image from Docker hub, then sets a default username, password, and database.
Notice that we’re using expose
instead of ports
like we do for the orion
service. The difference between them is that ports
exposes ports to the host machine and other machines on the same Docker network as the container, while expose
only makes the ports available to the Docker network.
So, we use expose
here since we don’t need to make Prefect’s Postgres database visible to the outside world. If you want to connect to Postgres so you can query and explore the database, you can add a ports
section to the database
service to map port 5432 to a port on your host machine.
We’ve also added a second volume named postgres
so Postgres won’t lose all its data if we restart its container.
Finally, note a couple key changes in the orion
service:
- A new environment variable named
PREFECT_ORION_DATABASE_CONNECTION_URL
to provide Orion with a connection string for Postgres. - A new
depends_on
property to tell Docker Compose that Orion depends on the database service and should not be started until Postgres is up and running.
As before, you can run everything with the docker compose up
command. The output will be similar, except you’ll see both Postgres and Prefect Orion logging to the console as they start.
Next Steps
Now that you’ve got Prefect Orion up and running with Docker Compose, you might wonder what comes next.
If you’d like to explore what’s possible with Prefect and Docker Compose in more depth, I’ve created a GitHub repository with a more comprehensive docker-compose.yml
file that demonstrates how to use Prefect Orion, the Prefect Agent, and even the Prefect CLI with Docker Compose.
It also shows how to run MinIO — open-source, S3-compatible block storage — alongside Prefect Orion. MinIO is an excellent choice if you want to use remote storage for your Prefect deployments but can’t use cloud-based options like S3 or Azure Blob Storage.
After that, why not see what you can come up with? We love hearing about interesting ways to use Prefect, so feel free to put your Docker Compose files on GitHub and let us know in our Slack community. Happy engineering!