How to Self-Host Prefect with Postgres Using Docker Compose

Prefect is a great way to make your Python dataflows observable and reliable. It’s like air traffic control for your data.

As an open-source tool, Prefect is flexible. You can run it how, when, and where you want. That flexibility means Prefect works well in a variety of Python environments. So whether you prefer Anaconda, venv, or Docker, Prefect has you covered.

Prefect Orion is Prefect’s open-source API and UI. It lets you create blocks, view past and scheduled flow runs, and view and edit deployments and work queues.

Orion contains a significant subset of Prefect Cloud’s features, so it is a good starting point if you’re getting started with Prefect and want to explore it. Orion is ideal if you are willing to self-host and don’t yet need cloud-only features like authentication and workspaces.

This post demonstrates how to use Docker Compose to run Prefect Orion.

Why Run Prefect with Docker Compose?

Docker Compose is like easy mode for Docker.

In fairness, running Docker containers using the CLI is already relatively easy. But when you start adding things like networking, volumes, resource constraints, and health checks, you begin to feel like Gandalf trying to cast a tricky spell.

Although many engineers wouldn’t mind being Gandalf, we shouldn’t need to be wizards just to run a container. Docker Compose lets us move our CLI wizardry to a YAML file so we can reuse it, share it, and add it to a Git repository.

Prefect Docker Images

To make engineers’ lives easier, Prefect provides a variety of Docker images on Docker Hub. For every Prefect release, there are images available for several Python versions, both with and without Anaconda.

We will use prefecthq/prefect:2-python3.10 in the examples below, but feel free to browse Prefect’s images on Docker Hub if you know you need a specific Prefect or Python version.

Running Prefect Orion with Docker Compose

Now, let’s look at how to run Prefect Orion using Docker Compose.

Start by creating a new directory and navigating to it. In this directory, create a file named docker-compose.yml and add the following:

Fortunately, this YAML is straightforward, especially if you’ve run a container using the Docker CLI. However, if you haven’t used a Docker volume before, you might wonder why we need one to run Orion.

Docker images are immutable. So, if you run a container, write to its filesystem, and then shut it down, your changes will be erased. Volumes are Docker’s way of providing persistent storage that saves changes across container restarts.

Since we don’t want information about our flow runs, deployments, blocks, and work queues to disappear any time we restart Orion, our docker-compose.yml file adds a volume named prefect and mounts it at ~/.prefect, which is where Prefect Orion stores its SQLite database and cached flow runs.

We’ve also set the entrypoint property to run the prefect orion start CLI command when the container loads.

Finally, note the ports property. This means we’re sharing port 4200 with the Docker host machine, so you’ll be able to interact with Prefect Orion as if it is running in a local process.

You wouldn’t typically want to run Prefect Orion in Docker Compose this way in production, but it makes testing and experimenting easy. We’ll cover production deployments with Docker Compose in a separate post.

Now that you understand docker-compose.yml, let’s run Orion! First, open a terminal, then change to the directory where you saved your YAML file and run:

docker compose up

You’ll see something like this:

Attaching to orion
orion  | 
orion  |  ___ ___ ___ ___ ___ ___ _____    ___  ___ ___ ___  _  _
orion  | | _ \ _ \ __| __| __/ __|_   _|  / _ \| _ \_ _/ _ \| \| |
orion  | |  _/   / _|| _|| _| (__  | |   | (_) |   /| | (_) | .` |
orion  | |_| |_|_\___|_| |___\___| |_|    \___/|_|_\___\___/|_|\_|
orion  | 
orion  | Configure Prefect to communicate with the server with:
orion  | 
orion  | prefect config set PREFECT_API_URL=http://0.0.0.0:4200/api
orion  | 
orion  | View the API documentation at http://0.0.0.0:4200/docs
orion  | 
orion  | Check out the dashboard at http://0.0.0.0:4200
orion  | 
orion  | 
orion  | INFO:     Started server process [8]
orion  | INFO:     Waiting for application startup.
orion  | INFO:     Application startup complete.
orion  | INFO:     Uvicorn running on http://0.0.0.0:4200

If you open a web browser and navigate to http://localhost:4200, you’ll see the Prefect Orion UI:

Running Prefect Orion and PostgreSQL with Docker Compose

Prefect Orion also supports PostgreSQL if you’d prefer to use it instead of SQLite. You can run both Orion and Postgres at the same time using a single Docker Compose file:

Note the new service named database. It pulls a minimal Postgres image from Docker hub, then sets a default username, password, and database.

Notice that we’re using expose instead of ports like we do for the orion service. The difference between them is that ports exposes ports to the host machine and other machines on the same Docker network as the container, while expose only makes the ports available to the Docker network.

So, we use expose here since we don’t need to make Prefect’s Postgres database visible to the outside world. If you want to connect to Postgres so you can query and explore the database, you can add a ports section to the database service to map port 5432 to a port on your host machine.

We’ve also added a second volume named postgres so Postgres won’t lose all its data if we restart its container.

Finally, note a couple key changes in the orion service:

• A new environment variable named PREFECT_ORION_DATABASE_CONNECTION_URL to provide Orion with a connection string for Postgres.
• A new depends_on property to tell Docker Compose that Orion depends on the database service and should not be started until Postgres is up and running.

As before, you can run everything with the docker compose upcommand. The output will be similar, except you’ll see both Postgres and Prefect Orion logging to the console as they start.

Next Steps

Now that you’ve got Prefect Orion up and running with Docker Compose, you might wonder what comes next.

If you’d like to explore what’s possible with Prefect and Docker Compose in more depth, I’ve created a GitHub repository with a more comprehensive docker-compose.yml file that demonstrates how to use Prefect Orion, the Prefect Agent, and even the Prefect CLI with Docker Compose.

It also shows how to run MinIO — open-source, S3-compatible block storage — alongside Prefect Orion. MinIO is an excellent choice if you want to use remote storage for your Prefect deployments but can’t use cloud-based options like S3 or Azure Blob Storage.

After that, why not see what you can come up with? We love hearing about interesting ways to use Prefect, so feel free to put your Docker Compose files on GitHub and let us know in our Slack community. Happy engineering!