PostgreSQL on Kubernetes the Right Way: Part One

The story for managing stateless microservices on Kubernetes is very clear. However, stateful legacy apps like PostgreSQL and other databases aren’t as obvious a fit. In this series, we’ll show what it takes to run an industrial-strength PostgreSQL deployment on Kubernetes.

What makes PostgreSQL hard?

A PostgreSQL deployment involves a pretty complex combination of concerns:

• High Availability
• Load Balancing
• Synchronization Between Instances
• Scaling Up and Down
• Automated Backup, Recovery from Backup
• Rolling Upgrades/Downgrades
• Health Monitoring, Debugging

How does PostgreSQL look outside of Kubernetes?

PostgreSQL follows the master-slave pattern. There’s a single authoritative primary server (master) and some number of standby servers (slaves) that mirror the primary.

This distributed architecture serves two purposes:

• Failover — If the primary fails, a standby can take its place.
• Load Balancing — Standbys can handle read-only requests, reducing load on the primary.

Additionally, the primary continuously archives its write-ahead log (WAL) and periodically creates backups of its entire state (called base backups). The WAL is a log of all the operations performed on the database — incremental changes. The standbys stay up-to-date through a combination of streaming WAL from the primary and reading archived WAL and base backups.

Now how does PostgreSQL look inside Kubernetes?

The PostgreSQL Pod

The core of a PostgreSQL instance is the postgresql executable. We run it in Kubernetes as a Pod with a postgresql container.

Primary and Standby Pods

Recall the two types of PostgreSQL instances:

• Primary — Typically only one instance, used for reads and writes.
• Standby — Many of these, used for reads, can be promoted to primary.

Because any PostgreSQL instance can become the primary, both kinds have the same Pod spec. If primaries and standbys had different Pod specs, we wouldn’t be able to transform a live instance from standby to primary.

Let’s look at what each Pod needs to do.

When the first Pod starts

For simplicity, let’s consider a PostgreSQL deployment that starts with a single instance. This mirrors the way a human operator would set up PostgreSQL for the first time.

• Since it’s the only instance, it will be the primary.
• We need to initialize the database, possibly from a preexisting backup.
• The primary needs to be configured for continuous backup.
• The primary needs to be configured for streaming replication. For example, standbys must be able to authenticate with the primary.

When the second Pod starts

The second instance is a standby. In order for it to be useful, it must be synchronized with the primary.

• Initialize the database from the primary’s backup.
• The standby connects to the primary to receive streaming updates.
• If the stream is interrupted, the standby should still synchronize from the continuous backup.

Failover

If something’s wrong with the primary, we need to replace it with one of the standbys. It could be that the primary crashed or there’s a network outage or an admin deleted it. Assuming a mechanism to detect this situation, here’s how the PostgreSQL deployment should react:

• In order to make sure there aren’t multiple primaries, ensure that the old primary is inaccessible. (i.e. delete it; it shouldn’t be part of any Service)
• Promote one of the standbys to primary. Set up continuous backup and streaming replication.

Architectural Preview

We’ve outlined some of the functional aspects of maintaining a highly available PostgreSQL installation. Some pieces match built-in Kubernetes features, but others require custom code.

Kubernetes follows a common pattern in the way it manages distributed applications — distributed agents (controllers) communicating through a central store (etcd via the API server). The same architecture can manage PostgreSQL for us:

• A controller tracks the status of our PostgreSQL instances. It manages the entire PostgreSQL application. It can replace unhealthy PostgreSQL Pods, choose a new primary, scale up and down, and handle any other application-level concerns.
• Each PostgreSQL Pod has a sidecar that performs instance-level management like setting up streaming replication, promoting to primary, or restoring from backup.

Next Steps

An industrial-strength PostgreSQL installation is a complex beast. Here are some of the concerns we still need to address:

• Observability
• Load Balancing
• Distribution Across Failure Zones
• Implementation Details — e.g. How does the controller coordinate with the sidecars? How do we connect standbys to the primary? How do we set up the continuous backup volume? What’s the user interface like for a human operator?

We’ll get to all of it in the rest of this series. Stay with us for the next post!

PostgreSQL (src)