A PostgreSQL pod that restarts is not a highly available database. Kubernetes can restore a workload object; it cannot, by itself, decide which copy of PostgreSQL is authoritative, fence an uncertain primary, or prove that acknowledged transactions survived a failure.

Situation

Kubernetes has made stateful workloads operationally plausible. A StatefulSet gives each replica a stable ordinal and can associate it with a persistent volume claim. A CSI driver can provision zonal block storage. Topology constraints can distribute pods across nodes and availability zones. An operator can reconcile a PostgreSQL cluster, create replicas, manage configuration, and automate parts of failover.

Those capabilities remove undifferentiated orchestration work. They do not collapse storage, replication, consensus, backup, and application recovery into one guarantee.

That distinction matters because the visible symptom of several different failures is identical: the primary pod is unavailable. The underlying events may be very different:

  • the PostgreSQL process exited while its node and volume remain healthy;
  • the node disappeared but the zonal volume is still attachable;
  • the availability zone is impaired and the primary volume cannot be reached;
  • the control plane cannot determine whether the old primary is dead or merely partitioned;
  • a standby is running but has not received or replayed the latest write-ahead log records;
  • the database has recovered, but clients still hold broken connections to the previous primary.

Treating each event as “restart the pod” produces unsafe automation. Production HA begins by naming the failure, its fault domain, and the evidence required before promotion.

The Problem

A StatefulSet provides stable network and storage identity. It does not provide PostgreSQL leader election, replication, write fencing, timeline management, or point-in-time recovery. Persistent volumes also have a lifecycle independent of pods: deleting or rescheduling a pod does not inherently erase its data. With common cloud block storage, however, a volume is normally bound to one zone and attached read-write to one node. If a replacement pod lands in another zone, the scheduler may leave it pending rather than silently move the disk across zones.

The harder case is ambiguity. Suppose the Kubernetes control plane stops hearing from the node that hosts the primary. Promoting a standby may restore writes, but if the original primary is still accepting traffic through another network path, the system now has two writable histories. PostgreSQL streaming replication does not merge divergent timelines. The old primary must be fenced from clients and replicas before a successor accepts writes.

Durability is a separate decision. In asynchronous streaming replication, a primary can acknowledge a commit before a standby has persisted the corresponding WAL. Promotion can therefore lose acknowledged transactions up to the replication lag at the failure boundary. Synchronous replication can make selected commit acknowledgements wait for standby confirmation, but its guarantee depends on synchronous_commit, synchronous_standby_names, the acknowledgement mode, and the placement and health of the selected standbys. It also trades write availability and latency for a smaller recovery-point objective.

The engineering question is therefore not “Does the operator support failover?” It is: under each failure mode, how does the system establish one writable primary, what committed data can be lost, and how do clients and operators verify recovery?

Build the HA Contract Around Failure Domains

A strong production baseline is three independently scheduled database instances, one volume per instance, explicit anti-affinity or topology-spread constraints, and a tested object-storage backup and WAL-archive path. Two instances can provide failover, but they leave less room to preserve both availability and a synchronous durability rule after one failure. The required instance count must follow the stated failure envelope. The replicas form the database data plane. The operator observes and reconciles that data plane; it is not a substitute for it.

flowchart TD
    APP["Application clients"] --> RW["Read-write service"]
    OP["PostgreSQL operator"] --> P["Primary — zone A"]
    OP --> S1["Standby — zone B"]
    OP --> S2["Standby — zone C"]
    RW --> P
    P -->|WAL stream| S1
    P -->|WAL stream| S2
    P -->|WAL archive| OBJ["Object storage — backups and WAL"]
    S1 --> PV1["Independent zonal volume"]
    S2 --> PV2["Independent zonal volume"]
    P --> PV0["Independent zonal volume"]
    MON["Metrics and alerting"] --> P
    MON --> S1
    MON --> S2

This topology supports failover because each standby owns a separate PostgreSQL data directory and continuously receives WAL. Sharing one writable filesystem among PostgreSQL servers is not a replication strategy. Likewise, restoring the primary’s volume in another zone is a disaster-recovery operation with storage-specific timing and constraints, not the normal database failover path.

Define the durability policy before choosing automation

Write down the contract in terms an application team can test:

PolicyCommit acknowledgementExpected failure behaviorPrincipal tradeoff
Asynchronous replicaPrimary local durabilityRecent acknowledged commits may be absent after promotionLowest steady-state latency; non-zero RPO
One synchronous acknowledgementPrimary plus a selected standby acknowledgementProtects the acknowledged commit when that standby is the promotion candidate or the failover policy proves an eligible candidate has the writeCommit latency crosses a failure domain; a different stale replica can still be unsafe to promote
Quorum synchronous acknowledgementPrimary plus a configured quorumProtects the configured commit rule only when promotion is constrained to a replica set that must contain the acknowledged writeMore network and capacity cost; failover may be refused when the proof cannot be established

“Synchronous” is not enough detail. PostgreSQL can wait for a standby to write, flush, or replay WAL, depending on configuration. It also does not control which replica an external failover system will promote. In a three-instance cluster using an ANY 1 rule, standby A can acknowledge a commit while standby B remains behind. If the primary and A then fail, promoting B can lose that acknowledged commit. The failover policy must therefore constrain promotion as well as commit acknowledgement.

CloudNativePG 1.30 can address this distinction when failoverQuorum: true is explicitly enabled. The operator then permits failover only when the reachable eligible replicas satisfy its quorum calculation; if that proof is unavailable, it refuses promotion. The separate dataDurability setting controls how synchronous-commit requirements behave when standbys are unavailable: required preserves the configured durability requirement, while preferred can relax it to favor write availability. Neither setting should be assumed to enable the other. Failover quorum is not free consensus added to PostgreSQL; it is a promotion guard whose availability tradeoff must be tested with the exact instance count, synchronous replica count, durability policy, and remote-replica configuration.

The team must also decide what happens when no synchronous standby is available. Automatically weakening the policy may preserve availability while silently changing the RPO. Refusing commits preserves the durability contract but causes a write outage. Either behavior can be valid; an unobserved transition between them is not.

Fence first, promote second

Promotion is safe only after the previous primary can no longer accept writes. The mechanism depends on the environment and operator, but the invariant is stable:

  1. Remove the old primary from the write endpoint.
  2. Establish that it is stopped, isolated, or unable to access the database volume and client network.
  3. Apply the named operator’s eligibility and durability rules, then rank the remaining candidates using WAL position, timeline, and health.
  4. Promote exactly one candidate.
  5. Point the write service at the new primary and verify its endpoints.
  6. Rejoin the old primary only after rewinding or rebuilding it as a replica of the new timeline.

Kubernetes leases and reconciliation state help coordinate controllers, but a generic lease alone does not prove that an unreachable PostgreSQL process has stopped. In a network partition, “not observed” is not the same as “dead.” A production design must identify the authority that performs or verifies fencing and must test what happens when that authority is unavailable.

CloudNativePG 1.30 combines a primary lease with a primary-isolation check. The primary instance manager renews the lease and, when it cannot reach both the Kubernetes API server and peer instances, initiates shutdown to isolate itself before the operator elects a successor. The default is a smart shutdown: existing sessions can continue until smartShutdownTimeout, documented as 180 seconds by default. That residual window is part of the fencing budget. A stricter design can set smartShutdownTimeout to 0 so isolation uses fast shutdown, but that changes the interruption behavior and must be validated. Infrastructure-level fencing may still be necessary when the database process or its isolation path cannot execute.

Treat client recovery as part of database recovery

Changing a Service selector or endpoint does not migrate established TCP sessions. During promotion, connection pools can retain dead sockets, in-flight transactions can fail with uncertain outcomes, DNS and endpoint propagation take time, and replicas may briefly reject writes while roles converge.

Clients need bounded connection and statement timeouts, backoff with jitter, and transaction retry rules that distinguish safe retries from ambiguous commits. Operations that may be replayed after an uncertain response should be idempotent or protected by a durable request key. Otherwise, a technically successful promotion can still create duplicate business actions or a prolonged application outage.

Recovery time must therefore include detection, fencing, candidate selection, promotion, endpoint convergence, and application reconnection. Reporting only the promotion duration understates the actual RTO.

Keep backup and restore outside the HA blast radius

Replicas reproduce logical mistakes and many forms of corruption. They do not replace backups. The design needs base backups, continuous WAL archiving, retention aligned with recovery requirements, and restoration into an isolated environment.

A restore test should prove more than “the command completed.” Verify the target timestamp, database timeline, expected rows or checksums, role and extension configuration, encryption access, and the time required to make the restored database usable. If the WAL archive is unhealthy, a running standby does not make the recovery plan complete.

In Practice

The documented Kubernetes and PostgreSQL behaviors lead to a concrete operating model.

Kubernetes documents StatefulSets as providing stable identity and persistent storage associations; its persistent-volume model deliberately separates volume lifecycle from pod lifecycle. The operational consequence is that a pod replacement normally attempts to reuse its claim. Placement must remain compatible with the volume’s topology. WaitForFirstConsumer helps align topology when a volume is initially provisioned; it does not relocate an already bound zonal volume when its pod is rescheduled.

PostgreSQL documents streaming replication as asynchronous by default and states that failover can lose transactions that were committed but not yet shipped. Its synchronous-replication controls allow a deployment to wait for one or more standbys, but the database may then wait indefinitely when the required standby acknowledgements are unavailable. That is the core CAP-style decision for this subsystem: the failover controller cannot promise both uninterrupted writes and a durability acknowledgement it can no longer obtain.

CloudNativePG 1.30’s documented failover process ranks eligible instances using status and WAL information and distinguishes automated failover from controlled switchover. WAL position alone does not prove that an arbitrary candidate contains every synchronously acknowledged commit; the configured failover-quorum rule determines whether promotion may proceed when durability is required. Its behavior illustrates the broader operator pattern: automation reduces detection and execution time, while replication state and promotion constraints determine whether the result meets the RPO. The operator should be evaluated by failure injection and observable state transitions, not by the existence of a “failover” feature.

For each release or infrastructure change, run at least these drills in a non-production environment that preserves production topology:

  • terminate only the PostgreSQL process and verify local recovery does not cause unnecessary promotion;
  • drain the primary node and observe volume attachment, scheduling, and disruption-budget behavior;
  • isolate the primary’s zone and verify fencing precedes promotion;
  • add replication lag, then fail the primary and measure the actual data-loss boundary;
  • make a required synchronous standby unavailable and confirm whether writes block or the policy is deliberately degraded;
  • hold stale client connections through promotion and verify pool recovery and retry safety;
  • restore a backup plus archived WAL to a target time and validate application-level data.

Record detection time, fencing time, last acknowledged transaction, promoted WAL location, endpoint convergence, first successful application write, and complete service recovery. These measurements produce defensible RPO and RTO evidence without inventing universal numbers.

Observe the invariants, not just pod health

Pod readiness is necessary but insufficient. Alerting and dashboards should expose:

  • the identity, timeline, and age of the current primary;
  • pg_stat_replication state and write, flush, and replay positions;
  • replication slots and retained WAL pressure;
  • synchronous-standby membership and whether commits are waiting;
  • operator conditions, reconciliation failures, and failover events;
  • PVC binding, attachment failures, node topology, and unschedulable pods;
  • WAL-archive success, last successful base backup, and restore-test age;
  • write-service endpoints, connection errors, transaction retries, and time to first successful write.

An alert should connect symptom to violated objective. “Replica lag is high” becomes actionable when it states that the measured lag exceeds the RPO available for an asynchronous promotion or threatens WAL retention on the primary.

Where It Breaks

Failure modeUnsafe assumptionLikely consequenceRequired control
Primary process exitsEvery restart requires failoverAvoidable promotion and connection churnDistinguish process recovery from instance failure
Node becomes unreachableKubernetes has proved the database stoppedDual-primary writes after premature promotionExternal or infrastructure-backed fencing
Primary zone failsThe primary volume follows the pod to another zonePending pod or inaccessible storageIndependent cross-zone replicas and topology-aware provisioning
Asynchronous primary failsA healthy standby contains every acknowledged commitRecent committed transactions disappearMeasured lag, explicit RPO, or synchronous acknowledgement
Required synchronous standby failsHA always preserves write availabilityCommits block, or durability is silently weakenedDocumented degradation policy and alerting
Service points to new primaryApplications recover immediatelyStale connections and ambiguous transaction resultsPool eviction, bounded retries, idempotency
Replica is healthyBackups are unnecessaryLogical corruption or deletion propagatesIsolated backups, WAL archive, timed restore drills
Old primary returnsIt can simply resume as a replicaDivergent timeline or renewed write splitRewind or rebuild before rejoining
Operator is runningRecovery objectives are provenUntested edge cases fail during an incidentFault-injection evidence and maintained runbooks

The design also has a cost boundary. Cross-zone synchronous replication adds network latency to commits. Three database instances and independent volumes increase infrastructure cost. Operators add a controller and custom-resource lifecycle that the platform team must understand during upgrades and degraded control-plane conditions. For workloads that can tolerate a managed service’s constraints, transferring part of this responsibility to a database provider may be the more reliable engineering choice.

What to Do Next

  • Problem: A StatefulSet is being treated as database HA. Solution: Document the separate guarantees for scheduling, storage, replication, fencing, backup, and client recovery. Proof: Map every supported failure to the component that detects and contains it. Action: Reject any architecture diagram that cannot identify the authoritative primary and fencing path.
  • Problem: The RPO is implied by labels such as “replicated” or “synchronous.” Solution: Specify the exact acknowledgement mode, standby count, topology, and degradation behavior. Proof: Kill the primary under measured lag and compare acknowledged writes with the promoted state. Action: Publish the result as the service’s tested RPO envelope.
  • Problem: Recovery timing ends at database promotion. Solution: Include endpoint convergence and client transaction recovery in the RTO. Proof: Measure from fault injection to the first successful, verified application write. Action: Add connection-pool and idempotency tests to every failover drill.
  • Problem: Replicas are being used as the recovery plan. Solution: Maintain isolated base backups and continuous WAL archives. Proof: Restore to a chosen point in time and validate business data. Action: Alert on backup age, archive gaps, and restore-test age.

Sources