A documented pattern in Spanner migrations is that teams move a write-heavy order-processing system off a vertically scaled PostgreSQL instance to escape write exhaustion — the existing database peaking at 90% CPU, with read replicas unable to relieve writer saturation — only to hit unexplained P99 latency spikes within weeks of launch, even while running well under Spanner’s theoretical throughput limits. The root cause is a fundamental architectural misunderstanding: treating Spanner like a traditional single-writer PostgreSQL database. This article unpacks how Google Cloud Spanner actually processes writes, what the PostgreSQL interface is (and isn’t), and how to avoid the architectural failures that catch senior DBAs by surprise.

Situation

Consider a realistic production workload: a write-heavy e-commerce order-processing system with a 15 TB active relational database, dominated by an orders table.

Application
     |
PostgreSQL Writer (db-main)
     |
CPU 85–95%
     |
Writes continue increasing (50K+ TPS)

The application is write-heavy (order creation, inventory decrements, payment state changes). Because standard PostgreSQL (and managed equivalents like Cloud SQL for PostgreSQL) relies on a single-writer architecture, adding read replicas provides no relief to the primary writer’s CPU, disk I/O, or lock contention. At 50K transactions per second (TPS), the vertical scaling limit of the largest available compute instance is reached.

The Problem

When a single writer reaches its physical limits, architects typically look to horizontal scaling. Standard PostgreSQL requires application-layer sharding to achieve this, which destroys the strict transactional consistency required for financial ledgers and introduces immense operational complexity.

Newer cloud databases attempt to solve this. However, it’s critical to understand the difference in vendor architectures. AlloyDB (and Amazon Aurora PostgreSQL) uses a disaggregated, log-structured storage engine, but still routes all writes through a single primary compute instance. They improve I/O and replica lag, but they do not horizontally scale the writer. Amazon Aurora PostgreSQL Limitless Database introduces a routing layer over fleets of Postgres shards, but it is effectively managed sharding.

This workload needs strict serializability, horizontal write scaling, and multi-region high availability without application-side sharding. Cloud Spanner fits this requirement perfectly on paper. But how do you operate a distributed database using standard PostgreSQL drivers without accidentally destroying its performance?

The Cloud Spanner PostgreSQL Architecture

To solve this, the team evaluates Cloud Spanner. Spanner decouples compute and storage, distributing both across a cluster. The PostgreSQL interface is a protocol and syntax compatibility layer—it does not make Spanner run the PostgreSQL storage engine under the hood.

flowchart TD
    App[Application Client] --> PG[PGAdapter Proxy]
    PG --> API[Spanner API]
    API --> C1[Compute Node]
    API --> C2[Compute Node]
    API --> C3[Compute Node]
    
    C1 --> S1[Split A — Paxos Group]
    C2 --> S1
    C2 --> S2[Split B — Paxos Group]
    C3 --> S3[Split C — Paxos Group]
    
    S1 --> Colossus[Colossus Distributed Storage]
    S2 --> Colossus
    S3 --> Colossus

What Happens to a Write?

When the application executes an INSERT via the PostgreSQL driver, Spanner does not write to a traditional Postgres Write-Ahead Log (WAL).

  1. Connection: The application connects via PGAdapter or a native Spanner Postgres driver.
  2. Transaction Coordinator: A Spanner compute node acts as the transaction coordinator.
  3. Data Ownership Determination: The coordinator looks up which “split” owns the primary key.
  4. Replication & Consensus: The write is sent to the Paxos leader for that split. The leader assigns a TrueTime timestamp and proposes the write to the Paxos replicas.
  5. Durable Commit: Once a quorum of replicas acknowledges the write, the leader commits.
  6. Acknowledgement: The coordinator returns the success to the application.

Because writes require Paxos consensus across a network, expect commit latencies in the single-digit-to-low-double-digit milliseconds — not the sub-millisecond latencies of a local SSD Postgres commit. Treat this as an illustrative order of magnitude rather than a documented guaranteed figure; actual latency depends on region topology and load.

How Write Scalability Actually Works

Spanner scales writes horizontally by dividing tables into “splits” (contiguous ranges of rows sorted by primary key). If a split becomes too large or too hot, Spanner dynamically rebalances the cluster by dividing the split in two and moving ownership of the new split to a different compute node.

If a transaction updates orders (in Split A) and inventory (in Split B), Spanner performs a distributed transaction using two-phase commit (2PC) powered by TrueTime. While TrueTime makes this efficient, 2PC still introduces latency.

In Practice

Google’s documented behavior for Spanner explicitly warns against treating it like a standard relational database when designing schemas. The following patterns dictate your physical hardware utilization.

Schema Design and Hotspots

If the application uses a monotonically increasing order_id (e.g., SERIAL), all new orders append to the exact same split at the end of the keyspace. Only the compute node owning that split does the work. The rest of the cluster sits idle. This is a “hotspot.” Adding more nodes increases your bill but will not improve throughput.

Good Schema (Spanner optimized):

CREATE TABLE orders (
    order_id varchar(36) PRIMARY KEY, /* Use UUIDv4 */
    customer_id BIGINT,
    order_timestamp TIMESTAMPTZ
);

UUIDv4 distributes writes evenly across the entire keyspace. Alternatively, you can use bit-reversed sequences to maintain integer keys without sequential hotspotting.

To avoid cross-split distributed transactions, Spanner allows you to physically co-locate child rows with their parent rows. The PostgreSQL interface supports this via Spanner-specific extensions:

CREATE TABLE order_items (
    order_id varchar(36),
    item_id varchar(36),
    product_id BIGINT,
    PRIMARY KEY (order_id, item_id)
) INTERLEAVE IN PARENT orders;

PostgreSQL DBA Translation and Compatibility

“PostgreSQL Compatibility” in Spanner means Wire Protocol and Syntax Compatibility, not feature parity. Spanner supports strict serializable isolation; it does not support Read Committed for read-write transactions.

PostgreSQL DBA ConceptSpanner EquivalentImportant Difference
InstanceSpanner InstanceSpanner instances are distributed clusters.
Primary KeyPrimary KeyCritical for physical data distribution.
WAL (Write-Ahead Log)Paxos LogsDistributed consensus logs.
AutovacuumAutomatic CompactionHandled transparently by Spanner.
Read ReplicasAutomatic ReplicasBuilt-in via Paxos.
Sequences (SERIAL)Bit-Reversed SequencesStandard sequences cause hotspots.
Triggers / ProceduresNot Supported (user-defined)User-defined triggers and stored procedures are unsupported — handle at the application layer. Two built-in system procedures exist (spanner.cancel_query(), spanner.compact_all()) but these are operational tools, not user logic.

Terraform and Automation

Configuring the PostgreSQL dialect requires explicitly setting the dialect at creation. It cannot be changed later.

resource "google_spanner_database" "example_db" {
  instance = google_spanner_instance.main.name
  name     = "orders-db"
  
  # CRITICAL: Define the dialect at creation. Cannot be changed.
  database_dialect = "POSTGRESQL"
  
  # Prevent accidental Terraform destruction
  deletion_protection = true

  # Prevent deletion via gcloud/Console/API too — deletion_protection alone
  # only blocks Terraform-initiated destruction, not out-of-band deletion.
  enable_drop_protection = true
}

Note: You cannot execute DDL inside the google_spanner_database resource block when using the POSTGRESQL dialect. DDL must be applied via separate tools (e.g., Liquibase) after provisioning. Also note the distinction above: deletion_protection and enable_drop_protection guard against different deletion paths, and both are needed for real production protection.

Where It Breaks

Do not use Spanner if Cloud SQL or AlloyDB can handle your workload. If your writes fit on a 96-core AlloyDB instance, the complexity of managing distributed primary keys, interleaving, and transaction retries is not worth the operational overhead.

Failure ModeDetectionMitigation / Tradeoff
Node / Process FailureAPI redirects seamlesslySpanner detects failure and elects a new Paxos leader in < 10s. Minimal latency spike.
Region Failure (multi-region instance config only)Multi-region Paxos quorumZero RPO. Surviving regions elect a leader. RTO is near zero. This property requires choosing and paying for a multi-region instance configuration — a regional (single-region) config has no cross-region replicas and does not survive a full region outage.
High ContentionABORTED errors in logsSpanner uses pessimistic two-phase locking with wound-wait. Clients must retry aborted transactions.
Large TransactionsHard limit rejectionsTransactions cannot exceed 100MB or 80,000 mutations. Must chunk large batch updates.
Schema HotspotsKey Visualizer shows red bandsYou cannot brute-force a bad schema with more nodes. Must rewrite primary keys to UUIDv4.
Deep InterleavingDDL rejectionYou cannot interleave deeper than 7 levels.

Migration Considerations

Migrating from Cloud SQL PostgreSQL to Spanner is not a lift-and-shift. You must remove all triggers, stored procedures, and unsupported data types (like PostGIS). Applications must be refactored to implement retry logic for ABORTED transactions.

What to Do Next

If you are evaluating Cloud Spanner for a PostgreSQL workload:

  • Problem: Single-node PostgreSQL writers eventually hit a physical CPU/IO wall.
  • Solution: Spanner’s PostgreSQL interface provides horizontal write scaling through distributed splits and TrueTime, using familiar Postgres syntax and drivers.
  • Proof: Google documents that sequential keys will bottleneck Spanner, requiring UUIDs or bit-reversed sequences to utilize the distributed architecture effectively.
  • Action: Run your schema through the Spanner migration tool to identify incompatible types, remove all triggers, and ensure your application ORM can gracefully handle and retry transaction aborts before committing to a migration.

Technical References

Google Cloud

HashiCorp Terraform

AWS

Claims Requiring Additional Human Validation

  • The 80,000-mutations-per-commit limit was live-verified against Google’s current Quotas page as of this writing, but Google’s own blog post announcing this figure carries an editor’s note that it “has since increased beyond 80,000” — worth a periodic re-check, since this is a number Google has moved before.
  • The single-digit-to-low-double-digit millisecond commit latency figure is an illustrative order of magnitude, not a documented guaranteed SLA — no single Google source pins an exact number for Paxos consensus commit latency.