Why Distributed NoSQL Databases Develop Hot Partitions

A database can have abundant fleet-wide capacity and still throttle one customer because routing concentrates that customer’s traffic on one physical ownership unit.

Situation

Partitioned databases scale when keys distribute storage and requests across independent workers. Production traffic rarely distributes uniformly: one tenant, device, product, or recent time range can dominate. Adding nodes does not immediately fix a key that still maps to one DynamoDB partition, Bigtable tablet range, Cassandra replica set, or MongoDB chunk.

“Hot partition” is overloaded. It may mean one logical key, a physical range, a replica set, or a shard. Diagnosis must identify the constrained ownership unit before changing the schema.

The Problem

Global CPU averages conceal local saturation. Retrying a throttled operation can increase work on the same owner, while rushed salting can trade one hot write for expensive read fan-out and loss of ordering.

Is pressure caused by key popularity, monotonic placement, an oversized logical partition, or lagging movement—and can mitigation preserve correctness?

Diagnose the Ownership Unit

flowchart TD
    A["Latency or throttling alert"] --> B{"Fleet-wide saturation"}
    B -->|Yes| C["Add or rebalance capacity after workload validation"]
    B -->|No| D["Group traffic by tenant and logical key"]
    D --> E{"Single logical key dominates"}
    E -->|Yes| F["Cache reads or split an application-owned bucket"]
    E -->|No| G["Inspect physical range or shard distribution"]
    G --> H{"Monotonic or low-cardinality key"}
    H -->|Yes| I["Redesign key and migrate deliberately"]
    H -->|No| J["Inspect movement, compaction, cache, and replica health"]

Group request count, bytes, latency, throttles, and errors by operation, tenant, and key hash. Correlate that with physical evidence:

DynamoDB: throttles, consumed capacity, contributor insights, and partition-key distribution. Adaptive capacity helps but does not make one item infinitely scalable.
Bigtable: Key Visualizer and ListHotTablets; Google exposes hot ranges rather than a fictional application “worker.” Hot tablet diagnostics.
Cassandra: coordinator and local latency, dropped messages, pending tasks, partition-size histograms, SSTables per read, compaction debt, and replica health.
MongoDB: shard-key distribution, per-shard operations, balancer state, range-deletion backlog, and currentOp. The default range size is 128 MB, but size alone does not identify a traffic hotspot. MongoDB balancer documentation.

Four Different Mechanisms

DynamoDB: one key can reach a physical boundary

A high-cardinality partition key is necessary but insufficient: a celebrity item or global counter remains one hot key. Prefer write-sharded counters, cacheable materialized reads, or independent subkeys. Preserve conditional-write semantics explicitly; random retries against different keys are not equivalent writes.

On-demand mode has documented warm-throughput behavior. A jump above twice the previous peak inside 30 minutes may throttle unless pre-warmed or staged. DynamoDB on-demand scaling.

Bigtable: ordered keys can create a hot range

Bigtable sorts rows lexicographically. Timestamp-first or sequential identifiers drive writes into a narrow range. Put a stable high-cardinality segment before time, but do not blindly hash the whole key: Google notes that hashing destroys useful row-range locality. Bigtable schema design.

Cassandra: a logical partition belongs to one replica set

Cassandra hashes the partition key, but all rows for one key still map to the same replicas. Bucketing by time or deterministic shard can distribute it at the price of bounded read fan-out. TTL expiry produces tombstones. Never lower gc_grace_seconds as an incident shortcut unless repairs provably finish inside the shorter window; otherwise deleted data can reappear. Cassandra tombstones.

MongoDB: shard keys couple placement and routing

A monotonically increasing ranged shard key concentrates inserts near the maximum range; a low-cardinality key limits distribution; omitting a queried key can scatter reads. Hashed sharding distributes writes but gives up range locality. The balancer moves ranges in the background and one shard participates in only one chunk migration at a time; stopping it is not a universal first response. MongoDB sharding.

Safe Remediation Order

Stop retry storms with bounded concurrency, exponential backoff, and jitter.
Protect unaffected tenants with admission control and per-tenant budgets.
Reduce read pressure with correctness-aware caching or request coalescing.
Add capacity only if the stressed ownership unit can use it.
Redesign keys through dual-write, backfill, parity verification, and reversible cutover.

Do not redirect a mutation to an arbitrary replica, acknowledge an in-memory queue as durable, or change consistency levels during an incident without a written correctness decision.

Verification gates

Treat the change as successful only when the previously hot owner no longer dominates request rate or tail latency, the next-hottest owners have not inherited the problem, and retry volume returns to its pre-incident ratio. For a new bucketed key, exercise page boundaries, duplicate suppression, partial-bucket failure, and historical reads before cutover. Keep the old read path available until a reconciliation job proves that every mutation version visible in the source is represented in the new layout. Capacity graphs alone cannot prove a key migration preserved results.

In Practice

The documented pattern differs by engine but shares one principle: distribution derives from the key. Bigtable warns against timestamp-prefixed keys. MongoDB documents background range balancing and migration limits. Cassandra documents tombstone and repair safety. DynamoDB documents warm-throughput boundaries. None supports a universal requests-per-partition threshold across payload sizes, consistency modes, and versions.

Where It Breaks

Intervention	New risk	Verification
Add deterministic buckets	Reads fan out and ordering becomes a merge problem	Bound bucket count; test page tokens and partial failure
Cache a hot item	Stale data or stampede on expiry	Define staleness; coalesce requests; jitter TTLs
Add nodes	Movement competes with foreground traffic	Rate-limit movement; watch tail latency and backlog
Pause balancing	Existing skew persists	Pause only for a diagnosed conflict with a rollback time
Lower Cassandra grace	Zombie resurrection after missed repair	Prove full repair completion inside the grace window
Change the key	Dual-write divergence	Reconcile by immutable identity and mutation version

What to Do Next

Problem: Fleet averages hide a saturated ownership unit.
Solution: Attribute requests to logical keys and physical owners before remediating.
Proof: Show that load and tail latency distribute without correctness drift.
Action: Add per-tenant and per-key observability, then rehearse a reversible key migration.