The 2 AM NoSQL Incident: Failure Modes Across Bigtable, DynamoDB, Cassandra and MongoDB

The first job in a distributed-database incident is to stop amplification and preserve evidence—not to change consistency, disable repair, or pause balancing on instinct.

Situation

The alert says “database latency,” but the cause may be client retries, a hot key, lost quorum, an election, storage saturation, or data movement competing with foreground work. Each engine exposes different ownership and replication boundaries. A cross-engine runbook should standardize decisions without pretending their commands are interchangeable.

The Problem

Unsafe incident advice often looks decisive: retry faster, lower Cassandra gc_grace_seconds, stop the MongoDB balancer, salt every key, or restart a slow node. Each can destroy evidence, increase load, or create correctness risk.

The core question is: can the team identify the failing boundary, reduce demand reversibly, and protect durability before attempting engine-specific recovery?

The First Fifteen Minutes

flowchart TD
    A["Page received"] --> B["Freeze deploys and name incident roles"]
    B --> C["Capture UTC window, client errors, latency, and demand"]
    C --> D{"Retry or traffic amplification"}
    D -->|Yes| E["Bound retries, concurrency, and low-priority traffic"]
    D -->|No| F["Continue evidence capture"]
    E --> G["Identify hot key, quorum loss, election, or resource saturation"]
    F --> G
    G --> H["Apply one reversible mitigation"]
    H --> I["Verify errors, tail latency, backlog, and correctness"]

Record an explicit UTC start and end time using portable ISO-8601 timestamps supplied by the incident commander; do not embed platform-specific date syntax in the runbook.
Freeze unrelated deploys and configuration changes.
Capture client request rate, retry rate, error codes, p50/p99 latency, backlog age, and recent topology changes.
Apply admission control before scaling retries. Preserve a small diagnostic traffic lane.
Make one reversible change at a time and annotate the timeline.

Engine-Specific Evidence and Safe Actions

DynamoDB

Separate throttling from service errors and client timeouts. Inspect table and index consumption, throttled events, account and table quotas, and key contributors. On-demand mode is elastic but has warm-throughput rules: a jump above twice the previous peak inside 30 minutes may throttle. AWS on-demand capacity.

Safe first actions are bounded exponential backoff with jitter, concurrency reduction, low-priority shedding, and targeted pre-warming or quota work when the evidence supports it. Do not state a fixed internal leader-failover duration; DynamoDB does not expose such a universal SLO as an application runbook primitive.

Cassandra

Capture nodetool status, nodetool describecluster, nodetool tpstats, nodetool proxyhistograms, nodetool compactionstats, table histograms, disk and network saturation, GC logs, and client consistency levels. Distinguish an unavailable error from a timeout: one says insufficient live replicas were known; the other says acknowledgements did not arrive before the deadline.

Do not lower gc_grace_seconds during an incident. Tombstones protect against deleted data returning from a replica that missed the delete; Apache Cassandra documents that a node absent beyond the grace period can reintroduce data. Cassandra tombstones.

Snapshots alone are not point-in-time recovery. Commit-log replay requires commit-log archiving and restore configuration established and tested before the incident. Cassandra backup and restore.

MongoDB

Inspect replica-set state, election and replication logs, replication lag, oplog window, currentOp, shard availability, config-server health, balancer state, and range deletion. An election may temporarily reject writes, but duration depends on configuration and conditions.

Do not stop the balancer by default. MongoDB documents it as the background mechanism that restores range balance; pausing it is justified only when active migration is demonstrably competing with recovery, and the incident timeline must include when to resume it. MongoDB balancer administration.

Bigtable

Inspect request latency, errors, CPU load, node count and autoscaling state, key distribution, Key Visualizer, and hot-tablet evidence. Bigtable exposes tablet ranges and cluster metrics; avoid runbooks that presume direct control of a named worker. Bigtable hot-tablet diagnostics.

If a key range is hot, shed or coalesce requests before changing row-key design. A schema change needs dual-write and backfill; it is rarely a safe 2 AM mutation.

Recovery and Verification

A falling error rate is not enough. Verify:

original request rate, retry ratio, and queue age are converging;
p99 latency stays within objective for at least one workload cycle;
replica and topology state are stable;
no acknowledged writes were lost and duplicate handling remains correct;
deferred repairs, balancing, compaction, or backfill have an owner and capacity budget;
temporary traffic controls have explicit removal criteria.

Restore traffic in steps. If a retry queue exists, cap replay so it cannot starve current traffic. Reconcile by immutable event ID and mutation version, not by approximate counts alone.

In Practice

This runbook derives from documented behavior: DynamoDB publishes warm-throughput rules; Cassandra documents tombstone safety and backup mechanisms; MongoDB documents range balancing; Bigtable provides hot-tablet diagnostics. It intentionally does not invent universal failover seconds or prescribe destructive commands without topology-specific evidence.

Where It Breaks

Response error	Why it is dangerous	Guardrail
Unbounded retries	Multiplies demand on the failing owner	Retry budget, jitter, bounded concurrency
Restart first	Destroys cache and evidence; may trigger movement	Capture state and prove the unhealthy component
Lower Cassandra grace	Can permit zombie resurrection	Change only through repair-validated design review
Pause MongoDB balancing blindly	Preserves skew and creates forgotten debt	Require evidence, owner, and resume condition
Replay at full speed	Recovery traffic causes a second outage	Rate-limit against spare capacity and queue age
Treat snapshot as PITR	Misses changes after the snapshot	Preconfigure and test engine-specific continuous recovery

What to Do Next

Problem: Generic recovery actions can amplify load or damage correctness.
Solution: Standardize evidence capture and demand control, then branch by engine mechanism.
Proof: Run game days that include retry storms, a hot key, replica loss, and bounded replay.
Action: Validate backup restoration and record the last successful restore test—not merely the last backup job.