In a distributed database, entropy is guaranteed. Nodes crash, network packets drop, AWS EBS volumes stall, and AWS Availability Zones partition. When a Cassandra node misses a write, the database relies on two mechanisms to fix it: Read Repair (on the hot path) and Anti-Entropy Repair (in the background).

When Anti-Entropy Repair is misunderstood, it becomes a weapon of mass cluster destruction.

Situation

In 2022–2024, the 36-node cluster (~40 TB dataset) began exhibiting silent replica drift. Applications running queries with a LOCAL_ONE consistency level started returning inconsistent results depending on which replica answered.

To fix it, an engineer SSH’d into a node and typed nodetool repair.

Within minutes, disk I/O saturated, CPU spikes hit 100%, and p99 client latencies breached 5 seconds. nodetool netstats showed gigabytes of data streaming uncontrollably across the network. A background process called “anti-compaction” began tearing apart existing SSTables, blocking normal compactions and filling the disk to the brink of capacity.

The Problem

Cassandra’s anti-entropy repair is designed to synchronize data across replicas. To do this efficiently without sending the actual data across the network, Cassandra uses Merkle Trees—cryptographic hash trees representing the data in a specific token range.

When a repair starts:

  1. Replicas generate a Merkle tree by scanning their SSTables and hashing the data.
  2. The coordinator compares the trees.
  3. If a leaf node (a hash of a specific sub-range) mismatches, the replicas stream only that specific slice of data to resolve the conflict.

The problem was twofold. First, running raw nodetool repair without the -pr (Primary Range) flag tells the node to repair all token ranges it holds replicas for, causing massive redundant repair work across the cluster.

Second, the cluster was using Incremental Repair. Incremental repair attempts to optimize the process by splitting SSTables into two physical buckets on disk: repaired and unrepaired. Once data is repaired, it is never repaired again. However, the process of separating this data—anti-compaction—physically rewrites SSTables. Because the repair process was ad-hoc, massive anti-compactions were colliding with normal traffic, fragmenting the disk, and locking up I/O threads.

The Anti-Entropy Architecture

To control entropy without destroying performance, you must understand the Merkle tree exchange.

flowchart TD
    Coord(Coordinator Node) -->|Trigger Repair| N1(Replica A)
    Coord -->|Trigger Repair| N2(Replica B)
    
    subgraph "Merkle Tree Construction"
        N1 --> M1[Compute Merkle Tree — Hash of Token Ranges]
        N2 --> M2[Compute Merkle Tree — Hash of Token Ranges]
    end
    
    M1 -.->|Exchange Hashes| Coord
    M2 -.->|Exchange Hashes| Coord
    
    Coord --> Compare{Hash Mismatch — in Leaf Node?}
    Compare -->|No| Done[Data is Consistent]
    Compare -->|Yes| Stream[Stream Mismatched — Token Range]
    
    Stream --> Sync[(Replicas Synchronized)]

1. Ditching Incremental Repair for Managed Full Repair

While Incremental Repair sounds theoretically optimal, the resulting anti-compaction fragmentation in Cassandra 4.0 often causes more operational pain than it solves. Fix: The pattern standardizes on Full Primary Range Repairs (nodetool repair --full -pr). The -pr flag ensures a node only repairs the token ranges it is the primary owner of, preventing duplicate work across the ring.

2. Time-Bounding the Repair (gc_grace_seconds)

Every token range in the cluster must be fully repaired before gc_grace_seconds expires (default 10 days). If a node misses a delete (tombstone) and is not repaired within 10 days, the tombstone is purged from healthy nodes. If the offline node comes back, the deleted data will be “resurrected” because the healthy nodes forgot it was deleted.

In Practice

The documented pattern across the industry is to strictly forbid manual nodetool repair executions in production. Instead, teams adopt a dedicated repair control plane, with Cassandra Reaper being the open-source standard.

Cassandra’s architecture requires that every token range in the cluster is fully repaired before gc_grace_seconds expires (default 10 days). If a node is offline and misses a delete, and a repair does not synchronize that tombstone before the grace period ends, the tombstone is purged from the healthy nodes. When the offline node is restored, the deleted data will be “resurrected” because the healthy nodes have forgotten the deletion ever occurred.

To manage this safely, Reaper automatically computes the token ring and splits it into tiny segments (sub-range repair). It schedules these small segments sequentially across the cluster.

Operators verify the state of repair on disk using offline tools, and maintain emergency commands to halt runaway processes:

# Verify the repair status of an SSTable
sstablemetadata /var/lib/cassandra/data/ks/tbl/*-Data.db | grep -i "Repaired at"

# Abort a runaway repair immediately (kills the validation compactions driving it)
nodetool stop VALIDATION

By actively monitoring cluster backpressure, Reaper will automatically pause repair schedules if PendingCompactions rise too high, ensuring that background anti-entropy never starves the client read and write paths.

Where It Breaks

Design ChoiceTradeoffMitigation
Running Repair on Down NodesIf a replica is offline, the remaining nodes generate gigantic Merkle trees but cannot complete the sync, wasting CPU and memory.Stop all repairs if a node is marked DOWN in nodetool status.
Too Many Concurrent RepairsMerkle tree generation requires massive sequential disk reads.Configure Cassandra Reaper to run only one segment repair at a time per datacenter.
Ignoring Sub-Range RepairRepairing an entire vnode token range at once generates massive Merkle trees that consume excessive JVM heap.Always split repairs into smaller sub-ranges. (Reaper does this automatically).

Modern Cassandra Context: Cassandra 5.0 Enhancements

Cassandra 5.0 GA significantly stabilizes the repair pipeline. It introduces native sub-range repair optimizations and leverages zero-copy streaming during repair, drastically reducing the CPU and heap overhead previously required to synchronize mismatched data. However, the architectural mandate remains: full primary-range repairs orchestrated by a tool like Cassandra Reaper remain the gold standard for long-term data consistency on EC2.

What to Do Next

  • Problem: Unmanaged repairs cause CPU saturation, network flooding, and anti-compaction fragmentation.
  • Solution: Abandon manual incremental repairs. Adopt automated, throttled, full primary-range (-pr) repairs using sub-ranges.
  • Proof: Deploying Cassandra Reaper stabilizes PendingCompactions and guarantees 100% token range repair within the gc_grace_seconds window.
  • Action: Check your SSTables with sstablemetadata. If you see a mix of repaired and unrepaired data but no orchestrator is running, you are at risk of data resurrection. Install Cassandra Reaper.