In a small Cassandra cluster, anti-entropy repair is a background maintenance task. In a 90-node, multi-region cluster holding 120 TB of data, repair is a distributed systems engineering problem that can easily destroy the database it was meant to protect.

Situation

By 2024, the Cassandra 4.1 cluster had scaled to 90 nodes spanning two AWS regions (6 Availability Zones, 15 nodes per AZ). We understood that anti-entropy repair was mandatory within the gc_grace_seconds window (10 days), but running the legacy repair scripts across 90 nodes created catastrophic operational incidents.

When the scripts fired, cross-region AWS network egress bills skyrocketed. CompactionStage queues overflowed on disk, and p99 read latencies on coordinator nodes spiked wildly as massive Merkle trees were computed across the network.

The Problem

Global repair across a multi-region cluster exposes fundamental physics limits in Cassandra.

If you execute a raw nodetool repair -pr across a 90-node cluster, Cassandra attempts to synchronize the entire primary token range of each node. Generating a Merkle tree for a 1 TB token range requires massive sequential disk I/O. Worse, if a token mismatch spans across the country, Cassandra initiates un-throttled inter-DC streaming, saturating expensive VPC Peering links.

Repairing an entire token range at once guarantees two things:

  1. Massive JVM heap usage to hold the Merkle tree.
  2. Complete failure if even a single node drops a connection midway through the 4-hour sync process.

How do you repair 120 TB of data continuously, every 10 days, without degrading client read latency or saturating cross-region network links?

Multi-DC Repair Architecture with Cassandra Reaper

To solve this, the migration decouples repair orchestration from the database layer and deployed Cassandra Reaper. Reaper is an external control plane designed specifically to handle the complexities of token sub-range chunking and multi-DC concurrency.

flowchart TD
    Reaper[Cassandra Reaper Control Plane] -->|Schedule API| JMX_DC1
    Reaper -->|Schedule API| JMX_DC2
    
    subgraph "AWS us-east-1 (DC1)"
        JMX_DC1(Reaper Agent) --> N1[Node 1]
        JMX_DC1 --> N2[Node 2]
    end
    
    subgraph "AWS us-west-2 (DC2)"
        JMX_DC2(Reaper Agent) --> N3[Node 3]
        JMX_DC2 --> N4[Node 4]
    end
    
    N1 <-->|Throttled Sub-Range Sync — 50 Mbps| N3
    
    %% Reaper logic
    Token[Large Token Range] -.->|Reaper splits into| Sub1[Sub-Range 1]
    Token -.->|Reaper splits into| Sub2[Sub-Range 2]
    Token -.->|Reaper splits into| Sub100[Sub-Range 100]

1. Sub-Range Repair

Instead of repairing the entire token range of a node at once, Reaper dynamically calculates the token boundaries and splits them into hundreds of tiny sub-ranges (e.g., 256 segments per node). If a segment fails to sync, Reaper simply retries that tiny segment without throwing away hours of work.

2. DC-Aware Parallelism

A standard pattern configures Reaper to use DATACENTER_AWARE repair parallelism. This ensures that repairs are isolated within a local datacenter first, validating local replicas against each other, before synchronizing across the expensive cross-region AWS links.

3. Strict Network Throttling

In cassandra.yaml and via nodetool, we enforced strict network streaming limits. We throttled general streaming but applied a much harder clamp on inter-DC streaming to protect the AWS VPC Peering bandwidth and prevent client request timeouts:

# Limit general streaming to 100 Mbps
nodetool setstreamthroughput 100

# Limit expensive cross-region streaming to 50 Mbps
nodetool setinterdcstreamthroughput 50

In Practice

A typical deployment involves Reaper backed by a highly available PostgreSQL database to persist repair segment states. Using Reaper’s REST API, we automated the scheduling of continuous rolling repairs.

Because Reaper continuously monitors the cluster via JMX, if PendingCompactions on a node breached 20, Reaper automatically paused the repair segment directed at that node, allowing the database to catch its breath. This eliminated the massive I/O thrashing we saw with the legacy cron jobs.

If a single replica became unresponsive, Reaper paused that specific segment schedule without blocking the remaining 89 nodes from continuing their repair workloads.

Where It Breaks

Design ChoiceTradeoffMitigation
High Segment CountSplitting a token range into 1,000 segments makes repair incredibly safe but very slow, potentially violating gc_grace_seconds.Monitor Reaper completion telemetry. Tune segment counts to balance safety with the 10-day deadline.
Anti-Compaction on Large NodesEven with sub-ranges, anti-compaction splits SSTables. If disk headroom is tight (< 20%), anti-compaction can fill the disk.Maintain adequate disk headroom. Ensure nodes are scaled out before they reach 75% capacity.
Un-throttled Inter-DC StreamsEven with Reaper, if inter_dc_stream_throughput_outbound_megabits_per_sec is unlimited, cross-region syncs will cause p99 read spikes.Always explicitly set setinterdcstreamthroughput.

Modern Cassandra Context: Cassandra 5.0 and Cost Optimizations

Cassandra 5.0 significantly optimized the Merkle tree generation code path, reducing the CPU cost of repair. Furthermore, its enhanced multi-DC repair coordination requires less raw bytes to be transferred. From a cloud FinOps perspective, managing inter-DC streaming directly correlates to reduced AWS Inter-Region VPC Peering costs—letting Reaper’s sub-range repairs and 5.0’s reduced Merkle-tree transfer volume minimize redundant cross-region data movement is a meaningful, if workload-dependent, line item to track rather than a fixed percentage.

What to Do Next

  • Problem: Unorchestrated repairs on large, multi-region clusters cause cross-DC network saturation, disk thrashing, and database freezes.
  • Solution: Deploy Cassandra Reaper to automate token sub-range chunking, enforce DATACENTER_AWARE parallel repair, and explicitly throttle inter_dc_stream_throughput.
  • Proof: Reaper’s backpressure awareness pauses repairs when PendingCompactions rise, eliminating client read-latency degradation during the 10-day repair cycle.
  • Action: Execute nodetool netstats during a repair. If you see gigabytes transferring simultaneously across regions, halt the repair via nodetool stop VALIDATION and rethink your orchestration.