Cassandra is notorious for punishing bad garbage collection tuning. Because Cassandra is a heavily concurrent, write-optimized database relying on immutable memtables, it generates massive amounts of short-lived objects. If the JVM fails to collect these efficiently, you get stop-the-world (STW) pauses.

Situation

In 2022–2023, the 36-node Cassandra 4.1 cluster running on OpenJDK 11 (r5.2xlarge, 64 GB RAM, 31 GB Heap) was suffering from periodic 800ms to 2,000ms latency spikes.

These spikes resulted in Cassandra client read timeouts (ReadTimeoutException), dropped coordinator mutations, and sporadic gossip flapping where nodes temporarily marked each other as Dead because they failed to respond to heartbeats.

The Problem

We examined the /var/log/cassandra/gc.log and found the classic signatures of G1GC struggling under load. We saw frequent Concurrent Mark aborts, mixed collections failing with To-space exhausted, and major spikes in nodetool gcstats.

The Garbage-First (G1) collector works by dividing the heap into regions. Objects are allocated in Eden regions, survive into Survivor regions, and are eventually promoted to Old regions. However, if an object is larger than 50% of a G1 region size, it is classified as a “Humongous Allocation” and bypasses the young generation entirely, allocating directly into the Old generation.

Because legacy tuning files from older Cassandra architectures had been carried over, the JVM was misconfigured. The JVM was configured with a fixed young generation size (-Xmn), which forcibly disabled G1’s ability to automatically tune itself (G1 ergonomics). Furthermore, wide CQL partitions were causing humongous allocations, fragmenting the Old generation until it ran out of free space to copy objects during collection, triggering catastrophic To-space exhausted STW evacuation failures.

The Tuning Architecture

G1GC is designed to be largely self-tuning, provided you don’t cripple its ergonomics and you explicitly define the boundaries of your workload.

flowchart TD
    App[CQL Query Data Allocation] --> Eval{Object Size > 50% of Region?}
    
    Eval -->|No: Standard Allocation| Eden[Eden Regions]
    Eden -->|Minor GC| Surv[Survivor Regions]
    Surv -->|Promotion| Old[Old Generation Regions]
    
    Eval -->|Yes: Humongous Allocation| Humong[Humongous Regions — Directly in Old Gen]
    
    Humong --> Frag[Old Gen Fragmentation]
    Frag --> Ex[Evacuation Failure — To-space Exhausted]
    Ex --> STW[Massive Stop-The-World Pause]
    Old --> Mixed[Mixed GC — Concurrent Mark]

1. Let G1 Control the Young Generation

The single biggest mistake engineers make with G1 is explicitly setting -Xmn or -XX:NewRatio. If you set a fixed young generation, G1 cannot resize Eden to meet your -XX:MaxGCPauseMillis target. Fix: We removed -Xmn, allowing G1’s ergonomics to dynamically size the young generation based on the pause target.

2. Defeat Humongous Allocations

On a 31 GB heap, the default G1 region size is usually 16MB (G1’s ergonomics target roughly 2048 regions, and 31744MB ÷ 2048 rounds up to the nearest power of two). Any object over 8MB is humongous. Cassandra reads, especially wide partitions, easily exceed 8MB in memory. Fix: Region size itself isn’t usually the lever — it’s already sized correctly by G1’s ergonomics for a given heap. The actual fix is upstream: find the CQL queries and partition shapes producing 8MB+ objects (nodetool tablestats surfaces oversized partitions) and redesign the schema, since no IHOP or region-size tuning survives a data model that keeps generating humongous allocations. IHOP tuning (below) reduces the blast radius of humongous allocations that do occur — it doesn’t prevent them.

3. Tuning the IHOP

G1 triggers a Concurrent Mark (to identify old garbage) when the Old generation reaches the Initiating Heap Occupancy Percent (IHOP). If it triggers too late, G1 runs out of space before it can finish cleaning, causing the dreaded To-space exhausted pause. Fix: Java 11’s adaptive IHOP recalculates its trigger point from observed allocation and marking-cycle duration — it’s a reasonable default when object lifetimes are stable. Under the humongous-allocation churn this cluster was seeing, the adaptive estimate lagged reality, so the runbook pins a static baseline of -XX:InitiatingHeapOccupancyPercent=40 (deliberately overriding the adaptive behavior, not layering on top of it) along with -XX:G1ReservePercent=15 to give G1 a larger safety buffer for object copying.

In Practice

Based on Oracle Java 11 G1GC tuning guidelines and Apache Cassandra production practices, a typical deployment involves the following jvm11-server.options:

# jvm11-server.options
-Xms31G
-Xmx31G
# Max 31G keeps JVM Compressed OOPs active. Do NOT use 32G.

-XX:+UseG1GC
-XX:G1RSetUpdatingPauseTimePercent=5
-XX:MaxGCPauseMillis=200

# Pin IHOP instead of relying on adaptive estimation, and give G1
# extra reserve space to breathe during mixed collections
-XX:InitiatingHeapOccupancyPercent=40
-XX:G1ReservePercent=15

After deploying these settings via a rolling restart, To-space exhausted errors vanished entirely. The MaxGCPauseMillis=200 target was consistently respected, bringing p99 latencies down to stable bounds.

Where It Breaks

Design ChoiceTradeoffMitigation
MaxGCPauseMillis too lowSetting -XX:MaxGCPauseMillis=50 forces G1 to aggressively shrink the young generation, causing it to run Minor GCs constantly. CPU burn spikes massively.Set realistic targets (200ms–500ms). Cassandra is a database, not high-frequency trading software.
Heap > 32GBSetting -Xmx32G or -Xmx64G disables JVM Compressed OOPs (Ordinary Object Pointers). Pointers double in size from 32-bit to 64-bit, instantly consuming 30-50% more RAM just to track objects.Strictly limit heap to 31 GB (or 31744m). Rely on off-heap memory and the OS page cache for the remaining system RAM.
Humongous Allocations persistIf CQL queries still request 50MB partitions, tuning region size won’t save you.Fix the data model. Run nodetool tablestats to find oversized partitions and redesign the schema.

Modern Cassandra Context: Cassandra 5.0 on JDK 17

Cassandra 5.0 officially supports OpenJDK 17. JDK 17 introduces significant G1 enhancements, better adaptive IHOP calculations, and string deduplication improvements. Furthermore, the introduction of Trie Memtables significantly reduces on-heap pressure. Generational ZGC (JEP 439) only became production-ready in JDK 21, so it isn’t an option on JDK 17 — G1GC remains the most battle-tested and predictable collector for massive-scale EC2 deployments on JDK 17, and is worth re-evaluating against Generational ZGC only once the fleet moves to JDK 21+.

What to Do Next

  • Problem: Fixed young generations and humongous allocations cause G1 to run out of copy space, resulting in multi-second database freezes.
  • Solution: Remove -Xmn, explicitly tune InitiatingHeapOccupancyPercent, set a realistic pause target, and cap the heap at 31 GB to preserve Compressed OOPs.
  • Proof: Eliminating To-space exhausted messages from gc.log stabilizes p99 latency bounds immediately.
  • Action: Execute jcmd $(pgrep -f CassandraDaemon) GC.heap_info and verify your current region size, then tail /var/log/cassandra/gc.log looking for the word humongous.