For a decade, operating Apache Cassandra at scale meant managing a delicate equilibrium. You had to choose the least-bad compaction strategy. You avoided secondary indexes at all costs. You wrestled with JVM Garbage Collection. You provisioned massive disk headroom to survive compaction spikes.

Cassandra 5.0 does not just add new features; it fundamentally replaces the physical storage engine to solve these historical constraints.

Situation

By late 2025, the 90-node cluster was holding 120 TB of data and serving a mix of transactional writes and new vector search workloads.

Under Cassandra 4.x, this architecture required heavy operational lifting. We had legacy tables stuck on Size Tiered Compaction Strategy (STCS) suffering from 50% space amplification. The attempts to run materialized views and legacy secondary indexes (2i) had resulted in memory fragmentation and unpredictable read latencies. The memtables churned massive amounts of JVM heap, forcing us to meticulously tune G1GC.

The Problem

Cassandra’s internal subsystems had hit architectural limits.

  1. Compaction: Operators had to choose between STCS (optimized for writes but requires 50% free disk space) and Leveled Compaction Strategy (LCS) (optimized for reads and space, but causes massive write amplification).
  2. Indexing: Legacy secondary indexes were global and didn’t share the SSTable lifecycle well. They were slow, expensive, and didn’t support vector search.
  3. Memory: Memtables and SSTable indexes were built on data structures that consumed massive amounts of JVM heap, forcing long GC pauses under heavy load.

Cassandra 5.0 solves these by overhauling the write path, the read path, the compaction engine, and the runtime.

The 5.0 Storage Engine Architecture

Cassandra 5.0 introduces three massive CEPs (Cassandra Enhancement Proposals) that interact to redefine how data is stored and indexed.

flowchart TD
    Client[Client Driver] --> Coord(Coordinator Node)
    Coord --> Guard[Dynamic Guardrails]
    
    Guard --> JDK[JDK 17 LTS / ZGC / Vector API]
    
    JDK --> TrieM[Trie Memtable — Off-Heap Allocator]
    
    TrieM -->|Flush| BTI[BTI Format Trie SSTable]
    TrieM -->|Flush| SAI[Storage-Attached Index — k-NN / HNSW Vector Graph]
    
    BTI --> UCS[Unified Compaction Strategy]
    SAI --> UCS
    
    subgraph "UCS Dynamic Sharding"
        UCS --> Shard1[Shard 1]
        UCS --> Shard2[Shard 2]
    end

1. Unified Compaction Strategy (UCS)

UCS (CEP-26) deprecates the need to choose between STCS and LCS. It introduces dynamic sharding and a configurable scaling parameter (W) per level. Instead of compacting massive SSTables into even larger ones, UCS breaks data into shards (e.g., 10 GB targets) and organizes SSTables into levels based on density (SSTable size relative to its owned token range). The scaling parameter determines whether a given level behaves tiered (like STCS) or leveled (like LCS) — it’s a configuration choice, not a runtime auto-detection — which is what lets one strategy replace hand-picking between STCS and LCS per table. The net effect is materially lower space amplification than STCS alone, though the exact reduction is workload- and configuration-dependent.

2. Storage-Attached Indexing (SAI)

SAI (CEP-7) replaces legacy 2i (and SASI, its storage-attached predecessor). SAI indexes are tied to the exact lifecycle of the SSTable they index, which meaningfully reduces disk usage compared to legacy global secondary indexes — the exact ratio depends on cardinality and data shape. Vector search (k-NN via HNSW/JVector) was added on top of SAI by a separate, later enhancement (CEP-30), and is exposed through the same StorageAttachedIndex interface alongside high-performance multi-dimensional queries directly within CQL.

3. Trie Memtables and BTI SSTables

Cassandra 5.0 replaces the legacy skip-list memtable with a concurrent prefix tree (Trie). This fundamentally changes serialization, cutting per-object allocation overhead and moving massive amounts of tracking data off the JVM heap. The on-disk Index.db and Summary.db are replaced by the BTI format, dramatically speeding up the read path.

4. JDK 17 Runtime

Running Cassandra 5.0 on OpenJDK 17 unlocks native support for ZGC and Shenandoah (reducing GC pauses to sub-millisecond ranges), Vector API optimizations for vector search math, and improved G1GC ergonomics out of the box.

In Practice

Migrating the 4.1 cluster to these features involved specific cassandra.yaml and schema changes. We transitioned the memtables to the new allocator:

# cassandra.yaml
memtable: trie
memtable_allocation_type: offheap_objects

For the tables, we altered the compaction strategy to UCS. Because UCS supports sharding, subsequent compactions began reshaping the massive monolithic SSTables into manageable chunks as new data was written and compacted — existing SSTables aren’t rewritten instantly, but the strategy change takes effect without a full cluster rebuild:

ALTER TABLE production.orders 
WITH compaction = {'class': 'UnifiedCompactionStrategy', 'target_sstable_size': '10GB'};

Finally, a typical deployment involves vector search for the ML workloads using SAI directly alongside the transactional data:

CREATE CUSTOM INDEX user_vector_idx 
ON production.users(embedding_vector) 
USING 'StorageAttachedIndex';

Where It Breaks

Design ChoiceTradeoffMitigation
Enabling SAI on High-Cardinality StringsSAI is incredibly efficient, but indexing an unbounded, totally unique UUID column will still balloon memory and disk requirements.Explicitly model memory requirements. Do not index every column blindly.
Misconfiguring UCS Shard SizesSetting the target_sstable_size too small on a multi-terabyte volume will generate thousands of tiny SSTables, exhausting OS file descriptors.Stick to the default UCS parameters initially (usually dynamic) before forcing specific shard targets.
Combined JDK 17 UpgradesBlindly upgrading the database binary and the JVM simultaneously masks root causes if a node fails to boot.Decouple the runtime upgrade. Get the cluster stable on C* 5.0 with JDK 11, then upgrade to JDK 17.

Modern Cassandra Context: 18 Months of Cassandra 5.0

After 18 months in production on AWS EC2, the GC-pause reduction from Trie Memtables and the space-reclamation efficiency of UCS let us shrink the EC2 instance sizes for this cluster, recovering meaningful cloud spend while absorbing new vector search workloads without bolting on an external, specialized vector database.

What to Do Next

  • Problem: Legacy compaction strategies, memory-heavy memtables, and brittle secondary indexes restrict Cassandra’s scalability and increase cloud costs.
  • Solution: Upgrade to Cassandra 5.0, adopt Unified Compaction Strategy (UCS), switch to Trie Memtables, and leverage Storage-Attached Indexing (SAI) on JDK 17.
  • Proof: Adopting UCS and Trie Memtables drastically reduces space amplification and eliminates To-space exhausted GC pauses.
  • Action: Before upgrading, run nodetool tablestats to identify tables suffering from high space amplification. These are your first candidates for an ALTER TABLE to UnifiedCompactionStrategy post-upgrade.