Cassandra 5.0’s headline features — Unified Compaction Strategy, Storage-Attached Indexes, and Trie memtables — all ship with defaults tuned for a generic workload, not your workload. Teams that adopt 5.0 and never touch these defaults get correct behavior, but not the performance the release was built to deliver.

Situation

A team has upgraded a cluster to Cassandra 5.0 and migrated tables from STCS or LCS to the new Unified Compaction Strategy (UCS), added a Storage-Attached Index (SAI) to support a new query pattern, and is running on Trie memtables by default. Correctness is fine — queries return the right results, compaction runs, indexes build. But nobody has gone back to actually tune any of the three for the workload’s actual shape, and the read/write amplification profile looks suspiciously similar to what STCS was already producing.

The Problem

All three of these features expose their most important tuning knob as a default that assumes nothing about your access pattern. UCS’s default scaling_parameters (T4) behaves like tiered compaction, optimized for write throughput over read amplification — a defensible generic default, but wrong for a read-heavy table that would benefit from leveled-style behavior instead. SAI has a build-order constraint that isn’t obvious until you hit it: building the index concurrently with a large data load is unsupported and slow, but nothing stops you from doing it. And Trie memtables reduce on-heap pressure versus the legacy skip-list memtable, but that memory profile shift interacts with G1GC heap sizing decisions made before the upgrade.

The core question: which of UCS, SAI, and Trie memtables actually needs explicit tuning for this workload, versus which one is fine left alone?

Tuning Surface Across the Three Features

Each feature has exactly one high-leverage tuning decision; the rest of their configuration surface is genuinely fine at defaults.

flowchart TD
    Upgrade[Cassandra 5.0 Upgrade] --> UCS{Compaction — UCS}
    Upgrade --> SAI{Indexing — SAI}
    Upgrade --> Trie{Memtable — Trie}

    UCS --> UCSWorkload{Read-Heavy or Write-Heavy?}
    UCSWorkload -->|Read-Heavy| Leveled[Leveled Scaling — L10, Higher Shard Count]
    UCSWorkload -->|Write-Heavy or Time-Series| Tiered[Tiered Scaling — T4 or T8, Fewer Shards]

    SAI --> SAIOrder{Index Before or After Data Load?}
    SAIOrder -->|Before Load| Fast[Index Builds Cleanly, No Rebuild Risk]
    SAIOrder -->|After Load, Concurrent| Slow[Unsupported, Slow Concurrent Build]

    Trie --> TrieHeap[Re-Evaluate Heap Sizing Post-Upgrade]

1. UCS: Match scaling_parameters to the Read/Write Shape

UCS’s default configuration uses scaling_parameters: T4 (tiered behavior, threshold 4) with target_sstable_size: 1GiB and base_shard_count: 4 — a write-throughput-favoring default, the same tradeoff STCS always made. For a read-heavy table, the documented alternative is a leveled-style profile: L10 with a smaller target SSTable size (around 256MiB) and a higher shard count (around 8), which bounds the number of SSTables a read has to consult at the cost of more write amplification. For time-series data, T8 with a mid-size target (around 512MiB) and higher shard count improves both write throughput and time-window expiration checks. Unlike switching between STCS and LCS, UCS can change these parameters in flight without a full strategy migration.

ALTER TABLE lookups.user_profiles WITH compaction = {
    'class': 'UnifiedCompactionStrategy',
    'scaling_parameters': 'L10',
    'target_sstable_size': '256MiB',
    'base_shard_count': 8
};

2. SAI: Build the Index Before Loading Data

Cassandra’s own SAI documentation is explicit that index settings can’t be changed without dropping and rebuilding the index, and that building an index while data is concurrently loading is unsupported and measurably slower than building it first. This is the opposite order most teams assume when retrofitting an index onto an existing table.

-- Correct order for a new table: index first, then load
CREATE TABLE catalog.products (
    product_id uuid PRIMARY KEY,
    embedding vector<float, 768>,
    category text
);
CREATE CUSTOM INDEX ON catalog.products (embedding)
    USING 'StorageAttachedIndex';
-- Now bulk-load data

For an existing table with data already loaded, plan the index build as its own maintenance window rather than assuming it’s a background no-op.

3. Trie Memtables: Revisit Heap Sizing, Don’t Assume It’s Free

Trie memtables store data in a memory-efficient trie structure instead of the legacy skip-list, which measurably reduces on-heap pressure — this was called out as a contributing factor when tuning G1GC for a 4.1 cluster, since lower memtable heap pressure means fewer humongous allocations from wide-partition memtable flushes. But “reduces heap pressure” isn’t the same as “requires no heap tuning.” A heap sizing decision made for skip-list memtables on 4.1 should be re-validated after the 5.0 upgrade, not assumed to still be optimal — the safe direction to test is a smaller heap or a lower InitiatingHeapOccupancyPercent, since Trie’s lower baseline pressure changes the humongous-allocation math that drove those settings in the first place.

In Practice

Confirm which compaction strategy and scaling parameters a table is actually running before assuming the upgrade already tuned it:

nodetool compactionstats
nodetool describering <keyspace> | grep -A 2 compaction

Check SAI index build status before treating it as done:

nodetool sai --status <keyspace> <table>

Where It Breaks

Design ChoiceTradeoffMitigation
Leaving every table on UCS defaultsT4 favors write throughput; a read-heavy table pays unnecessary read amplification it didn’t need to accept.Set scaling_parameters per table based on actual read/write ratio, not the UCS default.
Building SAI concurrently with a bulk loadUnsupported and slow — the index build competes with the load for I/O and can leave the index in an inconsistent intermediate state.Create the index before loading data, or treat an index build on an existing table as its own maintenance window.
Assuming Trie memtables need zero heap re-tuningHeap settings carried over from a skip-list-memtable-era JVM configuration may now be more conservative than necessary.Re-run heap sizing validation (IHOP, region occupancy) after the 5.0 upgrade instead of carrying forward 4.x settings unchanged.

What to Do Next

  • Problem: Cassandra 5.0’s UCS, SAI, and Trie memtables all ship with generic defaults, so upgrading without re-tuning them delivers correctness but not the performance improvement the release is capable of.
  • Solution: Match UCS scaling_parameters to each table’s actual read/write shape, build SAI indexes before loading data rather than concurrently, and re-validate JVM heap settings post-upgrade rather than carrying forward 4.x tuning.
  • Proof: Cassandra’s own UCS documentation states parameters can be changed in flight without a full compaction-strategy migration — there’s no excuse to leave every table on the write-favoring default if its actual traffic is read-heavy.
  • Action: Run nodetool compactionstats against your busiest tables this week and check whether any of them are read-heavy but still running UCS’s default T4 tiered scaling.