In a Log-Structured Merge-Tree (LSM) database like Cassandra, compaction is the heartbeat of the storage engine. If compaction falls behind, reads become exponentially slower. If compaction runs too aggressively, disk I/O saturates, and writes are blocked.

Choosing the right compaction strategy is an exercise in balancing write amplification, read amplification, and space amplification.

Situation

In 2023, the 36-node Cassandra 4.1 cluster was managing 15 distinct production tables across 3 keyspaces. We were experiencing two severe, isolated incidents simultaneously.

First, a massive time-series table exhausted 85% of its disk space during a major compaction, triggering capacity alerts. Second, a read-heavy key-value table suffered 20ms p99 read latencies, timing out upstream microservices.

Both tables were using Cassandra’s default: Size-Tiered Compaction Strategy (STCS).

The Problem

When Cassandra flushes data from memory, it writes an immutable SSTable to disk. As these files accumulate, Cassandra must read them, merge the data, resolve deleted data (tombstones), and write a new, larger SSTable.

STCS is optimized for write-heavy workloads. It waits for multiple SSTables of similar size and merges them. However, it carries two fatal flaws for specific workloads:

  1. Space Amplification: Because STCS requires writing the new, merged file to disk before deleting the old files, it temporarily requires 50% free disk space. For the time-series table, massive historical files were constantly being rewritten, consuming terabytes of overhead.
  2. Read Amplification: For the read-heavy lookup table, STCS allowed data for a single partition key to spread across 18 different SSTables. To satisfy a read request, Cassandra had to seek across 18 separate files on disk, causing the 20ms latency spikes.

The Compaction Strategy Matrix

Operators must align the compaction strategy to the physical access pattern of the data.

flowchart TD
    Data[Table Workload Profile] --> Eval{Access Pattern}
    
    Eval -->|Time-Series — Append Only| TWCS[Time-Window Compaction Strategy]
    Eval -->|Read-Heavy Lookup| LCS[Leveled Compaction Strategy]
    Eval -->|Write-Heavy — Updates| STCS[Size-Tiered Compaction Strategy]
    Eval -->|Modern Cassandra 5.x and later| UCS[Unified Compaction Strategy]
    
    TWCS -->|Isolates data by time buckets — Zero-cost TTL drops| Benefit1[Bounds Space Amplification]
    LCS -->|Guarantees 1-2 SSTables per read — High write amplification| Benefit2[Bounds Read Amplification]
    STCS -->|Low write amplification — High space amplification| Benefit3[Fast Ingestion]
    UCS -->|Dynamically adapts — Shards across disks| Benefit4[Unifies Tradeoffs]

1. Migrating to Time-Window Compaction Strategy (TWCS)

For the time-series table, we migrated the schema to TWCS. TWCS groups SSTables by time buckets (e.g., 1 DAY). It compacts files within the window, but never merges data across windows. This means old data is never rewritten, eliminating STCS space amplification. Furthermore, when the TTL expires, Cassandra simply drops the entire window from disk without requiring a compaction cycle.

2. Migrating to Leveled Compaction Strategy (LCS)

For the read-heavy table, we migrated to LCS. LCS organizes data into levels (L0, L1, L2), where each level is 10x larger than the previous. Crucially, LCS guarantees that 90% of reads will touch exactly one SSTable. This immediately solved the read amplification issue, bringing p99 latency back down to 2ms.

In Practice

Migrating a live production table requires careful execution. Altering a table to TWCS is relatively safe, but altering a large STCS table to LCS online triggers a massive I/O storm as Cassandra frantically reorganizes all existing data into Level 0 and Level 1.

-- Safe migration for time-series data
ALTER TABLE metrics.sensor_data 
WITH compaction = {
    'class': 'TimeWindowCompactionStrategy', 
    'compaction_window_unit': 'DAYS', 
    'compaction_window_size': 1
};

-- For LCS, we ensured NVMe instances and temporarily boosted throughput
ALTER TABLE lookups.user_profiles 
WITH compaction = {
    'class': 'LeveledCompactionStrategy', 
    'sstable_size_in_mb': 160
};

We temporarily increased compaction_throughput_mb_per_sec to 128 MB/s via nodetool to help the nodes chew through the LCS conversion backlog, carefully monitoring nodetool compactionstats to ensure client latencies remained stable.

Where It Breaks

Design ChoiceTradeoffMitigation
STCS on Large VolumesSTCS merging a 1 TB file requires 1 TB of free disk space.Never let STCS tables exceed 50% physical disk capacity.
LCS Write AmplificationLCS rewrites data constantly as it promotes SSTables through levels, burning disk IOPS.Only use LCS on read-heavy workloads. Ensure underlying storage is NVMe or provisioned IOPS EBS (gp3).
TWCS Out-of-Order WritesIf you write data with a timestamp from 30 days ago, it breaks the TWCS time windows, forcing old windows to compact and destroying the TTL drop mechanism.Strictly enforce append-only sequential writes at the application layer when using TWCS.

Modern Cassandra Context: Cassandra 5.0 and UCS

The operational pain of picking the “least bad” compaction strategy is exactly why Cassandra 5.0 introduces the Unified Compaction Strategy (UCS). UCS replaces STCS, LCS, and TWCS with a single algorithm that dynamically adapts to the workload. It introduces sharding, allowing it to cap space amplification like LCS while retaining the write speed of STCS. STCS remains the configured default in Cassandra 5.0 for backward compatibility, but UCS is the recommended strategy for most workloads going forward — adopting it still requires an explicit ALTER TABLE per table, it isn’t automatic, but it does end the practice of hand-picking STCS vs. LCS vs. TWCS per table based on write/read/TTL tradeoffs.

What to Do Next

  • Problem: Default STCS compaction causes disk capacity exhaustion on large tables and latency spikes on read-heavy workloads due to read amplification.
  • Solution: Align compaction strategies to workloads: use TWCS for time-series/TTL data, LCS for read-heavy lookups, and ensure adequate disk throughput.
  • Proof: Converting time-series tables to TWCS instantly halted 50% disk capacity swings, and LCS bounded read queries to a single SSTable lookup.
  • Action: Run nodetool tablehistograms <keyspace> <table>. If the SSTables metric frequently exceeds 3, your compaction strategy is failing your read path.