Capacity Planning Cassandra: When Do You Add Nodes?

In a stateless application tier, scaling is a one-dimensional problem: if CPU utilization crosses 70%, you add another container. If you apply this exact same logic to a stateful, distributed database like Apache Cassandra, you will experience a catastrophic production outage.

Situation

A 36-node Cassandra 4.1 cluster housed a 32 TB dataset and pushed 120k write QPS and 45k read QPS. The EC2 instances were hovering around 55% CPU utilization — a number that looks “healthy” if CPU is the only scaling signal being watched, which is exactly the trap this post is about.

This is a documented Cassandra failure pattern, not a one-off: clusters sized on CPU alone get blindsided by disk. A daily traffic peak triggered a heavy background compaction cycle. Disk utilization on several nodes spiked from 65% to 95%. Cassandra’s own low-disk-space guardrails kicked in and the affected nodes stopped accepting writes to avoid running out of disk mid-flush. With disk headroom gone, the anti-compaction step that repair depends on had nowhere to write new SSTables, so repair jobs stalled and failed outright. The cluster suffered a multi-hour partial outage while the team freed disk space node by node to restore write availability.

The Problem

Cassandra is a log-structured merge-tree (LSM) database. It does not update data in place. It writes immutable files (SSTables) to disk, and then a background process called compaction merges those files together to reclaim space.

If you are using the default Size-Tiered Compaction Strategy (STCS), when Cassandra merges four 100 GB SSTables, it must write a new 400 GB SSTable to disk before deleting the original 400 GB of data. This means that at the moment of compaction, disk space requirements temporarily double.

Sizing a Cassandra cluster based on average steady-state read/write metrics completely ignores:

1. Anti-entropy repair space amplification.
2. The massive disk headroom required for STCS compactions.
3. The disk I/O required to stream data when a new node bootstraps.
4. Snapshot storage accumulation.

If you wait until disks are 80% full to add nodes, you have already lost. The new nodes will require gigabytes of data to be streamed from the existing nodes, but the existing nodes won’t have the disk I/O or space headroom to safely execute the read streams.

The Multi-Dimensional Capacity Formula

We abandoned CPU as the primary scaling trigger and established a physical capacity budgeting framework based on storage headroom and disk IOPS.

flowchart TD
    ScaleDecision{Scale Out?}
    
    ScaleDecision --> CPU[CPU < 60%]
    ScaleDecision --> Thread[Mutation/Read — Queue Depths < 100]
    ScaleDecision --> Storage[Disk Headroom]
    ScaleDecision --> IOPS[EBS Burst Balance / IOPS]
    
    Storage --> STCS[STCS: Max 50% Full]
    Storage --> LCS[LCS/TWCS: Max 70% Full]
    Storage --> UCS[UCS: Variable Headroom]
    
    IOPS --> WriteAMP[Write Amplification]
    IOPS --> ReadAMP[Read Amplification + Bloom Filter Miss]

1. Storage Headroom Rules

The runbook establishes strict alerts on disk utilization:

• For tables using STCS, an EC2 instance must never exceed 50% disk utilization during steady state.
• For tables using LCS or TWCS, it must never exceed 70% disk utilization.

The capacity formula became: Total Required Storage = Dataset Size × Replication Factor × 1.4 (Headroom) + Snapshots

The 1.4 multiplier comes directly from the 70% max-utilization rule above (1 ÷ 0.7 ≈ 1.4) — for STCS tables capped at 50% utilization, the equivalent multiplier is 2.0, so size per-table headroom according to whichever compaction strategy that table actually uses.

2. IOPS Budget per Mutation

Cassandra converts random writes into sequential disk I/O via the commitlog, but compactions convert those writes back into massive sequential read/write operations. If you are using EBS gp3, you must provision IOPS not for your client write rate, but for your compaction and repair rates. We provisioned dedicated 12,000 IOPS gp3 volumes for data directories to ensure background compactions never stole IOPS from client read requests.

In Practice

The documented pattern for Cassandra capacity management relies on Prometheus and Grafana forecasting based on nodetool cfstats and nodetool tablestats.

Instead of alerting on “Disk is 80% full” (which is an incident, not a warning), the pattern builds a derivative alert for “Days to 60% Disk Capacity.” This calculates the daily growth rate of TotalDiskSpaceUsed and LiveDiskSpaceUsed, triggering a capacity planning ticket if the cluster is projected to cross the 60% threshold within 30 days. This provides the required lead time to procure hardware, update Terraform, and bootstrap nodes sequentially using zero-copy streaming.

Because of how Cassandra’s architecture handles ring expansion, adding nodes does not instantly free up disk space on the existing cluster. The documented behavior is that existing nodes will stream their token ranges to the new nodes, but they do not automatically delete the data they no longer own. Operators must manually execute nodetool cleanup on the original nodes that lost token ranges after the expansion completes to reclaim the un-owned disk space. If a team initiates a scale-out too late, they will lack the disk headroom required to even run this cleanup compaction safely.

If a cluster unexpectedly breaches the capacity threshold, the documented emergency mitigation pattern is:

1. Clear Snapshots: Run nodetool clearsnapshot across the cluster to instantly reclaim disk space from old backup files.
2. Throttle Compaction: Execute nodetool setcompactionthroughput to reduce I/O pressure if CPU I/O wait is starving the read path, buying time for an emergency expansion.

Where It Breaks

Design Choice	Tradeoff	Mitigation
Using CPU as a scaling metric	High I/O wait can make CPU look artificially low because JVM threads are blocked waiting on disk.	Monitor iostat -xz 1 to correlate CPU utilization with disk await times.
Scaling too late	Adding nodes when disks are > 75% full causes donor nodes to crash during token streaming.	Enforce strict 50-60% disk utilization scaling triggers.
Ignoring Tombstones	Deletes take up space until gc_grace_seconds passes. Massive delete workloads will fill your disk before freeing it.	Run nodetool tablestats to monitor tombstone payload sizes.

Modern Cassandra Context: Cassandra 5.0 and UCS

Cassandra 5.0’s Unified Compaction Strategy (UCS) fundamentally alters this math. By dynamically adapting fanout and sharding, UCS can materially reduce space amplification for appropriate configurations. However, operational disk headroom must still account for workload growth, compaction bursts, bootstrap streaming, repair operations, and pre-upgrade snapshots. Furthermore, if you are leveraging Storage-Attached Indexing (SAI) for vector search, you must calculate the off-heap memory requirements for JVector’s ANN graph structures alongside your physical disk capacity.

What to Do Next

• Problem: Sizing Cassandra clusters based on CPU or allowing disks to exceed 60% capacity results in cascading failures during compaction or repair.
• Solution: Baseline capacity scaling triggers on strict disk headroom rules (50% for STCS) and IOPS budgets, not CPU.
• Proof: Enforcing a “Days to 60% Disk” alert gives teams the 30-day runway required to safely stream data to new nodes without starving client I/O.
• Action: Run nodetool tablestats on your heaviest keyspace. Look at Space used (live) versus Space used (total). That difference is your current compaction overhead.