A single misordered step in a major-version Cassandra upgrade can leave half a 90-node cluster unable to parse the other half’s schema, or make an on-disk format change that no longer accepts a rollback. That is the actual risk profile of moving a 90-node multi-region cluster (managing ~100 TB of data) from Apache Cassandra 4.1 to 5.0 — not whether the new release is worth adopting.

In 2024, this cluster needed to make that move. Cassandra 5.0 brings real architectural gains — Unified Compaction Strategy (UCS), Storage-Attached Indexing (SAI), native JDK 17 support — but unlocking any of them first required navigating a dangerous mixed-mode operational state without downtime.

Situation

We were faced with a complex matrix of upgrade requirements. Operators must upgrade the database binary (4.1 to 5.0), upgrade the JVM runtime (OpenJDK 11 to OpenJDK 17), and evolve the physical SSTable format on disk—all with zero downtime and no client errors.

A poorly orchestrated major upgrade triggers schema agreement failures, JVM crash loops, driver disconnection storms, and catastrophic read-path latency. We needed an orchestration sequence that decoupled risk.

The Problem

Cassandra is designed for rolling upgrades: older and newer nodes can coexist, gossip together, and continue serving reads/writes without a coordinated cutover. This state is called mixed-mode.

However, if an operator attempts to enable Cassandra 5.0-only features (like creating an SAI index or switching to UCS) while the cluster is still in mixed-mode, the 4.1 nodes will instantly fail to parse the schema or the new file structures.

Furthermore, upgrading both the Cassandra binary and the JVM runtime simultaneously introduces severe risk. If a node fails to boot, the operator doesn’t know if the failure was caused by the database binary upgrade or missing JDK 17 module encapsulation flags.

Finally, once Cassandra writes data in the 5.0 format (bti or 5.0 big), online rollback becomes functionally impossible. nodetool upgradesstables is an irreversible checkpoint.

The Decoupled Upgrade Architecture

To mitigate these risks, the pattern architects a decoupled major upgrade sequence. We separated the database upgrade from the runtime upgrade, and separated the binary deployment from the feature enablement.

flowchart TD
    Start[Current: C* 4.1 / JDK 11] --> Stage1[Stage 1: Upgrade Drivers & Test]
    
    Stage1 --> Stage2[Stage 2: Single 5.0 Canary Node — in Non-Critical DC]
    
    Stage2 --> Stage3[Stage 3: Rolling Upgrade across 90 Nodes — Keep JDK 11]
    
    Stage3 --> Wait{Wait for Cluster Stability — & Schema Agreement}
    
    Wait --> Stage4[Stage 4: nodetool upgradesstables — Irreversible Checkpoint]
    
    Stage4 --> Stage5[Stage 5: Enable 5.0 Features — UCS, SAI]
    
    Stage5 --> Stage6[Stage 6: Rolling JVM Upgrade — JDK 11 to JDK 17]

1. The Lower-Risk Decoupled Path

Cassandra 5.0 natively supports both Java 11 and Java 17. Rather than doing a combined “big bang” upgrade, we kept the cluster on Java 11 during the 5.0 binary rollout. This meant we could reuse the battle-tested jvm11-server.options without risking Java 17 JVM flag compatibility issues (like strict JMX encapsulation). Once the cluster was stable on Cassandra 5.0 for a week, we orchestrated a separate rolling restart to upgrade the runtime to Java 17.

2. The Canary Strategy

Before touching the primary nodes, we upgraded a single node in a non-critical rack. We let it serve reads and writes for 48 hours to ensure the monitoring agents, backup scripts, and metrics exporters didn’t break under the new 5.0 JMX metrics format.

3. SSTable Upgrades (nodetool upgradesstables)

When a node is upgraded to 5.0, it can still read 4.x SSTables, but reading legacy tables invokes a translation penalty that increases CPU usage. Once all 90 nodes were on 5.0 and schema agreed (cqlsh -e "SELECT schema_version FROM system.local;"), the playbook runs a background script to execute nodetool upgradesstables sequentially across the cluster. This rewrote the 100 TB of data into the new 5.0 format.

In Practice

We wrote an orchestration pipeline to manage the rolling restarts rack-by-rack across the two AWS regions.

We heavily monitored the driver reconnection rates and CPU utilization during the mixed-mode transition. Cassandra 5.0 provides storage compatibility modes specifically designed to make this smooth.

Crucially, the runbooks explicitly mandated taking pre-upgrade snapshots (nodetool snapshot) before initiating nodetool upgradesstables.

If a node crashed while simply running the 5.0 binary (before SSTables were rewritten), we could safely downgrade the binary back to 4.1 and restart. However, if the crash occurred after rewriting the SSTables, the 4.1 binary would fail to read the new 5.0 files on disk. The only safe recovery in that scenario is wiping the data directory and restoring the pre-upgrade snapshot.

Where It Breaks

Design ChoiceTradeoffMitigation
Combined JDK 17 UpgradeUpgrading C* 5.0 and JDK 17 simultaneously is faster but risks JVM flag crashes.Decouple the runtime upgrade. Stick to JDK 11 until the 5.0 binary is stable.
Skipping upgradesstablesNot rewriting SSTables saves disk I/O, but leaves a permanent CPU penalty on the read path.Automate nodetool upgradesstables via a low-priority background script.
Enabling Features Too EarlyChanging a keyspace to UCS before all 90 nodes are on 5.0 will instantly crash 4.1 node gossip and schema agreement.Gate feature enablement behind a strict script that verifies nodetool version across 100% of the ring.

Modern Cassandra Context: Long-Term 5.0 Performance

In the post-upgrade operational review, the combination of Cassandra 5.0 running on JDK 17 (utilizing ZGC/Shenandoah and Vector API optimizations) deployed on AWS Graviton3 instances produced a meaningful price-performance improvement — the exact magnitude depends heavily on workload shape and table-by-table UCS/SAI adoption, so treat any single number here as illustrative rather than a benchmark to plan around. By decoupling the upgrade, we safely reached this architecture with zero dropped client mutations.

What to Do Next

  • Problem: Upgrading 90 nodes across major database and JVM versions carries massive risk of downtime, schema forks, and irreversible file corruption.
  • Solution: Decouple the binary upgrade from the JVM upgrade. Use a canary, validate mixed-mode stability, and run nodetool upgradesstables only when the entire ring is ready.
  • Proof: The decoupled sequence allowed us to halt and investigate monitoring agent issues on JDK 11 without compounding the risk with JDK 17 incompatibilities.
  • Action: Check your system.local schema versions. Do not execute a single ALTER KEYSPACE command for 5.0 features until every node reports the exact same schema hash.