A backup strategy that has never been restored is not a backup strategy—it is a hope strategy.

In Cassandra, because data is distributed across dozens of nodes, the concept of a “backup” is fundamentally different from taking an RDS snapshot. If you don’t understand how Cassandra manages file pointers, your backups will either silently fail or quietly fill up your disk until the cluster crashes.

Situation

In 2023, a developer running a migration script accidentally executed DROP TABLE on a critical production table in the 36-node cluster (housing ~35 TB of data).

When the infrastructure team attempted to restore the data, they discovered three fatal flaws:

  1. They had no automated schema backups, meaning they didn’t know the exact DDL to recreate the table.
  2. The snapshots were sitting locally on the EC2 instances. Half the instances were out of disk space because historical snapshots were never cleared.
  3. The restore runbook estimated 72 hours for a full recovery because it relied on running sstableloader over a slow network link for the entire 35 TB dataset.

The Problem

Cassandra backups do not copy data. When you run nodetool snapshot, Cassandra simply flushes the memtables to disk and creates a hardlink in the snapshots/ directory pointing to the existing SSTables.

This operation is instantaneous and atomic. However, because it is a hardlink, the backup only exists on the local EC2 disk. If the EC2 instance is terminated, the backup is gone. Furthermore, if you never run nodetool clearsnapshot, those hardlinks prevent the OS from reclaiming disk space when Cassandra compacts and deletes the original SSTable. The disk slowly fills to 100%.

To survive an AWS region failure or a dropped table, you must physically move those SSTable files off the EC2 instance to highly durable object storage (AWS S3) and capture the schema agreement.

The S3 Backup and Restore Architecture

The documented pattern is to engineer an automated Backup & Restore pipeline to guarantee strict RPO (Recovery Point Objective) and RTO (Recovery Time Objective) SLAs.

flowchart TD
    Cass[Cassandra Node] -->|nodetool snapshot| Link[Local Disk — Hardlinks created]
    Cass -->|DESCRIBE KEYSPACE| Schema[Schema DDL Capture]
    
    Link --> Agent(Backup Agent / Medusa)
    Schema --> Agent
    
    Agent -->|Throttled Upload| S3[(AWS S3 — Archival Storage)]
    
    S3 -.->|sstableloader| Remote[Cross-Topology Restore]
    S3 -.->|nodetool import| Local[In-Place Fast Restore]
    
    Agent -->|nodetool clearsnapshot| Purge[Purge Local Hardlinks]

1. Atomic Snapshots and S3 Offload

A standard pattern configures the nodes to run daily atomic snapshots. Immediately after the snapshot, a Python backup agent (we eventually adopted open-source Medusa) throttled the upload of those SSTables to an S3 bucket to prevent network saturation. Crucially, the agent automatically exported the schema via cqlsh -e "DESCRIBE KEYSPACE" and uploaded it alongside the data.

2. Incremental Backups

To reduce the RPO from 24 hours to 1 hour, we enabled incremental_backups: true in cassandra.yaml. This forces Cassandra to hardlink every newly flushed SSTable into a backups/ directory immediately. The agent continuously synced this directory to S3, giving us point-in-time recovery capabilities.

3. The Restore Paths

If we lost a single node, we didn’t use sstableloader. The documented pattern uses the zero-downtime In-Place Restore. We copied the files from S3 directly into the new node’s data directory and ran nodetool import <keyspace> <table> /path/to/sstables. This loads the SSTables directly into the live storage engine via local disk, without the network streaming overhead sstableloader incurs.

We only used sstableloader if we were restoring the data into a cluster with a completely different token ring topology (e.g., migrating from a 36-node cluster to an 18-node cluster).

In Practice

The most critical part of this architecture was the automated DR (Disaster Recovery) validation drill.

Every month, an automated pipeline spun up a sandboxed, single-node Cassandra cluster in a separate AWS account. It pulled the latest production schema from S3, recreated the keyspaces, downloaded a subset of the SSTables, and ran nodetool import. A script then ran a SELECT query to verify data integrity. If the query failed, the backup pipeline was immediately flagged as broken.

Where It Breaks

Design ChoiceTradeoffMitigation
Forgetting clearsnapshotHardlinks prevent the OS from freeing disk space. The disk will fill to 100% silently.Ensure your backup agent always runs nodetool clearsnapshot after a successful S3 upload.
Restoring without SchemaSSTables are useless binaries if you don’t know the partition key and clustering column definitions.Always back up the DDL schema in the exact same S3 prefix as the snapshot data.
sstableloader on Live Clusterssstableloader forces the receiving nodes to stream data through the network, destroying cluster performance.Use nodetool import (or nodetool refresh in older versions) for fast, local disk loading.

Modern Cassandra Context: Cassandra 5.0 and Backup Tooling

Cassandra’s virtual tables (system_views, since 4.0) expose operational state — caches, disk usage, SSTable tasks — as queryable CQL rows, but there is no dedicated snapshots virtual table as of 5.0; nodetool listsnapshots remains the way to inspect snapshot state. While AWS offers EBS-level snapshots, application-consistent Cassandra backups still require database-aware orchestration. Tools like Medusa remain the industry standard for bridging Cassandra’s distributed hardlinks with cloud-native S3 object storage.

What to Do Next

  • Problem: Relying on raw nodetool snapshot leaves data vulnerable to EC2 termination and silently exhausts disk capacity via orphaned hardlinks.
  • Solution: Deploy an agent (like Medusa) to offload snapshots and incrementals to S3, capture schema DDL, and automate the clearsnapshot purge.
  • Proof: Monthly automated DR restore drills guarantee that the S3 artifacts can actually be imported and queried, proving an RTO of minutes rather than days.
  • Action: Run nodetool clearsnapshot on your cluster right now. Then check your disk monitoring to see how many gigabytes you just recovered.