Cassandra is rarely the first database a team adopts. It usually enters the picture only when a relational database has been pushed past its write-scaling limits, and the organization demands zero-downtime survival across infrastructure failures.

Situation

In 2022, we were migrating a critical, high-write-volume application onto Apache Cassandra 4.0 (running on OpenJDK 11). The goal was resilience and horizontal scalability across AWS. However, an ad-hoc EC2 provisioning approach by early teams had led to severe operational instability.

We observed asymmetric write latencies across nodes, sporadic gossip drops during network bursts, and uneven disk usage across Availability Zones (AZs). Cassandra’s consistent hashing requires physical infrastructure to align perfectly with its logical awareness of failure domains. When it doesn’t, the database is forced to compensate.

The Problem

Cassandra assumes the operator has correctly mapped physical fault domains to its logical awareness. If an EC2 cluster is deployed blindly into a VPC without mapping AWS AZs to Cassandra racks, the SimpleStrategy replication model (or misconfigured NetworkTopologyStrategy) will distribute replicas arbitrarily.

If AWS us-east-1a suffers a power failure or network partition, and Cassandra unknowingly placed all three replicas of a partition in that single AZ, data becomes unavailable. Furthermore, co-locating the Cassandra commitlog and data directories on a shared OS disk—or worse, a burstable gp2 volume—forces sequential write-ahead logs to compete for IOPS with random-read SSTable fetches and heavy compaction workloads.

How do you design a foundational Cassandra architecture on EC2 that guarantees data survival across a complete AZ failure while preventing storage I/O starvation?

The 3-AZ EC2 Storage Topology

The solution is a strict architectural standard: explicit mapping of 3 AWS Availability Zones to 3 Cassandra logical racks, utilizing the GossipingPropertyFileSnitch, and hard physical separation of storage volumes.

flowchart TD
    Client[Application Client] -->|LOCAL_QUORUM| Coord(Cassandra Coordinator Node)

    subgraph "AWS Region (e.g., us-east-1)"
        subgraph "AZ A (Cassandra Rack: rack1)"
            N1[Node 1 — Seed]
            N2[Node 2]
            N3[Node 3]
        end

        subgraph "AZ B (Cassandra Rack: rack2)"
            N4[Node 4 — Seed]
            N5[Node 5]
            N6[Node 6]
        end

        subgraph "AZ C (Cassandra Rack: rack3)"
            N7[Node 7 — Seed]
            N8[Node 8]
            N9[Node 9]
        end
    end

    Coord -.-> N2
    Coord -.-> N4
    Coord -.-> N8

    %% Storage Detail for Node 1
    N1 --> OS[(EBS gp3 — OS / Logs)]
    N1 --> CL[(EBS gp3 — Commitlog — 3000 IOPS)]
    N1 --> DATA[(EBS gp3 — SSTable Data — 12000 IOPS)]

Storage Separation

Cassandra’s write path is heavily dependent on the sequential speed of the commitlog. We mandated three distinct EBS gp3 volumes per EC2 instance:

  1. OS / Binaries (/dev/sda1): For the operating system and system.log.
  2. Commitlog (/var/lib/cassandra/commitlog): Dedicated volume provisioned for high sequential throughput and 3,000 IOPS.
  3. Data (/var/lib/cassandra/data): Dedicated volume for SSTables, provisioned at 12,000 IOPS and 500 MB/s throughput to absorb compaction spikes and random reads without starving the write path.

In Practice

This architecture relies on strict adherence to the GossipingPropertyFileSnitch. The documented pattern is to define the exact datacenter and rack in cassandra-rackdc.properties:

# cassandra-rackdc.properties on Node 1 (us-east-1a)
dc=us-east-1
rack=rack1

Keyspaces are then created using NetworkTopologyStrategy:

CREATE KEYSPACE production_data 
WITH replication = {'class': 'NetworkTopologyStrategy', 'us-east-1': 3};

Because the replica count (3) matches the number of configured racks (rack1, rack2, rack3), Cassandra guarantees that one replica is placed in each AWS Availability Zone. If us-east-1b (rack2) goes entirely offline, queries executed at LOCAL_QUORUM will still succeed using the replicas in rack1 and rack3.

Seed Node Discipline

A standard pattern configures exactly one seed node per rack (AZ). Placing all seeds in a single AZ is a common failure mode; if that AZ degrades, new nodes cannot bootstrap and join the ring, and existing nodes struggle to reconcile cluster state. Seed nodes must be distributed across the fault domains, but not placed behind an AWS Elastic Load Balancer (ELB), as Cassandra nodes must communicate via direct IP routing.

Where It Breaks

Design ChoiceTradeoffMitigation
EBS gp3 vs. NVMe (i3en/i4i)EBS introduces network dependency for storage. High compaction loads can exhaust EBS bandwidth limits on smaller EC2 instances.Use instances with adequate EBS-optimized bandwidth. Monitor EBS Burst Balance and IOPS utilization heavily. For latency-critical workloads, NVMe instances (i4i) are required, but they trade durability for speed, requiring stricter backup strategies.
Misconfigured SnitchIf an operator uses SimpleStrategy instead of NetworkTopologyStrategy, Cassandra ignores racks and AZs.Enforce keyspace schemas via CI/CD. Monitor nodetool describecluster to ensure AZ mapping is active.
Seed Node OverloadMaking every node a seed node severely disrupts the gossip protocol and increases network chatter.Strictly limit seeds to 1–2 nodes per rack.

Modern Cassandra Context: Cassandra 5.0 on EC2

Cassandra 5.0 (running on JDK 17) introduces the CIDR filtering authorizer (CEP-33), which restricts client access by IP address range using CIDR notation, allowing operators to enforce network-level access boundaries directly at the database layer. Additionally, the shift toward AWS Graviton (c7g/r7g) processors has proven highly cost-effective, though organizations must account for the off-heap memory footprints of the new Storage-Attached Indexing (SAI) when sizing EC2 instances for vector workloads.

What to Do Next

  • Problem: Ad-hoc provisioning leads to unbalanced, fragile clusters that fail during single-AZ outages.
  • Solution: Standardize on NetworkTopologyStrategy, 3-AZ distribution, and strict separation of EBS volumes for the OS, commitlogs, and data.
  • Proof: Distributing seeds across racks and isolating I/O ensures steady LOCAL_QUORUM performance and survival of AZ loss.
  • Action: Audit your cluster using nodetool status and nodetool ring to verify that replicas are evenly distributed across physical hardware fault domains. Validate storage separation with lsblk -t.