Cassandra Data Modeling Reviews: How We Stopped Application Teams Creating Hot Partitions

The most dangerous thing an engineering team can do with Apache Cassandra is design the schema before they know the queries.

In a relational database, you model the entities (e.g., Users, Orders, Items), normalize the tables, and rely on JOINs and secondary indexes to make the queries fast. If you bring that mindset to Cassandra, you will create a distributed systems disaster known as a Hot Partition.

Situation

In 2022, the 27-node cluster began experiencing severe, localized degradation. One specific node (Node A) would hit 85% disk utilization and spike to 100% single-thread CPU usage, while Node B in the same rack sat idle at 30% disk and 5% CPU.

The system.log was spamming WARN: Partition size > 100MB. The background CompactionExecutor thread was hanging for hours trying to merge massive SSTable chunks, causing write backpressure that eventually took the node offline.

The Problem

Cassandra distributes data across nodes using a consistent hashing ring. When you create a table, you define a Primary Key. The first part of that key is the Partition Key.

Cassandra takes the Partition Key, hashes it (using Murmur3Partitioner), and uses that hash to determine exactly which nodes in the cluster own the data. All data for a single partition key is stored together on the same physical nodes.

An application team had created a status_logs table:

-- The Anti-Pattern
CREATE TABLE status_logs (
    tenant_id text,
    event_timestamp timestamp,
    event_data text,
    PRIMARY KEY (tenant_id, event_timestamp)
);

They had one massive enterprise tenant that generated 95% of the system’s traffic. Because tenant_id was the sole partition key, 95% of the writes and reads were hashed to the exact same partition. Cassandra dutifully sent all that traffic to the three nodes responsible for that hash.

The partition grew to 5 GB. Cassandra is designed to handle billions of tiny partitions, not a few monolithic ones. A 5 GB partition breaks JVM memory limits during read scans and crushes the compaction engine. The long-standing operational guidance for a healthy cluster is a maximum of 100 MB per partition and no more than 100,000 cells per partition — these are soft, widely-cited thresholds rather than database-enforced hard limits (Guardrails, discussed below, can turn them into enforced limits).

Data Bucketing Architecture

To fix a hot partition, you must fundamentally change the Primary Key to force the Murmur3Partitioner to distribute the data across the ring.

flowchart TD
    App[Client Writes Data] --> Hash{Murmur3 Hash of — Partition Key}
    
    subgraph "The Anti-Pattern (tenant_id)"
        Hash -->|Hash: 104| Node1[Node 1: 5 GB Partition — CPU 100%]
        Hash -.->|Empty| Node2[Node 2: Idle]
        Hash -.->|Empty| Node3[Node 3: Idle]
    end
    
    subgraph "The Bucketed Pattern ((tenant_id, month), event_id)"
        Hash -->|Hash: A| N1[Node 1: 50 MB]
        Hash -->|Hash: B| N2[Node 2: 50 MB]
        Hash -->|Hash: C| N3[Node 3: 50 MB]
    end

Synthetic Bucketing (Salting)

We enforced a new data modeling rule: any unbounded time-series or high-cardinality collection must use a compound partition key with a time or hash bucket.

We redesigned the table to split the massive tenant’s data into monthly buckets:

-- The Bucketed Pattern
CREATE TABLE status_logs_v2 (
    tenant_id text,
    month text,
    event_timestamp timestamp,
    event_data text,
    PRIMARY KEY ((tenant_id, month), event_timestamp)
);

By wrapping (tenant_id, month) in parentheses, we defined the compound partition key. Now, the tenant’s January data hashes to Node 2, February hashes to Node 15, and March hashes to Node 22. The 5 GB monolithic partition was shattered into 50 MB chunks, evenly distributed across the cluster.

In Practice

Schema migrations in Cassandra are not as simple as an ALTER TABLE statement, because changing a partition key changes where the data physically lives on disk.

Operators must create status_logs_v2, update the application to dual-write to both tables, run an asynchronous backfill script to copy the historical data from V1 to V2, and finally flip the application reads to V2 before dropping V1.

To prevent this from happening again, the runbook establishes a mandatory CI/CD schema linter. The pipeline parsed .cql migration files and automatically rejected PRs that used unbounded collections (SET, LIST, MAP), relied on ALLOW FILTERING, or created time-series tables without explicit time-bucketing in the partition key.

Where It Breaks

Design Choice	Tradeoff	Mitigation
Bucketing Too Finely	Bucketing by the minute instead of month fragments the data into millions of partitions. To read a day’s worth of data, the client must query 1,440 different nodes, causing read latency to skyrocket.	Size buckets so they hold ~50 MB of data. Model against your read queries.
ALLOW FILTERING	Using ALLOW FILTERING forces Cassandra to scan entire nodes, load data into memory, and filter it. It will crash your cluster under load.	Reject any schema or query using ALLOW FILTERING. Build a materialized table tailored to the query.
Unbounded Collections	A Cassandra SET or LIST is stored inside a single row. If it grows indefinitely, you recreate the hot partition problem.	Never use collections for data that grows over time. Use a clustered table.

Modern Cassandra Context: Cassandra 5.0 Guardrails and SAI

Cassandra 4.1 drastically improved an operator’s ability to defend against bad schemas by introducing native Guardrails (partition_size_warn_threshold_in_mb, partition_size_failure_threshold_in_mb) that can reject writes at the coordinator level before they corrupt the storage engine. Furthermore, the introduction of Storage-Attached Indexing (SAI) safely enables multi-column queries without creating the secondary index fanout disasters of the 4.x era, slightly easing the strict denormalization requirements of legacy Cassandra modeling.

What to Do Next

• Problem: Designing Cassandra schemas without understanding the Murmur3Partitioner creates massive, single-node hot partitions that cause cluster instability.
• Solution: Enforce query-first data modeling. Use synthetic bucketing to break unbounded partitions into chunks smaller than 100 MB.
• Proof: Migrating from a single-key partition to a time-bucketed partition evenly distributes disk I/O and CPU, eliminating hot-node crashes.
• Action: Run nodetool tablestats and check the Max partition size metric. If you see partitions exceeding 100 MB, you need to redesign your Primary Key.