When a pager goes off at 3:00 AM for “High Cassandra Latency,” the average on-call engineer starts guessing. They restart nodes. They blame the network. They blame the cloud provider.

Guessing during a database incident destroys page caches, compounds latency, and prolongs the outage. Troubleshooting a distributed system requires a deterministic hierarchy of elimination.

Situation

In 2023, the 36-node Cassandra 4.1 cluster began experiencing periodic p99 latency spikes of up to 1,500ms. Client applications were throwing ReadTimeoutException and WriteTimeoutException, causing cross-AZ traffic to drop.

Engineers responding to the incident were overwhelmed by the number of metrics available. They saw CPU spikes on some nodes, disk I/O spikes on others, and thousands of dropped messages in JMX. Without a structured way to interpret the data, they considered restarting the cluster—a catastrophic mistake that would have wiped out the OS page cache and triggered a massive read-from-disk storm.

The Problem

Cassandra is a distributed system where a single query touches multiple servers. When a client application executes a query, it connects to a Coordinator node. That coordinator then forwards the request to the Replica nodes that actually hold the data.

If a client experiences a timeout, the root cause could be:

  1. The network link between the client and the coordinator.
  2. The coordinator’s thread pools queueing the request.
  3. The network link between the coordinator and the replicas.
  4. A 2-second JVM Garbage Collection (GC) pause on a replica.
  5. A bad CQL query scanning 10,000 tombstones on disk.
  6. A saturated EBS volume blocking the read.

You cannot find the root cause by looking at a dashboard. You must isolate the bottleneck by following the query path.

The 7-Step Troubleshooting Hierarchy

We developed a strict triage playbook that forces engineers to evaluate the system from the outside in: Client → Coordinator → ThreadPools → Replicas → JVM → Disk.

flowchart TD
    Incident[High Latency Alert] --> Step1[1. proxyhistograms: Coordinator Latency]
    Step1 --> Step2[2. tablehistograms: Replica Latency]
    
    Step2 --> Step3{Are both high?}
    
    Step3 -->|Yes| Step4[3. tpstats: Check Dropped Messages & Thread Pools]
    Step3 -->|Replica Only| Step5[4. Check JVM & Disk on Replica]
    Step3 -->|Coordinator Only| Step6[Check Network / Speculative Retry]
    
    Step4 --> GC[5. gcstats: Check Stop-the-World Pauses]
    Step4 --> CQL[6. tablestats: Tombstones & SSTables per Read]
    Step4 --> IO[7. iostat: Storage EBS Latency]

1. Coordinator vs. Local Replica Latency

The first step is determining if the database is slow, or if the network between the database nodes is slow. Run nodetool proxyhistograms. This shows the latency the coordinator experiences while waiting for replicas. Then run nodetool tablehistograms <keyspace> <table>. This shows the latency of the local replica fetching data from its own disk.

  • If proxyhistograms is high but tablehistograms is low, the nodes are fast but network/gossip is degraded, or a single slow replica is poisoning the coordinator.
  • If both are high, the nodes themselves are physically struggling to read data.

2. Thread Pools and Dropped Messages

If the nodes are struggling, find out where the queue is backing up. Run nodetool tpstats. Look at MutationStage and ReadStage. If PendingTasks is in the thousands, the node is saturated. If you see high DroppedMessages, Cassandra is executing backpressure—it is purposefully dropping client requests because it cannot physically process them fast enough. This is usually not a network issue; it’s a compute or disk issue.

3. Tombstones and Read Amplification

Why is the read stage saturated? The most common cause is a bad CQL query. Look at nodetool tablestats <keyspace>. If SSTables in each read is consistently above the low single digits (commonly cited as a >3 warning sign, though the right threshold depends on your compaction strategy and read pattern), you are suffering from read amplification (likely due to compaction falling behind). If the query is hitting thousands of tombstones, the database is spending all its CPU iterating over deleted data.

4. GC Pauses and Disk I/O

If the queries are healthy but the node is locking up, run nodetool gcstats. If you see GC pauses exceeding 500ms, the JVM is freezing. If GC is healthy, run iostat -xz 1 5 to check the underlying Linux disk. If %util is 100% and await is high, your EBS volume is saturated.

In Practice

During incidents, the architecture requires engineers to capture this data immediately using a CLI triage script before taking any destructive action.

In the 1,500ms latency incident, proxyhistograms were high, but tablehistograms were normal. nodetool tpstats showed zero dropped messages on the replicas. The issue wasn’t the database disks or queries—it was a network degradation in a single AWS Availability Zone causing the coordinator to wait for TCP timeouts. Note that cassandra-rackdc.properties rack/DC assignment is only read at process startup — changing it requires a rolling restart, it isn’t a hot-applied fix. The actual mitigation was stopping the Cassandra process on the degraded AZ’s nodes: once the phi-accrual failure detector marked them DOWN in gossip, other coordinators stopped routing requests to them within seconds, and latencies recovered well before a topology change or restart would have completed.

Where It Breaks

Design ChoiceTradeoffMitigation
Restarting Nodes PrematurelyRestarting a node destroys the Linux page cache. When the node comes back, all reads must hit physical disk, causing a massive latency spike that masks the original issue.Never restart a node until you have captured tpstats and gcstats.
Misinterpreting Dropped MessagesEngineers often assume DroppedMessages means packet loss. It actually means the JVM intentionally discarded the request to prevent OOM.Treat dropped messages as a symptom of disk or CPU exhaustion, not a networking switch failure.
Dynamic RollbacksMaking permanent cassandra.yaml changes during an outage requires restarts.Use JMX to apply dynamic rollbacks: nodetool settimeout or nodetool setcompactionthroughput take effect instantly without restarting.

Modern Cassandra Context: Cassandra 5.0 Observability

Cassandra 5.0 dramatically modernizes this playbook by exposing internal states via virtual tables. Instead of relying entirely on shell scripts and JMX polling, operators can now query system_views.queries and system_views.thread_pools directly via CQL. Combined with improved slow query logging, this allows centralized Prometheus exporters to build real-time views of coordinator versus replica latency without requiring SSH access to the instances.

What to Do Next

  • Problem: Guessing during a database incident leads to destructive actions (like restarts) that compound latency and destroy caches.
  • Solution: Follow a deterministic triage hierarchy: isolate coordinator vs. replica latency, check thread pool queues, identify bad query shapes, and finally inspect JVM/Disk utilization.
  • Proof: Isolating the difference between proxyhistograms and tablehistograms prevents teams from debugging storage volumes when the actual problem is network topology.
  • Action: Next time you have a latency spike, do not restart the node. SSH in and run nodetool tpstats. Look at the ReadStage and MutationStage pending tasks.