A million telemetry writes per second is not a database selection criterion. It is an admission-control, key-distribution, retention, and recovery problem that must be quantified before a product name enters the design.

Situation

Assume ten million registered devices, one million 250-byte measurements per second at steady state, a three-million-per-second short burst, and 90 days of online retention. The raw payload rate is 250 MB/s, or 15 GB/minute, 21.6 TB/day, and 1.94 PB for 90 days. Those figures exclude replication, indexes, storage-engine metadata, compression, backups, and protocol overhead.

The workload also needs a durability contract. This design assumes at-least-once delivery, durable buffering before the database, idempotent writes, and queries scoped to one tenant, device, metric, and bounded time range. Cross-device analytics belongs in an object-store and analytical pipeline, not in the operational serving table.

The Problem

The three-million-write burst cannot be delegated to database autoscaling. DynamoDB on-demand capacity, for example, immediately accommodates up to twice a table’s previous peak; AWS warns that exceeding twice the previous peak inside 30 minutes can throttle unless capacity is pre-warmed or growth is staged. Bigtable, Cassandra, and MongoDB also need time to add capacity or rebalance data.

The design question is: how do we make peak arrival rate independent of database service rate while preserving per-device ordering and preventing a small tenant or key range from consuming the fleet?

The Buffered, Idempotent Ingest Architecture

flowchart TD
    D["Devices — at-least-once delivery"] --> G["Regional gateways — authentication and quotas"]
    G --> Q["Durable log — partition by tenant and device"]
    Q --> N["Normalizer — validate schema and assign event identity"]
    N --> W["Bounded writer pools — adaptive concurrency"]
    W --> O["Operational database — recent device history"]
    Q --> L["Object storage — immutable analytical copy"]
    O --> V["Read API — bounded device and time range"]
    W --> X["Retry topic and dead-letter quarantine"]

Every event carries an immutable event_id, tenant_id, device_id, device timestamp, server-ingest timestamp, and schema version. Consumers checkpoint only after the target acknowledges the idempotent mutation. Retries use exponential backoff with jitter and a bounded attempt budget; poison messages go to quarantine. Queue age, not queue depth alone, is the recovery objective.

Partition the log by a stable hash of tenant and device so events for one device remain ordered. Enforce quotas before the log and again at writer pools. A single customer reconnect storm must not monopolize every database partition.

Physical Models by Engine

EngineNatural modelCritical constraintBest fit
BigtableLexicographically sorted rowsTimestamp-first keys create sequential hotspotsGCP-native time-series serving with row-range reads
DynamoDBPartition key plus sort keyWarm throughput, quotas, and index capacityAWS-native workloads with explicit access patterns
CassandraHash-distributed partitions with clustering rowsRepair, compaction, disk, and partition-size headroomPortability or control justifies an operations team
MongoDBSharded BSON documents and secondary indexesShard-key distribution, index cost, cache, and balancingDocument-oriented operational queries accompany telemetry

For Bigtable, use a key such as tenant#bucket#device#reverse_timestamp, where bucket is a deterministic small shard derived from the device ID. Do not put a timestamp first. Google documents that timestamp-prefixed keys push sequential writes to one node and recommends bounded row-range reads. Bigtable schema guidance, time-series patterns.

For DynamoDB, use one item per event rather than one ever-growing item per device: PK=TENANT#DEVICE#DAY, SK=timestamp#event_id. Select the time bucket from measured events per device and item size. In provisioned mode, one write-capacity unit covers one standard write per second for an item up to 1 KB; transactional writes and affected index projections consume additional capacity. On-demand mode bills write request units instead. Model the base table and every GSI. AWS read and write capacity modes.

For Cassandra, use a bounded key such as PRIMARY KEY ((tenant_id, device_id, day), event_time, event_id) with TimeWindowCompactionStrategy for an immutable TTL workload. TTL expiry creates tombstones; it does not produce zero cleanup cost. Repairs must complete inside the table’s gc_grace_seconds policy before tombstones can safely be purged. Apache Cassandra tombstone documentation.

For MongoDB time-series collections, shard the user-facing collection using a key that includes the metaField; do not operate directly on the internal bucket collection. MongoDB time-series sharding.

Capacity and Cost Model

Do not publish a dollar total without region, date, durability, and discounts. Use a reproducible model:

raw_bytes_day = writes_per_second * payload_bytes * 86400
logical_write_units = writes_per_second * ceil(item_bytes / billing_quantum)
physical_storage = raw_storage * replication_factor * measured_engine_amplification
required_service_rate = peak_rate / target_utilization
recovery_time = queued_events / spare_service_rate

Measure engine amplification; it is not a universal constant. Include base writes, every secondary projection, replication, backups, cross-region traffic, retained queue storage, and a failed-zone capacity scenario.

At three million arrivals per second and 1.5 million sustained database writes per second, a 10-minute burst produces 900 million queued events. With 500,000 writes per second of spare capacity afterward, recovery takes 30 minutes. That consequence belongs in the SLO.

In Practice

The documented pattern across these engines is query-first key design plus bounded buffering. Bigtable starts with read requests and hotspot avoidance. DynamoDB documents warm-throughput behavior rather than unlimited instantaneous elasticity. Cassandra documents that expired TTL data becomes tombstones and depends on repair and compaction. MongoDB requires a valid shard key and balances ranges in the background. These are system behaviors, not benchmark results; this article intentionally makes no latency claim without a workload-specific test.

Where It Breaks

Failure modeObservable proofSafe response
Reconnect storm exceeds warm capacityQueue age and throttling rise togetherEnforce quotas; drain at a tested rate; pre-warm planned events
Hot tenant or key rangePer-key throttles, hot-tablet signals, or shard imbalanceIncrease deterministic buckets after validating read fan-out
Retry amplificationAttempt rate grows faster than original trafficCap concurrency and retries; honor backoff; quarantine poison events
Retention overwhelms serving storageCompaction, cache, or disk headroom degradesExport immutable history and shorten online retention
Duplicate or late eventsIdentity repeats or timestamps arrive out of orderMake writes idempotent and retain device and ingest time
Region lossRemaining region cannot accept peak plus replayTest failover capacity and queue retention

What to Do Next

  • Problem: Peak arrival rate, not average throughput, defines the failure boundary.
  • Solution: Put a durable log and bounded, idempotent writers between devices and the database.
  • Proof: Load-test key distribution, queue recovery, retry behavior, and one-zone loss with production-shaped payloads.
  • Action: Publish assumptions, formulas, test configuration, and measured percentiles before selecting an engine or quoting cost.