Design an IoT Platform for 10 Million Devices: Bigtable vs DynamoDB vs Cassandra vs MongoDB
A million telemetry writes per second is not a database selection criterion. It is an admission-control, key-distribution, retention, and recovery problem that must be quantified before a product name enters the design.
Situation
Assume ten million registered devices, one million 250-byte measurements per second at steady state, a three-million-per-second short burst, and 90 days of online retention. The raw payload rate is 250 MB/s, or 15 GB/minute, 21.6 TB/day, and 1.94 PB for 90 days. Those figures exclude replication, indexes, storage-engine metadata, compression, backups, and protocol overhead.
The workload also needs a durability contract. This design assumes at-least-once delivery, durable buffering before the database, idempotent writes, and queries scoped to one tenant, device, metric, and bounded time range. Cross-device analytics belongs in an object-store and analytical pipeline, not in the operational serving table.
The Problem
The three-million-write burst cannot be delegated to database autoscaling. DynamoDB on-demand capacity, for example, immediately accommodates up to twice a table’s previous peak; AWS warns that exceeding twice the previous peak inside 30 minutes can throttle unless capacity is pre-warmed or growth is staged. Bigtable, Cassandra, and MongoDB also need time to add capacity or rebalance data.
The design question is: how do we make peak arrival rate independent of database service rate while preserving per-device ordering and preventing a small tenant or key range from consuming the fleet?
The Buffered, Idempotent Ingest Architecture
flowchart TD
D["Devices — at-least-once delivery"] --> G["Regional gateways — authentication and quotas"]
G --> Q["Durable log — partition by tenant and device"]
Q --> N["Normalizer — validate schema and assign event identity"]
N --> W["Bounded writer pools — adaptive concurrency"]
W --> O["Operational database — recent device history"]
Q --> L["Object storage — immutable analytical copy"]
O --> V["Read API — bounded device and time range"]
W --> X["Retry topic and dead-letter quarantine"]
Every event carries an immutable event_id, tenant_id, device_id, device timestamp, server-ingest timestamp, and schema version. Consumers checkpoint only after the target acknowledges the idempotent mutation. Retries use exponential backoff with jitter and a bounded attempt budget; poison messages go to quarantine. Queue age, not queue depth alone, is the recovery objective.
Partition the log by a stable hash of tenant and device so events for one device remain ordered. Enforce quotas before the log and again at writer pools. A single customer reconnect storm must not monopolize every database partition.
Physical Models by Engine
| Engine | Natural model | Critical constraint | Best fit |
|---|---|---|---|
| Bigtable | Lexicographically sorted rows | Timestamp-first keys create sequential hotspots | GCP-native time-series serving with row-range reads |
| DynamoDB | Partition key plus sort key | Warm throughput, quotas, and index capacity | AWS-native workloads with explicit access patterns |
| Cassandra | Hash-distributed partitions with clustering rows | Repair, compaction, disk, and partition-size headroom | Portability or control justifies an operations team |
| MongoDB | Sharded BSON documents and secondary indexes | Shard-key distribution, index cost, cache, and balancing | Document-oriented operational queries accompany telemetry |
For Bigtable, use a key such as tenant#bucket#device#reverse_timestamp, where bucket is a deterministic small shard derived from the device ID. Do not put a timestamp first. Google documents that timestamp-prefixed keys push sequential writes to one node and recommends bounded row-range reads. Bigtable schema guidance, time-series patterns.
For DynamoDB, use one item per event rather than one ever-growing item per device: PK=TENANT#DEVICE#DAY, SK=timestamp#event_id. Select the time bucket from measured events per device and item size. In provisioned mode, one write-capacity unit covers one standard write per second for an item up to 1 KB; transactional writes and affected index projections consume additional capacity. On-demand mode bills write request units instead. Model the base table and every GSI. AWS read and write capacity modes.
For Cassandra, use a bounded key such as PRIMARY KEY ((tenant_id, device_id, day), event_time, event_id) with TimeWindowCompactionStrategy for an immutable TTL workload. TTL expiry creates tombstones; it does not produce zero cleanup cost. Repairs must complete inside the table’s gc_grace_seconds policy before tombstones can safely be purged. Apache Cassandra tombstone documentation.
For MongoDB time-series collections, shard the user-facing collection using a key that includes the metaField; do not operate directly on the internal bucket collection. MongoDB time-series sharding.
Capacity and Cost Model
Do not publish a dollar total without region, date, durability, and discounts. Use a reproducible model:
raw_bytes_day = writes_per_second * payload_bytes * 86400
logical_write_units = writes_per_second * ceil(item_bytes / billing_quantum)
physical_storage = raw_storage * replication_factor * measured_engine_amplification
required_service_rate = peak_rate / target_utilization
recovery_time = queued_events / spare_service_rate
Measure engine amplification; it is not a universal constant. Include base writes, every secondary projection, replication, backups, cross-region traffic, retained queue storage, and a failed-zone capacity scenario.
At three million arrivals per second and 1.5 million sustained database writes per second, a 10-minute burst produces 900 million queued events. With 500,000 writes per second of spare capacity afterward, recovery takes 30 minutes. That consequence belongs in the SLO.
In Practice
The documented pattern across these engines is query-first key design plus bounded buffering. Bigtable starts with read requests and hotspot avoidance. DynamoDB documents warm-throughput behavior rather than unlimited instantaneous elasticity. Cassandra documents that expired TTL data becomes tombstones and depends on repair and compaction. MongoDB requires a valid shard key and balances ranges in the background. These are system behaviors, not benchmark results; this article intentionally makes no latency claim without a workload-specific test.
Where It Breaks
| Failure mode | Observable proof | Safe response |
|---|---|---|
| Reconnect storm exceeds warm capacity | Queue age and throttling rise together | Enforce quotas; drain at a tested rate; pre-warm planned events |
| Hot tenant or key range | Per-key throttles, hot-tablet signals, or shard imbalance | Increase deterministic buckets after validating read fan-out |
| Retry amplification | Attempt rate grows faster than original traffic | Cap concurrency and retries; honor backoff; quarantine poison events |
| Retention overwhelms serving storage | Compaction, cache, or disk headroom degrades | Export immutable history and shorten online retention |
| Duplicate or late events | Identity repeats or timestamps arrive out of order | Make writes idempotent and retain device and ingest time |
| Region loss | Remaining region cannot accept peak plus replay | Test failover capacity and queue retention |
What to Do Next
- Problem: Peak arrival rate, not average throughput, defines the failure boundary.
- Solution: Put a durable log and bounded, idempotent writers between devices and the database.
- Proof: Load-test key distribution, queue recovery, retry behavior, and one-zone loss with production-shaped payloads.
- Action: Publish assumptions, formulas, test configuration, and measured percentiles before selecting an engine or quoting cost.