Why distributed databases scale storage effortlessly but hit a brick wall under high write contention.
Modern distributed databases promise infinite scale. Spin up more nodes, spread your partitions across data centers, and your storage capacity expands effortlessly.
While scale-out works wonders for partitioned read traffic, hot write workloads hit a sudden ceiling. Total throughput drops precipitously as concurrent transactions fight over the same keys.
A cross-partition transaction requires a Two-Phase Commit protocol layered on top of consensus groups like Paxos or Raft. Every decision demands multiple network roundtrips across physical machines.
Under Strict Two-Phase Locking, exclusive row locks cannot release until consensus is fully reached across all nodes. Distributed lock holding times skyrocket from microseconds to tens of milliseconds.
Neil Gunther's Universal Scalability Law proves that cross-node coherency and serialization penalties eventually produce retrograde throughput. Adding more nodes makes a contended database slower, not faster.
The maximum throughput for any hot record is strictly bounded: 1 divided by the sum of Network RTT, Consensus Latency, and Execution Time. No amount of cluster hardware can break this speed-of-light limit.
Google Spanner coordinates global ordering using TrueTime, forcing a commit-wait interval to ensure consistency. Hybrid Logical Clocks avoid custom hardware, but introduce restarts when clock drift spikes.
Interactive SQL compounds the crisis. When application servers issue sequential queries over multiple network hops while holding database locks, lock durations multiply by orders of magnitude.
Optimistic Concurrency Control tries to avoid locking by validating at commit time. Under high write contention, however, it triggers catastrophic abort storms and endless transaction retries.
Deterministic systems like Calvin sequence transaction inputs in an ordered consensus log before acquiring locks. By eliminating interactive multi-phase commits, lock hold times shrink drastically.
Engines like TigerBeetle process state transitions deterministically on a single CPU core. By batching operations and avoiding lock contention, a single thread can execute over 500,000 transactions per second.
Architectures like SpacetimeDB run application business logic directly inside database memory space. Eliminating the network boundary between code and data removes distributed coordination bottlenecks entirely.
Ask yourself: Is your workload clean and partitionable across users? Use distributed SQL. Is it dominated by hot rows and shared balances? Choose deterministic single-writer state machines.
Horizontal scale distributes data, but it cannot bend the laws of latency. Real architectural power comes from matching transaction mechanics to the physical realities of the network.
Discover more curated stories