Designing Data-Intensive Applications
Main Argument
Data systems are built from a small set of foundational ideas, and every design is a tradeoff. The job is not to find the single best technology but to understand the guarantees a system does and does not provide, reason explicitly about reliability, scalability, and maintainability, and pick tools whose tradeoffs fit the problem. This matters most in distributed systems, where the network, clocks, and nodes are all unreliable: things fail partially and unpredictably, so correctness must be reasoned about (consistency models, consensus) rather than assumed.
Key Takeaways
- Three concerns frame every data system: reliability (tolerate faults), scalability (cope with growth), and maintainability (operability, simplicity, evolvability).
- A fault is one component deviating from spec; a failure is the whole system failing to serve users. Build systems that keep faults from escalating into failures.
- Measure performance by percentiles (tail latency), not averages; one slow call can make a whole fan-out slow.
- In distributed systems you often cannot tell a dead node from a slow one; partial failure and unreliable clocks are the defining difficulties.
- Consistency is a spectrum from eventual consistency to linearizability, each a different tradeoff between guarantees and availability/performance.
- Consensus (distributed agreement) underlies leader election, atomic commit, and uniqueness; strong guarantees reduce to it and are therefore expensive.
- Data outlives code, so encodings must support backward and forward compatibility.
- Idempotence makes retries safe and is the practical route to "exactly once."
- Separate the system of record from derived data (caches, indexes, views) that can be rebuilt.