Master Revision Sheet — Designing Data-Intensive Applications

The shape of the book

The book climbs from a single node (storage and encoding), to multiple nodes (replication, sharding, transactions, the faults that make them hard, and consensus as the fix), to deriving new datasets in bulk (batch and stream), and finally to the human consequences.

Chapter 1: Trade-Offs in Data Systems Architecture

Essence: There is no single right architecture; every choice is a trade-off shaped by your workload and constraints.

Operational (OLTP) vs analytical (OLAP) systems: low-latency reads/writes of individual records vs scanning huge datasets for aggregates. Separated via ETL into a data warehouse; modern variants are the data lake and data lakehouse.
Systems of record (authoritative source) vs derived data (caches, indexes, materialized views, reproducible from the source).
Cloud vs self-hosting: trade control and cost predictability against operational burden and elasticity.
Distributed vs single-node: only go distributed when scale, fault tolerance, or geography demands it, distribution adds large complexity.
Balance business needs against user needs.

Chapter 2: Defining Nonfunctional Requirements

Essence: Reliability, scalability, and maintainability are the goals; define them precisely before building.

Reliability = working correctly despite faults (hardware, software, human). Build fault-tolerant systems; distinguish fault (one component) from failure (whole system). Deliberately inject faults (chaos engineering).
Scalability = coping with growing load. Describe load with the right parameters; describe performance with percentiles (p50/p95/p99), not averages. Beware tail latency and head-of-line blocking.
Maintainability = operability, simplicity, evolvability. Fight accidental complexity with good abstractions.
Case study: social network home timelines, fan-out-on-write vs fan-out-on-read, with a hybrid for celebrities (recurs throughout the book).

Chapter 3: Data Models and Query Languages

Essence: Pick the data model that fits your data's shape and access patterns; the query language follows.

Relational (joins, normalization), document (good locality, weak joins, "schema-on-read"), graph (highly interconnected data). The object-relational impedance mismatch motivated documents.
Normalization removes duplication (write once, join on read); denormalization speeds reads at the cost of consistency.
Query languages: SQL (declarative), Cypher/SPARQL/Datalog (graph), GraphQL (client-shaped responses).
Event sourcing (store immutable events, derive state) and DataFrames (analytical/ML tabular model).

Chapter 4: Storage and Retrieval

Essence: Storage engines optimize for either fast writes (logs) or fast reads (trees); columnar layouts power analytics.

LSM-trees (memtable + SSTables + compaction; used by RocksDB, Cassandra) vs B-trees (in-place, WAL; classic RDBMS).
Write amplification and read amplification are the core trade-offs; Bloom filters skip absent keys.
OLTP = row-oriented; analytics = column-oriented storage (compression, vectorized scans).
Indexes for retrieval: full-text (inverted index) and vector search (embeddings, ANN).

Chapter 5: Encoding and Evolution

Essence: Data outlives code, so encodings must support backward and forward compatibility.

Backward compatibility (new code reads old data) and forward compatibility (old code reads new data) are both required for rolling upgrades.
Formats: JSON/XML (textual, verbose), Protocol Buffers/Thrift (tagged binary, schema with field numbers), Avro (schema-based, no tags, great for evolution and large files).
Dataflow modes: through databases, service calls (REST/RPC), workflow engines, and event-driven / message brokers.

Chapter 6: Replication

Essence: Keep copies in sync for availability, latency, and read scaling, the hard part is conflicts and lag.

Single-leader / multi-leader / leaderless.
Replication lag causes anomalies; mitigations: read-your-writes, monotonic reads, consistent prefix reads.
Conflict handling: LWW (lossy), version vectors (detect concurrency), CRDTs/OT (auto-merge); sync engines for offline/local-first apps.
Leaderless quorums: w + r > n; read repair and anti-entropy.

Chapter 7: Sharding (Partitioning)

Essence: Split data across nodes to scale; the challenge is even distribution and routing.

Key-range sharding (ordered, range scans, risk of hot spots) vs hash sharding (even spread, no range scans).
Hot spots / skew; consistent hashing; rebalancing (fixed number of partitions, dynamic, proportional to nodes).
Request routing (which node has the key) via a coordination service or gossip.
Secondary indexes: local/document-partitioned (scatter-gather reads) vs global/term-partitioned (writes touch multiple shards).

Chapter 8: Transactions

Essence: Transactions collapse messy concurrency and partial failure into commit-or-abort, but isolation levels have sharp edges.

ACID; weak isolation levels and their anomalies: dirty reads/writes, read skew, lost updates, write skew, phantoms.
Snapshot isolation via MVCC; serializability via actual serial execution, two-phase locking (2PL), or serializable snapshot isolation (SSI).
Distributed transactions: two-phase commit (2PC), XA, and exactly-once semantics.

Chapter 9: The Trouble with Distributed Systems

Essence: Networks, clocks, and processes are all unreliable; the defining problem is partial failure and not knowing if an operation succeeded.

Unreliable networks: six request/response failure modes; only tool is a timeout (false-positive vs false-negative trade-off → cascading failure).
Unreliable clocks: time-of-day vs monotonic; LWW timestamp ordering is dangerous; confidence intervals (Spanner TrueTime).
Process pauses (GC, VM suspend, swapping) break leases → split brain, fixed by fencing tokens (monotonic, lower ones rejected).
Byzantine faults (nodes lying); system models (synchronous/partially synchronous/asynchronous; crash-stop/crash-recovery/Byzantine); safety vs liveness.

Chapter 10: Consistency and Consensus

Essence: Strong consistency means behaving like one node (linearizability); achieving it fault-tolerantly is consensus.

Linearizability (recency guarantee) vs serializability (isolation); together = strict serializability.
CAP: during a partition choose consistency or availability; linearizability is also inherently slow.
Logical clocks (Lamport, hybrid logical clocks) order events; linearizable IDs need a single-node oracle or TrueTime.
Consensus = single-leader done right (epoch numbers, overlapping quorums); coordination services (ZooKeeper/etcd: locks, fencing, ephemeral nodes, work allocation).

Chapter 11: Batch Processing

Essence: Transform a bounded, immutable input into output with no side effects, rerunnable, debuggable, throughput-oriented.

A batch framework is a distributed OS: distributed filesystem/object store + orchestrator + tasks.
MapReduce (read/map/sort/reduce, functional roots) → dataflow engines (Spark/Flink: one job, in-memory state, pipelining).
The shuffle is a distributed sort underlying sort-merge joins and grouping.
Use cases: ETL, analytics (lakehouse), ML (feature engineering, BSP/Pregel graphs), serving derived data (push to Kafka or bulk-load, never write directly to prod).

Chapter 12: Stream Processing

Essence: Batch's unbounded counterpart, process events continuously; a database's writes are themselves a stream.

Message brokers: AMQP/JMS (delete on ack, load balancing) vs log-based (Kafka: partitions, offsets, replay, fan-out).
Change data capture (CDC) and event sourcing; log compaction; state is the integral of an event stream.
Time: event time vs processing time, stragglers, window types (tumbling/hopping/sliding/session).
Stream joins (stream-stream window, stream-table enrichment, table-table materialized view); fault tolerance via microbatching/checkpointing/idempotence (exactly-once).

Chapter 13: A Philosophy of Streaming Systems

Essence: Designate a system of record, derive everything else from its ordered log ("unbundle the database"), and separate timeliness from integrity.

Data integration: derive all representations from one ordered change log (state machine replication); log-based beats distributed transactions.
Unbundling the database: federated (unifying reads) vs unbundled (unifying writes via CDC/logs); loose coupling.
Dataflow applications: app code as a derivation function; write path (eager) vs read path (lazy); push state to clients.
Aiming for correctness: the end-to-end argument (request IDs for idempotence); timeliness (temporary) vs integrity (permanent); coordination-avoiding systems; trust, but verify (auditing).

Chapter 14: Doing the Right Thing

Essence: Data systems have human consequences; engineers are responsible for them.

Predictive analytics can build an "algorithmic prison"; models learn and amplify bias (proxies for protected traits).
Feedback loops create self-reinforcing harm; use systems thinking.
Tracking slides into surveillance; consent is often not meaningful; privacy is a decision right, not secrecy.
Data is power and a toxic asset; the fix is data minimization ("data you don't have can't be leaked"), accountability, and a culture shift.

Cross-cutting big ideas (the threads that recur)

There is no single right tool. Every architecture is a trade-off for a workload (Ch 1, 3, 4, 13).
The log is fundamental. Write-ahead logs, replication logs, LSM logs, Kafka, CDC, consensus, event sourcing, all the same append-only idea (Ch 4, 6, 10, 11, 12, 13).
Derived data from a system of record. Indexes, caches, materialized views, ML models are all reproducible derivations kept in sync via an ordered log (Ch 1, 11, 12, 13).
State and an event stream are dual. State is the integral of events; the changelog is its derivative (Ch 12, 13).
Immutability buys safety. Rerunnability, time travel, auditability, human fault tolerance (Ch 11, 12, 13).
Partial failure + nondeterminism are the enemy. Timeouts, fencing tokens, idempotence, and consensus exist to tame them (Ch 9, 10, 12, 13).
Consensus = total order broadcast = a shared log = single-leader done right (Ch 10, 13).
Separate timeliness from integrity. Integrity (no corruption) usually matters more and can be kept without coordination (Ch 9, 13).
End-to-end correctness can't be delegated to lower layers; it needs the endpoints (Ch 13).
The home-timeline case study illustrates the write-path/read-path boundary from Ch 2 all the way to Ch 13.

Rapid-fire flashcards (highest-yield, whole book)

Cover the answer and recall it.

OLTP vs OLAP? → Low-latency record operations vs large-scale aggregate scans.
System of record vs derived data? → Authoritative source vs reproducible copies (indexes/caches/views).
Describe performance with what, not averages? → Percentiles (p95/p99); watch tail latency.
Three pillars of a good system? → Reliability, scalability, maintainability.
Fault vs failure? → One component misbehaving vs the whole system stopping.
Document vs relational strength? → Locality/flexible schema vs joins/normalization.
LSM-tree vs B-tree? → Log-structured fast writes vs in-place fast reads.
What storage layout powers analytics? → Column-oriented.
Backward vs forward compatibility? → New code reads old data vs old code reads new data.
Best format for schema evolution on big files? → Avro.
Three replication topologies? → Single-leader, multi-leader, leaderless.
Three replication-lag guarantees? → Read-your-writes, monotonic reads, consistent prefix.
Leaderless quorum condition? → w + r > n.
Detect concurrent writes with? → Version vectors.
Two sharding strategies? → Key-range and hash.
Local vs global secondary index trade-off? → Scatter-gather reads vs multi-shard writes.
Five transaction anomalies? → Dirty reads/writes, read skew, lost update, write skew, phantoms.
Snapshot isolation is implemented with? → MVCC.
Three ways to get serializability? → Serial execution, 2PL, SSI.
What makes distributed systems hard? → Partial failure; not knowing if an op succeeded.
Only tool to detect a dead node? → A timeout.
Fix for split brain after a pause? → Fencing tokens (monotonic; lower ones rejected).
Safety vs liveness? → "Nothing bad happens" (always) vs "something good eventually happens."
Linearizability vs serializability? → Single-object recency vs multi-object some-serial-order.
CAP during a partition? → Choose consistency or availability.
Lamport timestamp? → (counter, node-ID); causal total order, not linearizable.
Four equivalent faces of consensus? → Single-value/CAS, shared log, fetch-and-add, atomic commit.
What is consensus, in one line? → Single-leader replication done right (epoch numbers, overlapping quorums).
A coordination service provides? → Locks/leases, fencing, ephemeral nodes, work allocation (ZooKeeper/etcd).
Defining trait of batch processing? → Bounded, immutable input; no side effects; throughput-oriented.
What is a shuffle? → A distributed sort underlying joins/aggregations.
MapReduce vs dataflow engines? → Rigid map/reduce + disk I/O vs flexible operators + in-memory pipelining.
Defining trait of stream processing? → Unbounded input processed continuously.
Log-based vs AMQP/JMS broker? → Retained replayable log + offsets vs delete-on-ack per-message.
What is CDC? → Capturing a database's changes as an ordered stream.
State and event stream relationship? → State = integral of events; changelog = derivative.
Event time vs processing time? → When it happened vs when it was processed.
Three stream joins? → Stream-stream (window), stream-table (enrichment), table-table (materialized view).
Exactly-once techniques? → Microbatching, checkpointing, atomic commit, idempotence.
Unbundling: reads vs writes? → Federated (unify reads) vs unbundled via logs (unify writes).
Write path vs read path? → Eager precomputation vs lazy on-request reading.
End-to-end argument? → Some functions (e.g., dedup) need help from the endpoints; use request IDs.
Timeliness vs integrity? → Up-to-dateness (temporary if violated) vs no corruption (permanent if violated).
Coordination-avoiding system? → Strong integrity without synchronous coordination.
Privacy defined? → A decision right over what to reveal to whom, not secrecy.
Core ethical safeguard for data? → Data minimization (collect/retain only what's necessary).

One-line essence per chapter (the 60-second review)

No single right architecture, everything is a workload-driven trade-off.
Define reliability, scalability (percentiles), and maintainability precisely.
Match the data model (relational/document/graph) to data shape and access.
Storage engines trade write vs read amplification; columnar for analytics.
Data outlives code, so encodings need backward/forward compatibility.
Replicate for availability and latency; the hard parts are lag and conflicts.
Shard for scale; balance load and route requests; mind secondary indexes.
Transactions collapse concurrency/failure into commit-or-abort; isolation has edges.
Networks, clocks, and processes are unreliable; partial failure is the enemy.
Linearizability is the consistency model; consensus is how you achieve it.
Batch: bounded immutable input to output; the shuffle and dataflow engines.
Stream: unbounded events; logs, CDC, windows, joins, exactly-once.
Unbundle the database; derive from a log; separate timeliness from integrity.
Data has human consequences; minimize it and take responsibility.