A Philosophy of Streaming Systems

Data Integration

Since every tool is designed for a particular usage pattern, complex applications inevitably combine several (e.g., an OLTP database for the system of record plus a specialist search index). As the number of data representations grows, keeping them in sync gets harder.

Fullscreen

Reasoning about dataflows: be explicit about where data is written first and what is derived from what. If CDC from one system of record is the only way to update the index, the index is guaranteed consistent with the source (this is state machine replication, decide a total order, then derive everything deterministically and idempotently).

Derived data vs distributed transactions

Fullscreen

Both keep systems consistent, by different means. In the absence of a good, widely-adopted distributed transaction protocol, log-based derived data is the most promising approach, though read-your-writes guarantees are still valuable, so the chapter works toward a middle ground.

The limits of total ordering

Fullscreen

Most consensus algorithms assume a single node can process the whole event stream and don't let multiple nodes share the ordering work. Capturing causality is the subtle problem (the unfriend-then-complain example: if the unfriend and message-send events lose their ordering, the ex sees the message). Partial fixes: logical timestamps, logging an event ID for "what the user saw," and conflict resolution algorithms (which help with state but not external side effects like notifications).

Batch and stream processing for derived state

Both maintain derived data with a functional flavor (deterministic, immutable inputs, append-only outputs); stream processing adds managed fault-tolerant state. Asynchrony is what makes log-based systems robust, a fault is contained locally, whereas distributed transactions amplify failures by aborting everyone. Reprocessing enables application evolution: rebuild a dataset into a new model rather than being limited to additive schema changes.

Fullscreen

Unifying batch and stream: the lambda architecture (run both in parallel) had problems and fell out of use; the kappa architecture runs both in one system, needing replay of historical events, exactly-once semantics, and event-time (not processing-time) windowing (Apache Beam on Flink/Dataflow).

---

Unbundling Databases

Databases, batch/stream processors, and operating systems all store and process data. Unix offered low-level abstractions (pipes, byte streams); relational databases offered high-level ones (SQL, transactions). The chapter reconciles both. Running CREATE INDEX reprocesses the dataset to derive a new view, just like setting up a follower or bootstrapping CDC. Viewed organization-wide, all dataflow looks like one huge "meta-database," where batch/stream processors are elaborate triggers/materialized-view maintainers and derived systems are different "index types."

Fullscreen

Making unbundling work: synchronizing writes via distributed transactions across heterogeneous systems is problematic; an ordered event log with idempotent consumers is simpler and more feasible. Benefits of log-based integration: loose coupling at the system level (a slow/failed consumer is buffered and catches up; faults are contained) and at the human level (teams develop/deploy independently). Unbundled vs integrated: unbundling won't replace databases (still needed for state and serving queries); running many systems has real operational cost; the goal is breadth, not depth, only worthwhile when no single tool meets all requirements (building for scale you don't need is premature optimization).

Designing applications around dataflow

Like a spreadsheet (VisiCalc, 1979): change a cell and dependent cells recompute automatically. Application code is a derivation function (a secondary index, full-text index, ML model, or cache is data derived by a transformation), but custom derivations are where databases struggle (triggers/stored procedures are an afterthought). So separate application code from state: stateless services plus a database ("separation of Church and state"). The database is a passive mutable variable you can only poll, not subscribe to, dataflow flips this into an active interplay between state changes and code.

Fullscreen

"The fastest and most reliable network request is no network request at all." Composing stream operators resembles microservices but with one-directional asynchronous streams instead of request/response.

Observing derived state

Fullscreen

The write path (eager, like eager evaluation) and read path (lazy) meet at the derived dataset. Indexes, caches, and materialized views simply shift the boundary between them (the social-network timeline celebrity/ordinary split is exactly this). This extends to stateful, offline-capable clients (on-device state is a cache of server state; the screen is a materialized view) and to pushing state changes to clients (Server-Sent Events, WebSockets), extending the write path all the way to the end user, with reconnection handled like consumer offsets. Reads are events too: routing read events through a stream processor makes serving a request a stream-table join (a one-off read is a transient join; a subscription is a persistent one), which also aids provenance tracking. Multishard processing (Storm distributed RPC, fraud scoring across sharded reputation DBs) reuses the streaming infrastructure for cross-shard joins.

---

Aiming for Correctness

Stateful systems remember forever, so errors can last forever. The ACID toolkit is weaker than it seems (weak isolation confusion; Jepsen reveals gaps between claimed and actual safety), and serializability/atomic commit are costly and usually single-datacenter.

The end-to-end argument for databases

Strong safety (even serializable transactions) does not guarantee an application is free from corruption, a buggy app can still write bad data (an argument for immutability). A subtler issue is exactly-once execution:

Fullscreen

A nonidempotent money transfer retried after a lost COMMIT response transfers twice; even 2PC doesn't fix the browser-to-server hop. The fix is a client-generated request ID carried end to end, enforced by a uniqueness constraint (which DBs maintain even at weak isolation). This is the end-to-end argument (Saltzer, Reed, Clark, 1984): a function like duplicate suppression "can completely and correctly be implemented only with the knowledge and help of the application at the endpoints." It also applies to integrity checksums and encryption, low-level features (TCP dedup, Ethernet checksums, WiFi encryption) reduce but don't eliminate the need for end-to-end guarantees.

Enforcing constraints

Fullscreen

Uniqueness (and analogous constraints: non-negative balance, no overselling, no double-booking) requires consensus. In log-based messaging, route conflicting writes to the same shard and process them sequentially, the same shared-log consensus construction from Chapter 10. Multishard request processing (a payment touching source/destination/fees shards) achieves equivalent correctness without atomic commit: atomicity comes from the single atomic append of the request, and at-least-once delivery + determinism + request-ID dedup carry the rest.

Timeliness and integrity

Fullscreen

"Violations of timeliness are allowed under eventual consistency; violations of integrity result in perpetual inconsistency." A credit-card transaction not yet appearing (timeliness) is fine; a balance that doesn't equal the sum of transactions (integrity) is catastrophic. ACID gives both, so the distinction blurs there, but dataflow systems decouple them: no timeliness guarantee unless you explicitly wait, yet integrity is central (exactly-once/idempotence preserves it). Integrity is achieved by: a single atomic message per write (event sourcing), deterministic derivation of all other state, an end-to-end request ID for dedup, and reprocessable immutable messages.

Loosely interpreted constraints and coordination avoidance

Many "hard" constraints are really soft in business terms: overselling stock (apologize, restock, discount), airline/hotel overbooking, bank overdrafts. The fix-it-later mechanism is a compensating transaction (an apology), often cheap. These apps need integrity but not timeliness on the constraint, check optimistically after writing, only coordinating synchronously before truly irreversible actions.

Fullscreen

Coordination-avoiding systems trade a few "apologies for inconsistency" against "apologies for outages", aim for the sweet spot, not zero.

Trust, but verify

Fullscreen

System models are really probabilistic; at scale, unlikely corruption (memory, disk, network, software bugs, even MySQL/PostgreSQL have had constraint/isolation bugs) does happen. So audit: HDFS/S3 continually re-read and compare replicas; test your backups. Design for auditability, event sourcing makes provenance clear, derivations are repeatable (time-travel debugging), and integrity checks are best done end to end. Lighter-weight cryptographic tools (Merkle trees, certificate transparency) may make self-auditing systems more common; blockchains are append-only logs with Byzantine fault tolerance but usually too costly.

---

Real-world examples and analogies

• System of record + derived data, an accountant's ledger and the reports. The ledger (system of record) is the truth; the balance sheet and P&L (search index, cache, warehouse) are derived. Recompute them from the ledger and they always agree.
• Capturing causality, replying to a deleted group chat. You leave a group, then post a complaint elsewhere. If the system forgets that the leave happened before the post, it may still notify the people you left, the ordering dependency was lost.
• Derived data vs distributed transactions, a relay race vs a three-legged race. Log-based derivation is a relay: each runner proceeds independently, and a stumble is local. A distributed transaction is a three-legged race: if one partner falls, everyone stops.
• Reprocessing, converting railway gauges. You don't shut the line for years; you lay a third rail so both gauges run side by side, migrate gradually, then remove the old rail. Old and new derived views coexist the same way.
• Federated vs unbundled, a universal remote vs standardized plugs. Federation is a universal remote that reads from every device through one interface. Unbundling is standardized plugs and wiring so writes flow correctly between independent appliances, harder, but it's what keeps them in sync.
• Dataflow vs microservices, a subscription vs a phone call. Microservices phone the exchange-rate service for every purchase. Dataflow subscribes once and keeps a local copy, so each purchase is answered from memory, faster, and it still works if the other service is down.
• Write path vs read path, meal prep vs cooking to order. The write path is meal prep done in advance (eager); the read path is cooking when the customer orders (lazy). A cache/index decides how much to prep ahead versus cook on demand.
• End-to-end argument, a tracking number on a parcel. The courier's internal scans (TCP, transactions) only dedupe within their own leg. A tracking number the sender writes and the recipient checks (request ID) is what guarantees the parcel isn't processed twice across the whole journey.
• Timeliness vs integrity, a bank statement. A purchase missing for a day (timeliness) is normal. A statement where the balance doesn't equal the sum of transactions (integrity) is a disaster. One self-heals by waiting; the other needs investigation.
• Loosely interpreted constraints, airline overbooking. Selling more seats than exist isn't a crash; it's a business decision with a built-in apology (refund/upgrade). Integrity (no money vanishing) still holds; timeliness on "one person per seat" is relaxed.
• Trust, but verify, checking your smoke detectors. You assume they work, but you still press the test button periodically. HDFS/S3 do the same by continually re-reading data, because at scale "very unlikely" still happens.

---

Cheat-sheet flashcards

Cover the answer and recall it.

• Why must applications combine several tools? → No single tool fits all access patterns.
• What is a system of record? → The authoritative source from which other data is derived.
• How do you keep derived systems consistent? → Derive them from one ordered change log (CDC/event sourcing).
• Derived data vs distributed transactions, core mechanisms? → Deterministic retry + idempotence vs atomic commit.
• Why is log-based derivation preferred over XA? → XA has poor fault tolerance/performance and is single-datacenter.
• What is total order broadcast equivalent to? → Consensus.
• Four limits of total ordering? → Sharded throughput, geo-distribution, microservices, offline clients.
• How can you capture causality without total order? → Logical timestamps, logged event IDs, conflict resolution.
• Why does asynchrony make log-based systems robust? → Faults are contained locally instead of aborting everyone.
• What does reprocessing enable? → Restructuring data into a new model (application evolution).
• Lambda vs kappa architecture? → Run batch+stream in parallel (obsolete) vs in one system.
• Three requirements to unify batch and stream? → Replay, exactly-once, event-time windowing.
• What do databases, OSs, and processors have in common? → They all store and process data.
• What does CREATE INDEX really do? → Reprocesses the dataset to derive a new view.
• Federated database (unifying reads)? → One query interface over many stores (polystore, Trino).
• Unbundled database (unifying writes)? → CDC + event logs synchronizing writes across systems.
• Why is unbundling writes harder than federating reads? → No good cross-system transaction; needs ordered logs + idempotence.
• Two forms of loose coupling from log-based integration? → System-level (buffering/containment) and human-level (independent teams).
• When is unbundling worth it? → Only when no single tool meets all requirements (breadth, not depth).
• What is application code as a derivation function? → Custom code that transforms base data into a derived dataset.
• "Separation of Church and state" means? → Keep stateless app code separate from state in databases.
• Why is a database like a passive variable? → You can poll it but generally not subscribe to changes.
• Dataflow advantage over microservice RPC? → Subscribe + local copy = a stream-table join, no network request.
• Write path vs read path? → Eager precomputation vs lazy on-request reading.
• What do caches/indexes/materialized views do? → Shift the boundary between write-time and read-time work.
• How do you push state changes to clients? → Server-Sent Events / WebSockets, extending the write path to the user.
• "Reads are events too" implies what? → Serving a request is a stream-table join with the data.
• What does the end-to-end argument state? → Some functions are correct only with help from the endpoints.
• Why isn't TCP/transaction dedup enough? → It only covers one connection/transaction, not all hops.
• How do you make a request idempotent end to end? → A client-generated request ID + a uniqueness constraint.
• Two other things the end-to-end argument covers? → Integrity checksums and encryption.
• What do uniqueness constraints require? → Consensus (a single leader or a per-shard sequential processor).
• Why is async multi-leader ruled out for uniqueness? → Leaders could accept conflicting writes concurrently.
• How is multishard atomicity achieved without 2PC? → Atomic append of the request + determinism + request-ID dedup.
• Timeliness vs integrity? → Up-to-dateness (temporary if violated) vs no corruption (permanent if violated).
• Which is usually more important? → Integrity.
• How do dataflow systems treat the two? → They decouple them; integrity is central via exactly-once.
• What is a compensating transaction? → A later correction ("apology") for a violated soft constraint.
• What is a coordination-avoiding data system? → One keeping strong integrity without synchronous coordination.
• What is "trust, but verify"? → Continually auditing data instead of assuming correctness.
• How do HDFS/S3 verify integrity? → Background re-reading and comparing replicas.
• Why does event sourcing aid auditability? → Deterministic, repeatable derivation and clear provenance.
• Cryptographic auditing tools mentioned? → Merkle trees, certificate transparency, blockchains.

---

Common interview questions

1. Why compose multiple data systems, and how do you keep them consistent? (No tool fits all; derive from one ordered log / system of record.)
2. Compare derived-data integration with distributed transactions. (Idempotent retry vs atomic commit; async vs read-your-writes; robustness.)
3. Why can't you always rely on a single total order of events? (Sharded throughput, geo, microservices, offline clients; consensus doesn't scale across nodes.)
4. What does it mean to "unbundle the database"? (Federate reads, unbundle writes via CDC/logs; loose coupling; breadth not depth.)
5. Explain "application code as a derivation function" and the separation of code and state. (Custom transforms; stateless services + DB; passive variable.)
6. How does a dataflow design outperform synchronous microservice RPC? (Subscribe + local copy = stream-table join; no network request; time dependence.)
7. What is the write path vs read path, and how do caches/indexes relate? (Eager vs lazy; they shift the boundary; extend to clients via push.)
8. State the end-to-end argument and apply it to exactly-once. (Endpoints needed; request ID + uniqueness; TCP/txn insufficient.)
9. Why do uniqueness constraints require consensus, and how do logs enforce them? (Conflict decision; per-shard sequential processor; no multi-leader.)
10. How can a multishard operation be atomic without 2PC? (Atomic append + determinism + at-least-once + request-ID dedup.)
11. Distinguish timeliness from integrity; which matters more and why? (Temporary vs permanent; integrity; dataflow decouples them.)
12. What are coordination-avoiding systems and loosely interpreted constraints? (Integrity without coordination; optimistic writes + compensating transactions.)
13. Why "trust, but verify," and how do you design for auditability? (Probabilistic faults; background audits; event sourcing provenance; end-to-end checks.)

---

Key terms glossary

---

Forward links (where these ideas return)

• Ethics, privacy, surveillance, legislation and self-regulation → Chapter 14
• Predictive analytics, feedback loops, and the consequences of derived data → Chapter 14
• Reliability, scalability, maintainability (the framing this chapter fulfills) → Chapter 2
• Event sourcing/CQRS, CDC, log compaction, materialized views → Chapters 3, 12
• Consensus, total order broadcast, linearizability vs serializability → Chapter 10
• Two-phase commit, exactly-once, weak isolation, write skew → Chapter 8