Part III — Derived Data
The Future of Data Systems
How the ideas from every earlier topic combine into a broader philosophy: derived data, unbundled databases, and correctness at scale.
Kleppmann closes the book by zooming back out from individual mechanisms (replication, partitioning, consensus, batch, streaming) to a unifying way of thinking about how they all fit together in a real system.
Derived data is not the same as the system of record
This reframes almost every topic in this guide as one thing: batch and stream processing are both just mechanisms for keeping derived data in sync with a system of record, at different latency/complexity tradeoffs.
Unbundling the database
A traditional database bundles many jobs into one process: storage, indexing, query execution, replication, caching. The modern trend Kleppmann highlights is unbundling: using a log (Kafka) as the durable backbone, and letting specialized systems — a search index, a graph database, an analytics warehouse — each subscribe to that log and maintain their own derived view, optimized for their own access pattern. This is the same idea as Change Data Capture from the Stream Processing topic, generalized into an architectural philosophy rather than a point integration.
Every derived system subscribes to the same log instead of being written to directly — each one is provably reconstructable from it, and none of them can drift into being a second, conflicting source of truth.
Correctness without relying on strong guarantees everywhere
Rather than forcing every component to be linearizable (expensive, and sometimes impossible during a network partition), the practical approach is to make operations idempotent wherever possible — safe to retry, safe to reprocess — and to rely on end-to-end application-level checks (like a uniqueness constraint enforced once, at the true point of conflict) rather than assuming every intermediate hop is perfectly consistent.
The throughline
Every topic in this guide is really one long answer to the same question: given that networks drop messages, clocks drift, processes pause unpredictably, and machines fail — how do you still build a system that behaves correctly? Replication and partitioning scale it, transactions and consensus make it correct under concurrency, and batch/stream processing (and the derived-data philosophy this topic closes with) let you build new capabilities on top without ever risking the integrity of the system of record underneath.