Part III — Derived Data
Batch Processing
Turning large, bounded datasets into derived outputs — the origin of MapReduce and the dataflow engines that followed it.
Batch processing takes a known, bounded set of input files and produces derived output files — a search index, an analytics report, a recommendation table. The defining property is that the input is finite: the job has a clear “done” state, unlike a stream that never ends.
MapReduce
The model that made this pattern practical at scale, popularized by Google and then Hadoop:
Map tasks run fully in parallel with no coordination; only the shuffle step moves data between workers.
# Word count: the canonical MapReduce example
def map(document_id, text):
for word in text.split():
emit(word, 1)
def reduce(word, counts):
emit(word, sum(counts))The elegance of this model is fault tolerance: because map tasks are pure functions of their input, a failed task can simply be re-run on another machine without coordinating with anything else — the framework handles retries transparently, which is essential when you’re running thousands of tasks across thousands of machines and some will inevitably fail.
Beyond MapReduce: dataflow engines
Plain MapReduce forces every multi-step job through disk between each stage, which is wasteful for jobs with several chained transformations. Dataflow engines (Spark, Apache Flink’s batch mode, Tez) generalize the model into an arbitrary directed acyclic graph (DAG) of operators, keeping intermediate data in memory where possible and only materializing to disk when necessary — often 10-100x faster for multi-stage pipelines.
Batch processing’s core limitation is freshness — the output is only ever as current as the last run, whether that’s hourly, nightly, or on-demand. When “how current is this?” becomes the whole point, that’s the problem stream processing solves.