Stream Processing

A stream is an unbounded, continuously arriving sequence of events — there’s no “last record,” only “the most recent one so far.” Stream processing applies the same map/filter/aggregate ideas as batch processing, but incrementally, as each event arrives, instead of waiting for a scheduled run over a complete dataset.

Message brokers: the backbone of streaming

A log-based message broker (Kafka is the dominant example) is the key infrastructure piece: producers append events to a durable, ordered, append-only log, and any number of consumers can read from it independently, each tracking their own position (offset). Because the log is durable, a consumer can go offline and resume exactly where it left off — this is what makes reliable stream processing possible at all.

# Kafka's core abstraction: a partitioned, ordered, durable log
topic "orders" partition 0: [order_1, order_2, order_3, order_4, ...]
                                                          ^
                                            consumer offset (resumable)

Change Data Capture (CDC)

Instead of writing application code that updates a database and publishes an event (which can fail halfway, leaving them inconsistent), CDC reads a database’s own internal replication log — the same one used for replication — and turns every row-level change into a stream of events automatically. This makes the database the single source of truth, and every downstream system (a cache, a search index, a data warehouse) just consumes the resulting stream.

The hard problems unique to streams

• Out-of-order and late-arriving events — network delays mean events don’t always arrive in the order they occurred; a stream processor computing “orders per minute” has to decide how long to wait for stragglers before finalizing a window’s result.
• Exactly-once semantics — a consumer that crashes after processing an event but before committing its offset will reprocess that event on restart. Achieving effectively-once results (not double-counting) requires either idempotent operations or transactional offset commits tied to the output write.