Questions That Actually Come Up

Name the two-stage pipeline before drawing a single box — build an inverted index from the documents already in the database, serve a ranked query against that index — and say out loud why each stage earns its place: one transforms rows into something queryable in milliseconds, one answers under a tight latency budget. Then rattle off what you're deferring — sharding, caching, indexing backpressure, autocomplete — in one breath, and name 2–3 things explicitly out of scope, like ads or personalized ranking. Doing this unprompted is what separates "I'm scoping deliberately" from "I don't know where to start."

Talk through the arithmetic out loud instead of stating a memorized number. For example: "Say the database holds 1 billion documents and the change stream delivers 4,000 writes/second sustained during a normal day. If each document averages 2KB of postings after compression, the index is on the order of terabytes, which is why it has to be sharded across many machines. Query volume might be 100K QPS at peak, each with a latency budget under 200ms." The specific numbers matter less than showing that write throughput, index size, and query latency are three separate constraints that justify three separate subsystems, not one blob called "the backend."

Start from what a query has to do in each case. Document partitioning (split by doc_id, every shard holds a complete local index) means every query touches every shard, but load stays evenly distributed regardless of which terms are popular. Term partitioning (split by term, every shard owns a slice of the vocabulary) means a rare-term query only touches one shard, but common terms create severe hotspots and merging one term's postings list across shards is awkward. Document partitioning's even load distribution is why it's the standard default — say that explicitly, rather than asserting "sharding" as if it were one undifferentiated choice.

BM25 is a precomputed, formula-based score from term frequency, document length, and term rarity — cheap enough to run against every document in a shard on every query. A learning-to-rank model can fold in signals BM25 can't (click-through rate, page authority, freshness) but is far too expensive to run against millions of candidates. The answer worth landing on is two-phase ranking: BM25 narrows the field to a few hundred candidates, then the learned model re-ranks only that shortlist — naming the cost asymmetry explicitly is what shows you understand why one doesn't replace the other.

This is the sync problem, and it's worth raising unprompted. Writing to the database and the search index in the same request (dual writes) isn't atomic — a crash between the two leaves them silently inconsistent, with no log to replay to catch it. The production answer is Change Data Capture: a tool like Debezium tails the database's own write-ahead log or binlog and emits an event for every insert/update/delete, and an idempotent consumer applies each one to the index via a bulk API. The database stays the single source of truth; the index becomes a derived, replayable projection of it instead of a second write the app has to remember to make.

A full MapReduce-style rebuild over the whole corpus is inherently batch — freshness is capped at "since the last run," which could be hours. The fix is segment-based incremental indexing: write each new batch of documents as its own small immutable segment with its own mini-index, merge small segments into larger ones in the background over time, and have queries search across all live segments and merge results. Combined with a CDC-driven consumer, new or updated documents become searchable within seconds of being written to the database instead of waiting on the next full rebuild — this is the same approach Lucene-based engines use in production.

The broker enforces a per-shard timeout during scatter-gather — if a shard doesn't respond in time, the broker returns results from the shards that did respond, marked with a partial: true flag, rather than blocking the entire request on the slowest shard. This is a deliberate availability-over-completeness tradeoff: a legitimate match on the slow shard is silently missing from that one response, which is an acceptable cost given the alternative — every user waiting on one bad machine.

A hard TTL means every concurrent request for that query misses at the same instant and all of them re-trigger scatter-gather simultaneously — a self-inflicted spike. Two fixes address it directly: TTL jitter, so entries cached around the same time don't all expire in the same instant, and request coalescing (single-flight), where only the first request on a miss actually re-queries the index while concurrent requests for the same query wait on that one in-flight result instead of duplicating the work.

There's an inherent lag baked into the design, and naming it explicitly is the strong answer: CDC has to capture the change from the write-ahead log, the consumer has to process the event into a new (or delete-marked) segment, and any cached query results referencing the old version stay stale until their TTL expires. This is exactly why "latency over freshness" was named as a priority during scoping — the system is deliberately built to tolerate a document being a few seconds to a few minutes out of date rather than trying to guarantee instant propagation end to end.

Split it by pipeline stage: ingestion health (CDC consumer lag behind the database's change stream, Kafka topic backlog, bulk-index error rate), index freshness (time-since-last-segment-merge), and query-path latency (broker p99, per-shard timeout rate, cache hit rate). A rising per-shard timeout rate or a growing CDC consumer lag are both leading indicators of trouble before users notice degraded results — naming those specifically, instead of a generic "I'd monitor uptime," is what shows operational thinking.

Treat it as the same problem as API rate limiting, just pointed inward instead of outward: a token bucket whose refill rate adapts to the cluster's own reported write-queue depth and rejection rate, so the indexer consumer never pushes bulk writes faster than the cluster can actually absorb them. Backing off dynamically — instead of assuming a fixed safe rate — is the detail that matters, since a cluster's real bulk-write capacity varies with what else is happening on it (merges, queries) at any given moment.

Don't answer with a random feature — answer with whatever the design's current weakest assumption is. A strong version names something already flagged in this guide: real personalized ranking that folds in user history without blowing up the cache model, finer-grained rebalancing for uneven shard load, or multi-tenant access control on documents instead of assuming one flat corpus everyone can see. It shows you know your own design's edges rather than treating it as finished.