Query Serving & Ranking

Overview

This is the read, and the hard problem — the search-engine equivalent of Twitter’s timeline read. Indexing can lag by minutes without anyone noticing; a search query cannot. Everything here is about answering “which documents match, ranked how” against an index that may be spread across hundreds of machines, inside a latency budget measured in tens of milliseconds.

API Design

GET /api/search?q=machine+learning&cursor=

// 200 OK
{
  "items": [
    { "doc_id": 41, "url": "https://example.com/ml-intro", "title": "Intro to ML", "score": 8.7 },
    { "doc_id": 7,  "url": "https://example.com/ml-course", "title": "ML Course", "score": 6.2 }
  ],
  "next_cursor": "eyJwYWdlIjoyfQ",
  "partial": false
}

The partial field is the detail worth calling out unprompted: it’s true when one or more index shards didn’t respond in time and the result is a best-effort answer from the rest — an explicit signal that the read path chose to degrade gracefully rather than block the whole request on the slowest shard.

Database Schema

Query serving doesn’t own a schema of its own — it reads the inverted index built by Indexing and, optionally, writes to a query-result cache:

search_cache:{normalized_query}  → cached response, short TTL

Normalizing the query (lowercase, trim, stable ordering of filters) before using it as a cache key is what makes repeated identical-intent queries actually hit the cache — worth naming explicitly, since a naive raw-string key would miss on trivial variations like extra whitespace.

Basic Approach — Linear Scan, Naive Scoring

How it works

For a query, scan every document, check whether its text contains the query terms, and rank matches by raw term frequency.

Query ──▶ Scan All Documents ──▶ Filter Matches ──▶ Sort by Term Frequency

Tradeoffs

• Pro: No index infrastructure required at all — works on day one.
• Con: A full scan over a large corpus per query is many orders of magnitude too slow to be usable.
• Con: Raw term frequency is a poor relevance signal on its own — a document that repeats “the” a thousand times would outrank a genuinely relevant one.

Scaled Approach — Single-Node Inverted Index Lookup with BM25

How it works

Look up each query term’s postings list directly in the inverted index (no scanning), intersect/union the lists depending on query semantics (AND vs. OR), and rank the result using BM25 — a term-frequency scoring function that accounts for document length and term rarity across the corpus, instead of raw counts.

Query ──▶ Tokenize ──▶ Postings Lookup (per term) ──▶ Merge ──▶ BM25 Rank ──▶ Top-K

Tradeoffs

• Pro: Query latency now depends on postings-list size, not total corpus size — the entire point of building the index in the first place.
• Pro: BM25 is a well-understood, tunable default that already accounts for term rarity (a match on a rare word counts for more than a match on a common one) without any machine learning involved.
• Con: Still a single machine — once the index no longer fits on one node, or query volume exceeds what one node can serve, this stops working.
• Con: BM25 alone ignores signals beyond the text itself — page authority, freshness, and user engagement all go unused.

Advanced Approach — Scatter-Gather Across Shards, Two-Phase Ranking

How it works

A broker node receives the query and fans it out to every index shard in parallel (scatter). Each shard independently ranks its own local top-K matches with BM25 and returns them; the broker merges all shards’ results into a single global top-K (gather). To keep tail latency bounded, the broker enforces a timeout per shard and returns a partial: true result using whichever shards responded in time rather than waiting on the slowest one.

A second refinement — two-phase ranking — runs BM25 as a cheap first pass to narrow millions of candidates down to a few hundred, then re-ranks only that shortlist with a more expensive learning-to-rank model that folds in signals BM25 can’t (click-through rate, page authority, freshness), since that model would be far too slow to run against the full corpus.

Client ──▶ Broker ──(scatter)──▶ [Shard 1 | Shard 2 | ... | Shard N]
                    ◀─(gather, per-shard timeout)──┘
Broker ──▶ Top ~500 candidates ──▶ Learning-to-Rank re-rank ──▶ Top-K to client

Tradeoffs

• Pro: Scatter-gather is what actually lets the index scale past one machine while a single query still sees the whole corpus.
• Pro: Two-phase ranking gets the benefit of an expensive, high-quality model without paying its cost against every document in the corpus.
• Con: Per-shard timeouts trade completeness for latency — a legitimate match on a slow or momentarily unavailable shard can simply be missing from results, silently.
• Con: The broker is a new single point of coordination — it needs its own scaling and failure story (typically a stateless, horizontally scaled layer, but worth naming as a new component rather than assuming it away).

Tech Choices

• BM25 — the default first-pass relevance function; cheap, well understood, no training required.
• Scatter-gather (broker/leaf) — the standard pattern (used by Elasticsearch and Lucene-based systems) for querying a sharded index as if it were one index.
• Learning-to-rank model — a second-phase re-ranker over a small candidate set, where richer signals become affordable.
• Redis — caches full query responses and/or hot postings lists to avoid repeating the scatter-gather round trip for popular queries.

How to Vocalize This in an Interview

Start from “look it up in the index, don’t scan” to motivate the inverted index dependency, then let scale force scatter-gather the same way write volume forced sharding elsewhere — say explicitly that one query now touches many machines, not one. Bring up per-shard timeouts and partial results only once “what if a shard is slow” comes up, since naming that tradeoff unprompted is a strong signal you understand the latency/completeness tension, not just the happy path.