Requirements & Data Flow

Functional Requirements

Core Requirements

• Users can submit a text query and receive a ranked list of relevant documents from the corpus.
• The index reflects documents already stored in the system's database, kept in sync as documents are added or changed.
• Users can page through a large result set.

Below the line (out of scope)

• Ads & sponsored results — an orthogonal auction and billing system layered on top of ranking.
• Personalized ranking from user history — a separate signals pipeline, not the base relevance problem.
• Image, video, and geospatial search — different content types need different indexes entirely.
• Multi-tenant access control on documents — a separate authorization layer, not the base relevance problem.

Non-Functional Requirements

Before locking in non-functional requirements, it's worth sizing the system out loud, because scale is what decides the sharding and freshness architecture here. Say the corpus holds 100 million documents, receiving updates at a modest but constant rate, and query traffic peaks at 10,000 queries per secondwith a much lower sustained average. Query volume and indexing volume are two separate constraints with very different latency budgets — that asymmetry is why indexing and query serving are built as separate systems, not one blob called "the backend."

Core Requirements

• Low query latency — well under a second, ideally in the low hundreds of milliseconds.
• Highly available — a good-enough ranked result fast beats blocking on one slow shard for a perfect one.
• Index freshness on the order of seconds to minutes is acceptable; true real-time freshness is not required.
• Horizontally scalable index as the corpus grows.
• The database stays the source of truth — the index is a derived, eventually consistent projection of it.

Below the line (out of scope)

• Sub-second (real-time) index freshness for every write.
• Ranking personalized per user.
• Strict linearizable consistency between the database and the index.

Data Flow

1. A document is created or updated in the system's database.
2. A change-data-capture pipeline picks up the change and feeds it to the indexing job.
3. The indexing job tokenizes and analyzes the document, then updates the inverted index.
4. A user submits a search query.
5. The query-serving layer looks up matching postings, ranks them, and returns results.

High-Level Design

With requirements set, sketch the minimal set of components that satisfy them, wired directly to the endpoints above — not a generic "backend" box.

1. Searching — GET /api/search?q=...

Client → Load Balancer → Query Service, which parses the query, fans it out to every relevant Inverted Index shard, merges the returned postings, ranks them, and returns the top results. Query Serving & Ranking →

2. Keeping the Index in Sync — background, no client-facing endpoint

Changes to the source Database are captured by a change-data-capture stream and queued; an Indexing Worker consumes each change, tokenizes the updated document, and writes it into the correct Inverted Index shard — entirely off the query path, so indexing volume never competes with query latency. Indexing & the Inverted Index →