Caching Layer

Overview

A small number of queries account for a large share of total traffic — the same handful of terms get searched constantly. This page treats caching as its own design problem: what to cache, what pattern to use, and what breaks when a cache node fails or a popular query’s cached entry expires all at once. Worth a first-class page rather than a detail buried in Query Serving, since interviewers frequently probe it on its own.

Basic Approach — No Cache, Query the Index Directly

How it works

Every search request goes straight through the broker to the sharded index, every time, even for identical repeated queries.

Client ──▶ Broker ──▶ Scatter-Gather Across Shards

Tradeoffs

• Pro: Zero staleness — every result reflects the current index exactly.
• Con: Popular queries redo the full scatter-gather round trip on every single request, even though the result was identical a second ago — wasted work that a cache would eliminate entirely.
• Con: Read latency is tied directly to scatter-gather latency, with no way to serve a hot, frequently-asked query faster than a rare one.

Scaled Approach — Cache-Aside on the Query Result

How it works

Normalize the query (lowercase, trim, sorted filters) into a cache key. On a request, check Redis first; on a hit, return immediately; on a miss, run the normal scatter-gather path, populate Redis with a TTL, and return.

Client ─▶ Broker ─▶ Redis ──(miss)──▶ Scatter-Gather ─▶ (populate Redis)

GET search:machinelearning        → cache hit, return cached result
SET search:machinelearning EX 60  → populate on miss, 60s TTL

Unlike a typical write-heavy system, there’s usually no explicit invalidation step here — a single re-indexed document could affect results for an unbounded number of cached queries, so trying to invalidate precisely isn’t practical. Instead, lean on the “latency over freshness” priority named up front: a short TTL bounds staleness to an acceptable window without needing to track which cached queries a given document update might affect.

Tradeoffs

• Pro: Removes the vast majority of read load from the index — hot queries are served entirely from memory.
• Pro: Cache-aside degrades gracefully — if Redis is briefly unavailable, requests just fall through to the normal scatter-gather path.
• Con — cache stampede: when a popular query’s entry expires, every concurrent request for it misses at once and all of them hit scatter-gather simultaneously, momentarily spiking load on every shard.
• Con: A document that was just re-indexed won’t be reflected in a cached result for that query until its TTL expires — an explicit, accepted staleness window, not a bug.

Advanced Approach — Stampede Protection, Hot Queries, and Layering

How it works

Three refinements on top of plain cache-aside:

• TTL jitter: randomize each entry’s TTL slightly (e.g., 60s ± 10s) so queries cached around the same time don’t all expire in the same instant.
• Request coalescing (single-flight): when a query misses, only the first request is allowed to run scatter-gather and repopulate the cache; concurrent requests for the same query wait on that in-flight result instead of each independently re-triggering scatter-gather.
• Edge/CDN caching for the most common queries: an extremely popular query (a trending news term) can be cached at a geographically distributed edge layer, serving repeat requests without even reaching the origin broker.

Client ─▶ Edge Cache ─▶ Redis ─▶ Request Coalescing ─▶ Scatter-Gather

Tradeoffs

• Pro: Directly addresses the two failure modes that make caching hard in practice — stampedes and hot-key skew — rather than adding a cache and hoping traffic is evenly distributed.
• Pro: Edge caching removes the very hottest, smallest slice of traffic before it even reaches the broker or any shard.
• Con: Meaningfully more moving parts — coalescing needs an in-flight request map per cache node, and jitter/edge placement both need tuning specific to real traffic patterns.
• Con: Multi-layer caching (edge + Redis) means multiple places a result can go stale, each on its own TTL.

Tech Choices

• Redis — the default shared result cache: fast key lookups, TTLs, and atomic operations if request coalescing is implemented with a lock.
• CDN / edge cache — for the smallest, hottest slice of globally popular queries, serving them geographically close to the requester.
• Local (in-process) cache — an LRU cache inside the broker itself for the very hottest queries, trading a little memory per broker instance for avoiding a network hop to Redis entirely.

How to Vocalize This in an Interview

Don’t introduce Redis as a given — start from “scatter-gather is expensive to redo for the same query every time” to motivate caching at all. Be explicit that search caching skips precise invalidation on purpose (unlike a typical write path), because one document change can touch too many cached queries to track — naming that tradeoff unprompted, tied back to the freshness-vs-latency priority from scoping, is a strong signal of coherent design thinking rather than a bolted-on cache.