Indexing Backpressure & Query Rate Limiting

Overview

This subsystem is really two rate limiters wearing one hat: the indexing consumer has to throttle itself against the search cluster’s own write capacity, so a burst of database changes (a bulk import, a backfill, a CDC replay) never floods it — and the public query API has to throttle clients, so one abusive caller can’t degrade search for everyone else. Different direction, same underlying algorithm. It’s a small subsystem next to the others, but a frequent interview add-on because it tests distributed state and algorithmic tradeoffs, not just “add a component.”

API Design

Indexing backpressure isn’t a client-facing API — it’s an internal constraint the indexer consumer respects, driven by the search cluster’s own bulk-write capacity. Elasticsearch signals this itself: a bulk request that arrives faster than the cluster can absorb it gets rejected rather than silently queued forever:

// 429-equivalent from the _bulk endpoint when the write queue is full
{ "type": "es_rejected_execution_exception", "reason": "rejected execution... queue capacity 200" }

Query API rate limiting shows up as middleware wrapping the search endpoint — visible in response headers, and as a distinct error once a client exceeds its limit:

GET /api/search?q=...

// response headers
X-RateLimit-Limit: 120
X-RateLimit-Remaining: 113
X-RateLimit-Reset: 1723459200

// 429 Too Many Requests
{ "error": "rate_limited", "retry_after_seconds": 8 }

Returning limiter state on every response (not just the 429) lets well-behaved clients back off before they’re ever rejected — worth mentioning explicitly, since it’s the detail that separates “I added a rate limiter” from “I designed a rate-limited API.”

Database Schema

No durable table for either side — both are short-lived state in Redis, keyed by the thing being throttled:

index_backpressure:{cluster}      → Hash { tokens, last_refill_ts, min_delay_ms }
ratelimit:{client_id}:{endpoint}  → Hash { tokens, last_refill_ts }

A Lua script does the read-refill-decrement-write as one atomic step for both — without atomicity, two concurrent indexer-consumer partitions (or two concurrent API requests from the same client) could both read tokens: 1, both decide they’re allowed through, and the limit gets bypassed under exactly the concurrent load it’s supposed to protect against.

Basic Approach — Fixed Delay / Fixed Window Counter

How it works

Indexer side: sleep a fixed number of milliseconds between bulk-index batches. API side: a counter per client that increments per request and resets every fixed window (e.g., once per minute).

Indexer ──▶ sleep(fixed_delay) ──▶ Bulk-index batch
Client  ──▶ API ──▶ In-Memory Counter (resets every window)

Tradeoffs

• Pro: Extremely simple on both sides — a sleep call and a counter.
• Con — boundary burst: a client can send the full limit right at the end of one window and the full limit again right at the start of the next, doubling the effective rate for a short burst.
• Con: An in-memory counter (or per-worker delay state) doesn’t work once either the indexer consumer or the API is scaled across multiple machines — each one tracks its own state independently, so the real, aggregate rate isn’t actually bounded.

Scaled Approach — Centralized State, Sliding Window

How it works

Move both counters into Redis so every indexer-consumer partition and every API server (for a given client) shares the same state via an atomic increment-with-TTL operation. Replace the fixed window with a sliding window that smooths out the boundary-burst problem by considering a rolling time range instead of discrete resettable buckets.

Indexer Partitions ──▶ Redis (shared bulk-write throttle)
API Servers        ──▶ Redis (shared per-client sliding window)

Tradeoffs

• Pro: All indexer partitions now respect one true bulk-write budget, and all API servers enforce one true per-client limit — no more independent, under-counting local state.
• Pro: Sliding windows remove the fixed-window boundary-burst vulnerability on the API side.
• Con: Redis is now on the critical path of every bulk-index batch and every API request — its latency and availability directly affect both.
• Con: A sliding window log (storing every request timestamp) has storage cost proportional to request volume; the counter-based variant trades some precision to avoid that cost.

Advanced Approach — Token Bucket, Tiered by Endpoint and Cluster Load

How it works

Use a token bucket on both sides: a bucket refills at a steady rate and is drained per batch/request, implemented atomically in Redis via a Lua script. This tolerates short bursts up to the bucket size while still enforcing a long-run average — a better model of real usage than a hard window. Tier the limits: the indexer’s bulk-write budget adapts dynamically to the cluster’s own reported queue depth and rejection rate rather than a fixed constant, and API limits differ by endpoint tier (a plain search query gets a much looser limit than an expensive, resource-heavy query type).

Indexer ──▶ Token Bucket (Redis, refill rate adapts to cluster queue depth)
Client  ──▶ Edge Rate Limiter ──▶ Token Bucket (Redis, per client + endpoint tier) ──▶ API

Tradeoffs

• Pro: Token bucket tolerates natural burstiness on both sides without being unfairly strict, while still respecting the cluster’s real write capacity.
• Pro: Enforcing API limits at the edge protects backend/index capacity from ever seeing abusive load in the first place.
• Con: More moving pieces — adaptive bulk-write throttling for the indexer and per-endpoint-tier configuration for the API both need to be kept in sync and monitored.
• Con: Getting per-endpoint API tiers wrong is as much a product decision as a technical one — too strict frustrates real users, too loose under-protects the system.

Tech Choices

• Redis — atomic counters and token buckets, shared across all indexer consumer partitions and all API servers respectively.
• Lua scripting in Redis — makes the check-and-decrement of a token bucket atomic, avoiding race conditions under concurrent load.
• Bulk API backpressure signals — Elasticsearch’s own es_rejected_execution_exception (or a queue-depth metric) is the source of truth for how fast the indexer is allowed to write, the same role robots.txt’s Crawl-delay plays in a crawler-based design.
• API Gateway (e.g., Envoy, Kong) — enforces client-facing limits at the edge before requests reach application servers.

How to Vocalize This in an Interview

Frame it as one algorithm serving two directions — that framing itself is a signal you’re not just reciting “add rate limiting” as a checklist item. Walk through the fixed-window boundary-burst flaw with actual numbers (“100 requests at 0:59, 100 more at 1:00”) since that’s the detail that proves you understand why sliding window and token bucket exist, and connect the indexing-backpressure side back to a concrete failure mode: a large backfill or CDC replay that, unthrottled, saturates the search cluster’s write queue and degrades query latency for every live user at the same time.