Design an API Rate Limiter — System Design Interview Practice

Interview prompt

Design a distributed API rate limiter that enforces tenant, user, credential, IP, and endpoint quotas with predictable burst behavior, low overhead, and safe degradation.

• Define quota identity, policy precedence, token/refill semantics, burst capacity, response headers, time source, and scope inheritance.
• Partition counters by policy key, isolate hot tenants/endpoints, and choose strict versus approximate enforcement for multi-region traffic.
• Keep the decision path bounded and atomic; handle clock skew, retries, failover, counter loss, policy updates, and cache stampedes.
• Explain fail-open/closed choices by endpoint risk, abuse resistance, observability, auditability, and degraded local limits.

Requirements and scale assumptions

• Evaluate requests against hierarchical quotas, return allow/deny/retry-after metadata, and support fixed, sliding, and token-bucket policies.
• Manage plans and overrides, apply endpoint/tenant/user/IP dimensions, expose usage, and support distributed or regional enforcement.
• Make decisions safe under retries, expire counters, audit policy changes, protect trusted traffic, and recover from limiter-store failure.
• Keep limiter decision p99 overhead under 10ms for the normal path and make endpoint-specific degradation explicit.
• Handle 1M decisions per second and hot public endpoints without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 1M decisions/second across 100k tenants and 10k endpoints
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 1M decisions/s; 100k tenants — Capacity assumption that drives partitioning and backpressure.
• Latency target: p99 overhead < 10ms; retry-after accurate — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Versioned quota policies and monotonic counter decisions are authoritative; local caches are bounded accelerators.
• Async boundary: At-least-once workers — Keep Token bucket or leaky bucket algorithm, Redis for distributed counters, Sliding window log or fixed window counter off the synchronous path.

Key entities

• ResourceSpecresourceId, tenantId, desiredState, version, policyVersion, updatedAt

  Versioned desired state for a api rate limiter managed resource.

• OperationoperationId, resourceId, requestHash, step, attempt, status

  Durable api rate limiter reconciliation operation with per-step progress.

• PolicyVersionpolicyId, scope, version, rules, effectiveAt, status

  Auditable api rate limiter policy evaluated before provisioning or mutation.

• ReconciliationCheckpointresourceId, provider, observedVersion, cursor, lastError, updatedAt

  Provider-specific api rate limiter observation and recovery cursor.

Data flow

1. 1. Accept a desired-state commandThe api rate limiter control plane authenticates the tenant, validates policy and quotas, checks the expected version, and records the desired state.
2. 2. Plan a safe operationA planner turns api rate limiter desired state into ordered, bounded steps with dependency checks, blast-radius limits, and rollback metadata.
3. 3. Reconcile providers asynchronouslyWorkers apply api rate limiter operations through provider adapters, persist checkpoints, rate-limit calls, and treat unknown outcomes as observable state.
4. 4. Publish observed healthThe serving projection joins desired and observed api rate limiter state with operation status, policy version, freshness, and actionable errors.
5. 5. Recover and auditRetries, dead letters, drift detection, and operator approvals repair api rate limiter resources without losing the original command or provider evidence.

Deep dives and trade-offs

• Desired versus observed stateKeep api rate limiter desired state separate from provider-observed state and show both to operators. Make every reconciliation step conditional and resumable so a worker crash does not restart unsafe effects. Version policy and resource state so old operations cannot overwrite newer intent.
• Provider failures and unknown outcomesUse provider-specific idempotency tokens and query-after-timeout behavior for api rate limiter operations. Bound retries with exponential backoff, circuit breakers, and per-provider quotas. Route irreconcilable drift to an approval or quarantine path instead of retrying forever.
• Blast radius and operationsPartition api rate limiter work by tenant, region, cluster, or resource class and cap concurrent mutations. Audit who changed desired state, which policy allowed it, and what provider evidence was observed. Alert on drift age, operation backlog, failed steps, policy denials, and stale observations.
• Push versus pull reconciliationUse event triggers for fast response and periodic scans for missed events, drift, and recovery. A push-only api rate limiter controller silently misses changes when a provider event is lost.
• Central control plane versus provider-native controllersKeep policy, intent, and audit centralized while isolating provider-specific application logic behind adapters. A monolithic controller becomes hard to scale and couples unrelated provider failure domains.
• Automatic repair versus approvalAutomate low-risk, reversible api rate limiter changes and require approval for destructive or high-blast-radius operations. Full automation without policy or blast-radius controls can turn a transient signal into a widespread outage.