Design a Personalized Recommendation Engine — System Design Interview Practice

Interview prompt

Design a personalized recommendation engine that ingests behavior, retrieves candidates, ranks them under business and safety constraints, serves low-latency results, and learns from feedback without reinforcing harmful loops.

• Separate raw behavior events, consent and deletion state, features, candidate indexes, model versions, ranking policy, and impressions.
• Use a multi-stage retrieve/rank architecture with freshness, diversity, eligibility, and exploration controls.
• Log exposures and outcomes so offline evaluation matches what users actually saw and avoids training on unshown candidates.
• Explain cold start, popularity bias, feedback delay, privacy, abuse, model rollback, and cache invalidation.

Requirements and scale assumptions

• Collect consented events, build features, generate candidates, rank them, filter eligibility, and return a cursor of recommendations.
• Support home, similar-item, and session-based recommendations with explanations, impressions, feedback, and experimentation.
• Support deletion propagation, blocked content, frequency caps, model versions, fallback lists, and offline evaluation.
• Target p95 recommendation latency below 100 ms with bounded candidate retrieval and ranking.
• Serve 100,000 requests per second and update features incrementally without one popular item or user becoming a hot key.
• Make event ingestion, impression logging, model publication, and experiment assignment idempotent.
• Fall back to a cached, popular, or editorial list when feature stores, models, or candidate indexes are degraded.
• Serve 100K recommendations per second, 100 million users, and billions of daily behavior and impression events.
• Partition features and candidate indexes by tenant, user, item, region, and time; isolate hot items and launches.
• Retain consented raw events, feature snapshots, model artifacts, impressions, outcomes, and deletion tombstones.

• Peak scale: 100k requests/s; 100M users — Serving and event volume drive cache locality, candidate sharding, and asynchronous feature updates.
• Latency target: p95 recommendation < 100ms; features fresh — The serving budget is separate from offline training and feature-materialization latency.
• Durable boundary: Committed before async — Consented behavior/impression events and approved model artifacts are authoritative; recommendations are derived.
• Async boundary: At-least-once workers — Keep Recommendations AI for pre-built models, BigQuery ML for custom models, Dataflow for feature engineering off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable personalized recommendation engine input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time personalized recommendation engine features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited personalized recommendation engine run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable personalized recommendation engine model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe personalized recommendation engine gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join personalized recommendation engine inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules personalized recommendation engine runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares personalized recommendation engine quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so personalized recommendation engine retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin personalized recommendation engine data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate personalized recommendation engine model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for personalized recommendation engine rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive personalized recommendation engine inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.