Design an AI-Powered Semantic Search Engine — System Design Interview Practice

Interview prompt

Design a semantic search engine that indexes versioned documents, combines lexical and vector retrieval, reranks results by relevance, enforces access control, and supports fresh updates at large scale.

• Define document/tenant/ACL identity, chunking, embedding/model versions, lexical and vector indexes, freshness, ranking, and deletion semantics.
• Combine keyword and ANN candidates, rerank with bounded work, preserve citations, filter unauthorized documents before display, and handle empty/ambiguous queries.
• Separate ingestion and index builds from serving; publish atomic snapshots, support incremental updates, replay, rollback, and multi-region reads.
• Explain relevance evaluation, poisoning/prompt risks, privacy, hot queries, observability, and lexical fallback when vector search fails.

Requirements and scale assumptions

• Ingest and parse documents, create versioned chunks/embeddings and lexical indexes, and publish searchable snapshots.
• Accept natural-language and keyword queries, retrieve/rerank authorized results, return highlights/citations, filters, scores, and index freshness.
• Support document updates/deletion, ACL changes, synonyms, evaluation sets, backfills, index rollback, and degraded lexical search.
• Meet p95 query latency under 100ms for common searches and expose recall/relevance and freshness metrics.
• Scale to 100M documents and 10k queries per second without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 100M documents, 10k queries/second, and 1k tenants
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 100M documents; 10k queries/s — Capacity assumption that drives partitioning and backpressure.
• Latency target: query p95 < 100ms; ACL-safe results — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Versioned document snapshots and ACLs are authoritative; lexical/vector indexes are rebuildable projections.
• Async boundary: At-least-once workers — Keep Use vector databases (Pinecone, Weaviate, Milvus), Generate embeddings with sentence transformers, Implement ANN (Approximate Nearest Neighbor) search off the synchronous path.

Key entities

• DocumentVersiondocumentId, sourceVersion, contentHash, aclVersion, language, updatedAt

  Canonical ai powered semantic search engine content and access-policy version used for indexing.

• IndexGenerationgenerationId, sourceWatermark, schemaVersion, status, alias, createdAt

  Rebuildable ai powered semantic search engine index generation that can be validated before an atomic alias swap.

• QuerySessionqueryId, tenantId, normalizedQuery, filters, generationId, nextCursor

  Auditable ai powered semantic search engine query context with filters, cursor, and the generation used to answer it.

• RankingFeedbackqueryId, documentId, position, action, modelVersion, occurredAt

  Privacy-scoped ai powered semantic search engine relevance signal for offline evaluation and ranking improvement.

Data flow

1. 1. Accept and authorize source changesThe ai powered semantic search engine ingestion boundary validates content, tenant ownership, ACLs, versions, and idempotency before publishing a document change.
2. 2. Retrieve and rank candidatesThe query service applies authorization filters, retrieves from the active ai powered semantic search engine generation, ranks within the latency budget, and returns generation freshness.
3. 3. Build a safe index generationPartitioned workers transform ai powered semantic search engine documents, checkpoint progress, validate counts and ACL parity, then atomically swap the serving alias.
4. 4. Handle freshness and deletesTombstones and ACL changes propagate through the same pipeline so deleted or newly restricted ai powered semantic search engine content is not left searchable.
5. 5. Measure relevance and recoverFeedback, query traces, lag, and failed partitions drive ai powered semantic search engine ranking evaluation, replay, and bounded degraded behavior.

Deep dives and trade-offs

• ACL correctness and index generationsFilter ai powered semantic search engine results by tenant and effective ACL, or prove the active generation contains the same policy snapshot. Build shadow generations and swap aliases atomically so partial reindexes are never visible. Keep source versions and ACL snapshots for replay when permissions or content change.
• Latency, cursors, and graceful degradationUse bounded candidate retrieval, stable sort keys, and generation-aware cursors for ai powered semantic search engine pagination. Serve the last healthy generation when a new build is incomplete, but expose freshness and avoid silently violating authorization. Protect the query path with timeouts, circuit breakers, and per-tenant quotas.
• Relevance feedback without leakageSeparate ai powered semantic search engine click or conversion signals from personally identifying data and honor retention or deletion requests. Evaluate ranking by query class and tail latency, not only aggregate click-through. Use replayable query sets and staged model or synonym changes before production rollout.
• Synchronous indexing versus queued indexingCommit the source version synchronously and index asynchronously with a visible freshness contract. Waiting for index mutation makes writes fragile and cannot guarantee immediate consistency at scale.
• Denormalized ACL fields versus filter-time checksDenormalize safe, versioned authorization facts when it meets the policy model, while retaining a source-of-truth check for sensitive results. Stale permissions can become a data-leak path if index updates are treated as authoritative.
• Lexical, vector, or hybrid retrievalStart with the retrieval method that matches the corpus and latency budget, then add hybrid ranking behind an experiment and rollback boundary. Adding embeddings without freshness, explainability, or access-control design increases cost without improving user trust.