Design a Conversational AI Platform with RAG — System Design Interview Practice

Interview prompt

Design an enterprise RAG platform that ingests permissioned documents, retrieves fresh evidence, maintains conversation context, and generates cited answers with safe model fallbacks.

• Separate source documents, chunk and embedding versions, access-control metadata, conversations, prompts, and model outputs.
• Use hybrid dense and sparse retrieval with reranking, bounded context, citations, and tenant-aware filters.
• Make indexing asynchronous and versioned so deletes and permission changes cannot leak stale evidence.
• Explain prompt injection, hallucination, model failure, feedback evaluation, cost budgets, and auditability.

Requirements and scale assumptions

• Upload and parse enterprise documents, chunk and embed them, and publish a searchable version with freshness status.
• Answer a question with authorized retrieved passages, citations, conversation context, safety checks, and streamed output.
• Support document deletion, permission changes, reindexing, feedback, evaluation sets, usage controls, and tenant export.
• Target p95 time to first token below 500 ms and completed grounded responses below 3 seconds for normal queries.
• Scale document indexing and query traffic independently, with per-tenant quotas and bounded retrieval work.
• Never lose a source document or feedback event; make embedding, indexing, generation, and tool retries safe.
• Fall back to keyword search, a smaller model, or an explicit no-answer response when dependencies fail.
• Support 10,000 queries per second, 100 million indexed chunks, and thousands of enterprise tenants.
• Partition by tenant and knowledge collection; isolate large reindex jobs and hot assistants.
• Retain encrypted source and audit data while keeping vector indexes, conversation context, and caches bounded.

• Peak scale: Provide feedback collection — Capacity assumption that drives partitioning and backpressure.
• Latency target: End-to-end response latency under 3 seconds — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Ingest and index enterprise documents; Retrieve relevant context for user queries.
• Async boundary: At-least-once workers — Keep Use chunking strategies for document processing, Implement hybrid retrieval (dense + sparse), Use re-ranking models to improve retrieval quality off the synchronous path.

Key entities

• DocumentVersiondocumentId, version, aclVersion, contentHash, sourceTimestamp, status

  Permissioned canonical document version used for indexing.

• ChunkchunkId, documentVersion, textHash, embeddingVersion, aclVersion, status

  Searchable chunk retaining document and access-policy lineage.

• ConversationTurnconversationId, turnId, question, retrievedChunkIds, modelVersion, citations, status

  Persisted question/answer turn with evidence and model metadata.

• RetrievalPolicypolicyId, tenantId, allowedSources, modelPolicy, retention, version

  Tenant policy controlling source access, model safety, and retention.

Data flow

1. 1. Ingest permissioned enterprise documentsThe ingestion service validates source ownership, ACLs, version, and deletion policy, then stores the canonical document before creating chunk and embedding work.
2. 2. Retrieve evidence for a questionThe retriever applies tenant and document ACLs, combines lexical and vector search, reranks bounded candidates, and records the index generation used.
3. 3. Build a grounded conversation contextThe context builder loads the conversation window, trims or summarizes within a token budget, attaches citations, and rejects evidence that is stale or unauthorized.
4. 4. Generate and validate a cited answerThe model gateway uses a pinned prompt and model policy, checks citation coverage and unsafe output, and falls back to a retrieval-only response when generation is unavailable.
5. 5. Stream the answer and retain the turnThe response gateway streams tokens with a resume cursor, then commits the conversation turn, citations, model version, and feedback without blocking retrieval workers.

Deep dives and trade-offs

• ACL-aware retrieval and deletionFor the enterprise RAG platform, every citation must resolve to an authorized retrieved passage from a versioned source. Design for the failure case where stale permissions, missing evidence, or a model timeout must produce a safe qualified response; keep retries, versions, and repair state explicit. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Context budgets and grounded generationFor the enterprise RAG platform, every citation must resolve to an authorized retrieved passage from a versioned source. Keep this concern off unrelated request paths and partition it by the enterprise RAG platform access key. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Streaming reliability and model fallbacksFor the enterprise RAG platform, every citation must resolve to an authorized retrieved passage from a versioned source. Keep this concern off unrelated request paths and partition it by the enterprise RAG platform access key. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Hybrid retrieval versus one indexUse lexical and vector retrieval with ACL filters and a lexical fallback for exact identifiers and predictable latency. A vector-only design can lose exact matches and makes freshness and authorization harder to reason about.
• Long context versus bounded costUse a bounded conversation window, evidence budget, and deterministic summarization before generation. Unbounded history increases token cost, latency, and the chance of unsupported claims.
• Model quality versus safe fallbackGate generation on citation and safety checks and fall back to cited retrieval results or abstention. Returning a fluent answer without evidence is worse than a transparent refusal for enterprise knowledge.