Design a Complete CI/CD Pipeline — System Design Interview Practice

Interview prompt

Design a complete container CI/CD platform that builds, tests, scans, signs, and deploys services across environments with approvals, progressive delivery, monitoring, and rollback.

• Use immutable source revisions, signed artifacts, environment policy, and a release pointer as the deployment source of truth.
• Isolate untrusted build workers, cache safe dependencies, and enforce quality, vulnerability, license, and approval gates.
• Deploy with health checks and progressive traffic, then roll back the release pointer rather than rebuilding production code.
• Explain secret handling, provenance, retries, artifact retention, agent loss, and supply-chain compromise.

Requirements and scale assumptions

• Register repositories, pipeline definitions, runners, environments, and protected deployment targets.
• Run build, unit, integration, security, image, and deployment stages with logs, artifacts, approvals, and status.
• Support canary or blue-green rollout, health-based promotion, rollback, cancellation, retention, and audit export.
• Target deployment completion below 10 minutes for a normal service and queue bursts without blocking approvals.
• Scale runners by repository and stage, with per-tenant concurrency limits and isolated deployment workers.
• Make stage retries and artifact publication idempotent; keep an auditable release history and immutable provenance.
• Keep the last known-good version serving when runners, registries, or target clusters are unavailable.
• Enforce code quality gates
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: Enforce code quality gates — Capacity assumption that drives partitioning and backpressure.
• Latency target: Deployment time under 10 minutes — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Automate build and test processes; Deploy to multiple environments.
• Async boundary: At-least-once workers — Keep Azure Pipelines for CI/CD, Container Registry for images, AKS for container orchestration off the synchronous path.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable complete ci cd pipeline source evidence and ingestion cursor.

• SchemaVersiondatasetId, version, compatibility, owner, effectiveAt, status

  Governed complete ci cd pipeline contract used to validate producers and consumers.

• ProcessingRunrunId, inputWatermark, checkpoint, qualityStatus, codeVersion, status

  Checkpointed complete ci cd pipeline processing attempt with quality and lineage metadata.

• AnalyticalDatasetdatasetId, partition, watermark, schemaVersion, qualityStatus, location

  Curated complete ci cd pipeline serving partition with freshness and quality state.

Data flow

1. 1. Register sources and contractsThe complete ci cd pipeline catalog records owners, schemas, compatibility rules, retention, lineage, and partitioning before data is accepted.
2. 2. Ingest with backpressureConnectors checkpoint complete ci cd pipeline source cursors, validate schema and deduplication keys, and slow producers when downstream capacity is exhausted.
3. 3. Process event time with checkpointsStream or batch engines compute complete ci cd pipeline transformations using watermarks, late-data policy, state checkpoints, and deterministic code versions.
4. 4. Publish quality-gated datasetsOnly complete ci cd pipeline outputs that pass completeness, freshness, validity, and privacy checks become visible to analytical consumers.
5. 5. Serve, replay, and reconcileConsumers read bounded partitions with freshness metadata while operators replay failed complete ci cd pipeline ranges and compare output checksums.

Deep dives and trade-offs

• Schema evolution and data qualityVersion complete ci cd pipeline contracts and make compatibility rules explicit for every producer and consumer. Quarantine malformed partitions instead of poisoning the whole dataset. Track row counts, null rates, duplicates, distribution changes, and policy violations by partition.
• Watermarks, late data, and exactly-once effectsUse source cursors and event-time watermarks for complete ci cd pipeline progress, not wall-clock assumptions. Make checkpoints, output keys, and sink commits retry-safe under at-least-once delivery. Document how late events revise windows, aggregates, or snapshots.
• Replay, lineage, and costKeep immutable complete ci cd pipeline raw evidence and code or schema versions so failed outputs can be reproduced. Separate hot serving storage from cold retention and cap replay concurrency. Measure freshness, backlog, compute cost, storage growth, and quality-gate failure rate.
• Streaming versus batchUse streaming for freshness-critical complete ci cd pipeline paths and batch for backfills, compaction, and expensive recomputation. Forcing every workload into streaming makes state, replay, and cost harder to operate.
• Raw retention versus curated-only storageRetain enough immutable raw evidence for replay, audit, and correction, then tier or expire it according to policy. Without raw evidence, a bad transformation can require an unreproducible emergency fix.
• Central warehouse versus domain-owned datasetsCentralize governance and discovery while letting domain owners own contracts and quality signals. A single team owning every transformation becomes a delivery bottleneck and hides data ownership.