Design a Video Processing and Transcoding Pipeline — System Design Interview Practice

Interview prompt

Design a durable video media pipeline for resumable uploads, validation, transcoding into adaptive-bitrate renditions, thumbnails, moderation hooks, and CDN delivery.

• Define upload sessions, immutable source objects, job graphs, codec profiles, rendition manifests, content ownership, and publication state.
• Make large uploads resumable and processing idempotent; isolate expensive workers, enforce quotas, and prioritize interactive jobs.
• Handle corrupt media, retries, partial outputs, poison jobs, cancellation, moderation/legal holds, retention, and CDN invalidation.
• Explain cost/quality tradeoffs, encryption, access control, observability, replay, and degraded delivery.

Requirements and scale assumptions

• Create resumable uploads, validate media, enqueue a versioned processing graph, generate renditions/thumbnails, and publish a manifest.
• Expose progress, retry/cancel, playback-ready status, signed delivery URLs, adaptive bitrate selection, and content moderation state.
• Support replacement/deletion, legal holds, failed-job quarantine, replay from source objects, and cleanup of unreferenced outputs.
• Acknowledge uploads quickly and make 95% of standard jobs playback-ready within the defined processing SLA.
• Process 1,000 hours of video daily with burst capacity and bounded asynchronous workers, without a single hot key.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 1,000 hours/day; 10x upload bursts; multi-terabyte source retention
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 1,000 hours/day; 10x burst — Capacity assumption that drives partitioning and backpressure.
• Latency target: upload ack < 2s; 95% jobs within SLA — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Immutable source objects and versioned job records are authoritative; renditions and manifests are rebuildable.
• Async boundary: At-least-once workers — Keep Cloud Storage for video storage, Transcoder API for encoding, Cloud Run for custom processing off the synchronous path.

Key entities

• MediaAssetassetId, ownerId, sourceUri, checksum, privacy, status

  Canonical uploaded video processing and transcoding pipeline asset and lifecycle state.

• MediaRenditionassetId, profile, codec, uri, checksum, status

  Derived video processing and transcoding pipeline output identified by a deterministic profile and content hash.

• PlaybackSessionsessionId, assetId, viewerId, entitlementVersion, edgeRegion, expiresAt

  Short-lived video processing and transcoding pipeline access session that binds authorization to delivery.

• ProcessingJobjobId, assetId, operation, attempt, checkpoint, status

  Retry-safe video processing and transcoding pipeline processing job with checkpoints and per-rendition progress.

Data flow

1. 1. Reserve a resumable uploadThe video processing and transcoding pipeline gateway authenticates the owner, reserves metadata, validates size and checksum, and returns a scoped upload URL.
2. 2. Commit and verify the sourceA completion callback verifies the video processing and transcoding pipeline object, records an immutable checksum, and publishes a processing job only once.
3. 3. Process renditions asynchronouslyWorkers execute video processing and transcoding pipeline transforms with deterministic profiles, checkpointing, bounded retries, and a dead-letter path for corrupt inputs.
4. 4. Publish an entitlement-aware manifestA manifest projection exposes only completed video processing and transcoding pipeline renditions and carries policy, checksum, and freshness metadata.
5. 5. Deliver, invalidate, and recoverCDN delivery is protected by expiring URLs and revocation signals; failed video processing and transcoding pipeline jobs and stale manifests are replayable without duplicating outputs.

Deep dives and trade-offs

• Integrity and idempotent processingUse checksums and immutable source objects for video processing and transcoding pipeline deduplication and audit. Derive output keys from asset, profile, and transform version so retries cannot corrupt a completed rendition. Make completion callbacks and worker claims conditional on job version and attempt.
• Authorization at the edgeBind video processing and transcoding pipeline manifests and signed URLs to the viewer, entitlement version, and expiry. Propagate takedown, privacy, and subscription changes to edge caches with bounded revocation delay. Never let a cache hit bypass the policy decision for private or paid content.
• Cost, hot assets, and backpressureSeparate interactive manifest latency from expensive video processing and transcoding pipeline processing and encode work. Use queue priority, concurrency limits, and lifecycle policies for source and rendition storage. Measure cache hit rate, startup latency, processing backlog, failed bytes, and egress cost by profile.
• Process on upload versus on demandPrecompute common video processing and transcoding pipeline profiles and generate rare profiles on demand with a durable job state. Generating every possible profile up front wastes storage and processing budget.
• Origin storage versus CDN cachingKeep the origin authoritative and use CDN caching for immutable or versioned outputs with explicit invalidation. A cache cannot be the only copy of a video processing and transcoding pipeline rendition or the recovery path becomes undefined.
• Quality versus delivery costChoose profiles from device, bandwidth, and business requirements, then measure quality and egress by cohort. Maximal bitrate or resolution can make tail startup and cost unacceptable without improving viewing outcomes.