Build a Serverless Event-Driven Architecture — System Design Interview Practice

Interview prompt

Design a serverless event-driven platform that accepts versioned events, routes them to independent consumers, supports retries and ordering where needed, and remains observable under bursty workloads.

• Define event envelopes, schemas, ownership, delivery guarantees, ordering keys, replay, consumer checkpoints, and idempotent side effects.
• Separate ingress, routing, queues, workers, and durable state; isolate noisy tenants and apply backpressure, quotas, and dead-letter policies.
• Explain fan-out, deduplication, poison events, schema evolution, partial consumer failure, and replay without duplicating business effects.
• Cover authorization, encryption, auditability, tracing, cost controls, and degraded operation when workers or downstream systems fail.

Requirements and scale assumptions

• Publish validated events, route them by type/tenant/key, invoke independent consumers, and expose delivery, retry, and processing status.
• Support ordered partitions where required, fan-out subscriptions, scheduled events, dead-letter inspection, replay, and consumer versioning.
• Make side effects idempotent, preserve event lineage, enforce producer/consumer authorization, and recover after queue or worker outages.
• Deliver ordinary events to healthy consumers with p95 under one second while exposing lag and freshness explicitly.
• Scale to 1M events per second and thousands of consumers without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 1M events/second peak, 10k consumers, and 1,000 tenants
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 1M events/s; 10k consumers — Capacity assumption that drives partitioning and backpressure.
• Latency target: delivery p95 < 1s; lag observable — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Accepted events and consumer checkpoints are authoritative; handler outcomes and projections are derived.
• Async boundary: At-least-once workers — Keep Cloud Functions for event handlers, Eventarc for event routing, Pub/Sub for messaging off the synchronous path.

Key entities

• ResourceSpecresourceId, tenantId, desiredState, version, policyVersion, updatedAt

  Versioned desired state for a serverless event driven architecture managed resource.

• OperationoperationId, resourceId, requestHash, step, attempt, status

  Durable serverless event driven architecture reconciliation operation with per-step progress.

• PolicyVersionpolicyId, scope, version, rules, effectiveAt, status

  Auditable serverless event driven architecture policy evaluated before provisioning or mutation.

• ReconciliationCheckpointresourceId, provider, observedVersion, cursor, lastError, updatedAt

  Provider-specific serverless event driven architecture observation and recovery cursor.

Data flow

1. 1. Accept a desired-state commandThe serverless event driven architecture control plane authenticates the tenant, validates policy and quotas, checks the expected version, and records the desired state.
2. 2. Plan a safe operationA planner turns serverless event driven architecture desired state into ordered, bounded steps with dependency checks, blast-radius limits, and rollback metadata.
3. 3. Reconcile providers asynchronouslyWorkers apply serverless event driven architecture operations through provider adapters, persist checkpoints, rate-limit calls, and treat unknown outcomes as observable state.
4. 4. Publish observed healthThe serving projection joins desired and observed serverless event driven architecture state with operation status, policy version, freshness, and actionable errors.
5. 5. Recover and auditRetries, dead letters, drift detection, and operator approvals repair serverless event driven architecture resources without losing the original command or provider evidence.

Deep dives and trade-offs

• Desired versus observed stateKeep serverless event driven architecture desired state separate from provider-observed state and show both to operators. Make every reconciliation step conditional and resumable so a worker crash does not restart unsafe effects. Version policy and resource state so old operations cannot overwrite newer intent.
• Provider failures and unknown outcomesUse provider-specific idempotency tokens and query-after-timeout behavior for serverless event driven architecture operations. Bound retries with exponential backoff, circuit breakers, and per-provider quotas. Route irreconcilable drift to an approval or quarantine path instead of retrying forever.
• Blast radius and operationsPartition serverless event driven architecture work by tenant, region, cluster, or resource class and cap concurrent mutations. Audit who changed desired state, which policy allowed it, and what provider evidence was observed. Alert on drift age, operation backlog, failed steps, policy denials, and stale observations.
• Push versus pull reconciliationUse event triggers for fast response and periodic scans for missed events, drift, and recovery. A push-only serverless event driven architecture controller silently misses changes when a provider event is lost.
• Central control plane versus provider-native controllersKeep policy, intent, and audit centralized while isolating provider-specific application logic behind adapters. A monolithic controller becomes hard to scale and couples unrelated provider failure domains.
• Automatic repair versus approvalAutomate low-risk, reversible serverless event driven architecture changes and require approval for destructive or high-blast-radius operations. Full automation without policy or blast-radius controls can turn a transient signal into a widespread outage.