Design an Intelligent Observability Platform — System Design Interview Practice

Interview prompt

Design a unified observability platform that ingests metrics, logs, and traces across cloud boundaries, discovers service topology, supports correlated investigation, and keeps retention/query cost predictable.

• Define common telemetry envelopes, resource identity, timestamps, exemplars, trace context, topology edges, tenant boundaries, and retention tiers.
• Batch/compress at agents, partition by tenant/series/time, control cardinality, and support correlated queries without coupling hot ingestion to analytics.
• Separate collection, durable WAL, hot/cold storage, indexing, topology discovery, alerting, and AI-assisted investigation with replayable pipelines.
• Explain clock skew, sampling, gaps, privacy, cost, query isolation, observability of the platform, and degraded local buffering.

Requirements and scale assumptions

• Collect, normalize, enrich, store, and query metrics/logs/traces with trace correlation, service topology, exemplars, and freshness indicators.
• Provide dashboards, distributed trace search, log/metric correlation, topology views, alerts, anomaly context, and tenant-scoped access.
• Support sampling/retention policies, deletion, schema evolution, replay, backfill, ingestion health, and recovery after storage/query failure.
• Acknowledge telemetry within one second for healthy agents and serve common investigation queries with p95 under five seconds.
• Ingest 1M metrics, 10M logs, and 100k traces per second across hybrid environments without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 1M metrics/s, 10M log lines/s, and 100k traces/s across 10k services
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 1M metrics/s; 10M logs/s; 100k traces/s — Capacity assumption that drives partitioning and backpressure.
• Latency target: ack < 1s; query p95 < 5s — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Durable telemetry and resource metadata are authoritative; indexes, topology, and correlations are derived.
• Async boundary: At-least-once workers — Keep Use OpenTelemetry for standardized telemetry collection, Implement columnar storage (ClickHouse) for metrics and logs, Use graph databases for topology relationships off the synchronous path.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable intelligent observability platform source evidence and ingestion cursor.

• SchemaVersiondatasetId, version, compatibility, owner, effectiveAt, status

  Governed intelligent observability platform contract used to validate producers and consumers.

• ProcessingRunrunId, inputWatermark, checkpoint, qualityStatus, codeVersion, status

  Checkpointed intelligent observability platform processing attempt with quality and lineage metadata.

• AnalyticalDatasetdatasetId, partition, watermark, schemaVersion, qualityStatus, location

  Curated intelligent observability platform serving partition with freshness and quality state.

Data flow

1. 1. Register sources and contractsThe intelligent observability platform catalog records owners, schemas, compatibility rules, retention, lineage, and partitioning before data is accepted.
2. 2. Ingest with backpressureConnectors checkpoint intelligent observability platform source cursors, validate schema and deduplication keys, and slow producers when downstream capacity is exhausted.
3. 3. Process event time with checkpointsStream or batch engines compute intelligent observability platform transformations using watermarks, late-data policy, state checkpoints, and deterministic code versions.
4. 4. Publish quality-gated datasetsOnly intelligent observability platform outputs that pass completeness, freshness, validity, and privacy checks become visible to analytical consumers.
5. 5. Serve, replay, and reconcileConsumers read bounded partitions with freshness metadata while operators replay failed intelligent observability platform ranges and compare output checksums.

Deep dives and trade-offs

• Schema evolution and data qualityVersion intelligent observability platform contracts and make compatibility rules explicit for every producer and consumer. Quarantine malformed partitions instead of poisoning the whole dataset. Track row counts, null rates, duplicates, distribution changes, and policy violations by partition.
• Watermarks, late data, and exactly-once effectsUse source cursors and event-time watermarks for intelligent observability platform progress, not wall-clock assumptions. Make checkpoints, output keys, and sink commits retry-safe under at-least-once delivery. Document how late events revise windows, aggregates, or snapshots.
• Replay, lineage, and costKeep immutable intelligent observability platform raw evidence and code or schema versions so failed outputs can be reproduced. Separate hot serving storage from cold retention and cap replay concurrency. Measure freshness, backlog, compute cost, storage growth, and quality-gate failure rate.
• Streaming versus batchUse streaming for freshness-critical intelligent observability platform paths and batch for backfills, compaction, and expensive recomputation. Forcing every workload into streaming makes state, replay, and cost harder to operate.
• Raw retention versus curated-only storageRetain enough immutable raw evidence for replay, audit, and correction, then tier or expire it according to policy. Without raw evidence, a bad transformation can require an unreproducible emergency fix.
• Central warehouse versus domain-owned datasetsCentralize governance and discovery while letting domain owners own contracts and quality signals. A single team owning every transformation becomes a delivery bottleneck and hides data ownership.