Design a Log Anomaly Detection System — System Design Interview Practice
Design a system that automatically detects anomalies in application logs using ML, identifies new error patterns, clusters similar log messages, and correlates log anomalies with system events. Work through the requirements, architecture trade-offs, and an interactive design review.
Requirements and concepts to consider
- Parse and normalize heterogeneous log formatsRequirement
- Detect anomalous log patterns automaticallyRequirement
- Cluster similar log messages into templatesRequirement
- Identify new/unseen error patternsRequirement
- Correlate log anomalies with metrics and eventsRequirement
- Track log volume and pattern trendsRequirement
- Alert on critical log anomaliesRequirement
- Support natural language search over logsRequirement
- aiopsConcept to explore
- log analysisConcept to explore
- anomaly detectionConcept to explore
- log parsingConcept to explore
- observabilityConcept to explore
- mlConcept to explore