Design a Distributed Stream Processing System like Kafka — System Design Interview Practice
Design a distributed streaming platform for real-time data pipelines and event-driven architectures. Work through the requirements, architecture trade-offs, and an interactive design review.
Requirements and concepts to consider
- Publish and subscribe to streams of recordsRequirement
- Store streams durably and fault-tolerantlyRequirement
- Process streams in real-timeRequirement
- Support high throughput (millions of messages/sec)Requirement
- Guarantee message ordering within partitionRequirement
- Support message replayRequirement
- Handle failures gracefullyRequirement
- Scale horizontallyRequirement
- streamingConcept to explore
- messagingConcept to explore
- distributed systemsConcept to explore
- event drivenConcept to explore
- kafkaConcept to explore