Build a Real-time Data Streaming Pipeline — System Design Interview Practice
Design a real-time data pipeline that can ingest, process, and analyze millions of events per second with sub-second latency. Work through the requirements, architecture trade-offs, and an interactive design review.
Requirements and concepts to consider
- Ingest millions of events per secondRequirement
- Process data in real-timeRequirement
- Perform streaming analyticsRequirement
- Store processed data for queriesRequirement
- Visualize real-time dashboardsRequirement
- Handle data transformationRequirement
- Support replay of eventsRequirement
- Monitor pipeline healthRequirement
- awsConcept to explore
- kinesisConcept to explore
- streamingConcept to explore
- elasticsearchConcept to explore
- analyticsConcept to explore
- real timeConcept to explore