Design a Cost-Optimized Architecture for Batch Processing — System Design Interview Practice
Design a batch processing system that can process large datasets efficiently while minimizing compute costs and handling interruptions gracefully. Work through the requirements, architecture trade-offs, and an interactive design review.
Requirements and concepts to consider
- Process large datasets in batchesRequirement
- Minimize compute costsRequirement
- Handle interruptions gracefullyRequirement
- Schedule batch jobsRequirement
- Manage job dependenciesRequirement
- Store input/output data efficientlyRequirement
- Monitor job progressRequirement
- Retry failed jobs automaticallyRequirement
- awsConcept to explore
- batch processingConcept to explore
- ec2Concept to explore
- spot instancesConcept to explore
- step functionsConcept to explore
- cost optimizationConcept to explore