Design a Model Serving Infrastructure with Canary Deployments — System Design Interview Practice
Design a model serving infrastructure that deploys ML models as scalable APIs with support for canary deployments, traffic splitting, auto-scaling, and multi-framework model support. Work through the requirements, architecture trade-offs, and an interactive design review.
Requirements and concepts to consider
- Serve models as REST/gRPC endpointsRequirement
- Support canary deployments with gradual rolloutRequirement
- Auto-scale based on request volume and latencyRequirement
- Support multiple ML frameworks (PyTorch, TF, ONNX)Requirement
- Implement request batching for throughputRequirement
- Handle model warm-up and pre-loadingRequirement
- Provide request logging and latency monitoringRequirement
- Support A/B testing between model versionsRequirement
- mlopsConcept to explore
- model servingConcept to explore
- canary deploymentConcept to explore
- kubernetesConcept to explore
- auto scalingConcept to explore
- inferenceConcept to explore