Menu

Model Serving

Serving 10,000 req/s with Ray Serve: Dynamic Batching, Model Multiplexing, and GPU Utilization