Ray Serve
A production-grade model serving component in the Ray ecosystem. It supports high-concurrency inference requests, enables rapid deployment of trained models as online services, and provides elastic scaling capabilities.
Timeline (last 90 days)
In Ray Serve, the num_gpus in @serve.deployment(ray_actor_options={"num_gpus": 0.25}) is a logical resource quota declared to the scheduler, and at most 4 actor replicas can reside simultaneously on one GPU.
num_gpus is only a logical declaration and does not automatically isolate VRAM; whether they can coexist depends on unified memory capacity and llama.cpp's actual memory usage, and setting it too small while launching multiple 7B model instances may cause OOM.
Engineers need to make trade-offs between different model serving frameworks based on specific business scenarios
Ray Serve is built on top of the distributed computing framework Ray and is suited for complex service scenarios that require flexible orchestration of multiple models and business logic
Triton, vLLM, and Ray Serve are the mainstream model serving frameworks in the current MLOps landscape.
BentoML and Ray Serve focus more on flexible Python ecosystem integration as model serving solutions.
Ray Serve is a production-oriented model serving capability that transforms training results into inference services
All Facts (7)
In Ray Serve, the num_gpus in @serve.deployment(ray_actor_options={"num_gpus": 0.25}) is a logical resource quota declared to the scheduler, and at most 4 actor replicas can reside simultaneously on one GPU.
50%Unverifiednum_gpus is only a logical declaration and does not automatically isolate VRAM; whether they can coexist depends on unified memory capacity and llama.cpp's actual memory usage, and setting it too small while launching multiple 7B model instances may cause OOM.
50%UnverifiedEngineers need to make trade-offs between different model serving frameworks based on specific business scenarios
50%UnverifiedRay Serve is built on top of the distributed computing framework Ray and is suited for complex service scenarios that require flexible orchestration of multiple models and business logic
50%UnverifiedTriton, vLLM, and Ray Serve are the mainstream model serving frameworks in the current MLOps landscape.
50%UnverifiedBentoML and Ray Serve focus more on flexible Python ecosystem integration as model serving solutions.
50%UnverifiedRay Serve is a production-oriented model serving capability that transforms training results into inference services
50%