Use Case
LLM Serving — replica persistence, not scale-to-zero
LLMs behave nothing like recommendation models. Long-running sessions, high VRAM footprint, and concurrency-limited throughput mean scale-to-zero is the wrong policy — and most auto-scalers get this wrong.
Request AccessThe problem: RPS-based scaling misunderstands LLM workload
Standard auto-scalers trigger scale-out on RPS (requests per second). For an LLM endpoint handling 3 concurrent sessions at 2am, RPS is near zero — so the scaler removes replicas. When the next session arrives, there are no warm replicas, and the 90-second cold-start destroys the user's first interaction.
Worse: LLMs are concurrency-limited, not throughput-limited. An H100 GPU running a 70B parameter model can handle roughly 4–8 concurrent requests before queue latency climbs. An RPS-based scaler never sees this saturation until p99 has already spiked well above SLO.
How MLSrvyn handles LLM serving
MLSrvyn classifies LLM endpoints as Steady archetype. The profiler detects that inter-arrival times are long, burst coefficient is low, and session duration is high — all hallmarks of LLM traffic. The resulting policy:
model: llm-chat-70b
archetype: steady
scaling_metric: concurrent_sessions # not RPS
slo:
ttft_p90_ms: 800 # time to first token
p99_ms: 4000 # full response
scaling:
min_replicas: 2 # never scale to zero
max_replicas: 8
sessions_per_replica: 5The scaling engine adds replicas when active session count approaches sessions_per_replica × current_count and removes them during genuine low-load windows — only after confirming zero active sessions on the replica being drained.
Scale your LLM serving intelligently
Request access and we'll map your LLM endpoints to the right scaling archetype based on your traffic data.