The default scaling signal for most ML serving infrastructure is CPU utilization. This made sense when the tools were built. CPU is a universal metric, easy to instrument, and for web services it correlates reasonably with request handling capacity. For inference endpoints, the correlation breaks down.
Here's what happens when a recommendation model approaches saturation: GPU compute units become saturated before CPU shows anything unusual. Request queue depth grows. Individual request execution time extends as requests wait for GPU scheduling slots. p99 latency starts climbing. Then p90. Then p50. Meanwhile, CPU utilization — which reflects the inference gateway's request routing and response serialization work, not the GPU's forward pass — stays flat at 15–25%.
The CPU-based autoscaler sees 20% CPU utilization and doesn't scale. By the time CPU rises (because the inference gateway itself starts queuing requests that would previously have dispatched immediately), p99 is already above the SLO budget. Scale-out triggers. A new replica comes online 45–90 seconds later. For the duration of that cold-start window, every p99 request is a breach.
This isn't a misconfiguration problem. You could set the CPU threshold to 10% and trigger scale-out earlier — but you'd also trigger it constantly during normal traffic variation, thrashing your replica count and spending GPU-hours on replicas you don't need. The signal is wrong, not just the threshold.
Why latency histograms are the right scaling signal
The metric that actually determines whether your inference endpoint is serving correctly is the one defined in your SLO: latency. Specifically, the latency at a percentile that represents your worst-acceptable-case user experience — p99 for synchronous serving, p95 for batch pipelines, p50 for best-effort endpoints.
A latency histogram is not a single number. It's a distribution of how long recent requests took to serve. The shape of that distribution tells you things that a point metric doesn't: whether tail latency is detaching from median latency (a sign of queue buildup that precedes saturation), whether the p90–p99 gap is widening (a sign of resource contention under burst), and whether the entire distribution is shifting up (a sign of sustained load approaching capacity).
When you use the latency histogram as your scaling signal — specifically, when you trigger scale-out based on p99 approaching the SLO budget rather than waiting for it to exceed the SLO budget — you pre-empt the breach rather than react to it.
How SLO-driven scaling works in practice
MLSrvyn's SLO Engine maintains a per-endpoint latency histogram updated in real-time (15-second rolling windows). For each endpoint, the operator defines a latency SLO tier:
- A p99 ceiling: the hard latency budget for this model. Scale-out must be triggered early enough that new replicas are warm before this ceiling is breached.
- A scale-out trigger threshold: the percentage of the p99 budget at which scale-out begins. Default is 80% — meaning if your p99 SLO is 30ms, scale-out triggers when p99 reaches 24ms.
- A scale-down delay: the minimum time after p99 drops below the trigger threshold before replicas are released. Default is 300 seconds, to prevent thrashing during traffic fluctuation.
The scale-out trigger threshold is the critical parameter. Setting it too high (95% of budget) means you're reacting with only 5% headroom, which isn't enough if your model has a 45-second cold-start. Setting it too low (50% of budget) means you're over-reacting to normal traffic variation and spending on replicas you don't need. The right threshold is derived from the relationship between your model's cold-start time and the rate at which p99 climbs when your endpoint is approaching saturation — which varies by traffic shape archetype.
The 30–60 second lead time advantage
In our measurements across recommendation and fraud-scoring endpoints, SLO-driven scaling consistently triggered scale-out 30–60 seconds before the corresponding CPU-based threshold would have fired. That's the window that determines whether a new replica can be warm and serving before your SLO is breached.
For a model with a 45-second cold-start time: a 50-second lead time means the new replica comes online 5 seconds before the SLO breach. Borderline — but it works. A 30-second lead time means the new replica comes online 15 seconds after the SLO breach. You've already failed.
For a model with a 90-second cold-start time (a larger 13B parameter model): you need at least 100 seconds of lead time to guarantee a breach-free scale-out. CPU-based scaling can't reliably deliver 100 seconds of advance warning, because CPU doesn't show meaningful elevation until the GPU is already saturated — which at that point means p99 is already above budget.
SLO-driven scaling from latency histograms delivers the lead time because it observes the metric that actually deteriorates first: the distribution of execution times, not the utilization of the CPU that's routing requests to the GPU that's doing the work.
What you have to give up
SLO-driven scaling requires instrumenting your inference gateway to expose a real-time latency histogram, not just a summary p99 metric. Most observability stacks (Prometheus with histogram_quantile, OpenTelemetry with exponential histograms) support this, but it requires more careful configuration than dropping in a CPU exporter.
It also requires stating a p99 budget for each endpoint. Many teams have never formally defined their inference SLOs — they know their models need to be "fast" but haven't put a number to it. SLO-driven scaling makes that step mandatory. That's a feature, not a limitation: if you don't know what "acceptable latency" means for your recommendation model, you can't reason about whether your scaling configuration is correct.
The operational investment is worth it. A scaling signal that leads the actual problem by 30–60 seconds, rather than lagging it, is the difference between a scaling policy that prevents SLO breaches and one that minimizes how long they last.