When an ML platform engineer asks "how should I scale this endpoint?" the answer almost always starts with the wrong question. The right question isn't "how much traffic does it get?" — it's "what shape does that traffic take?"
RPS (requests per second) is a scalar. Traffic shape is a distribution. Two models can have identical average RPS but completely different infrastructure requirements because one spikes to 50× its mean for 90 seconds every morning, while the other maintains a flat 1.2× variance around its daily mean. A scaling policy that fits one will fail the other — and on a fleet of 100 models, you will have both, and dozens of archetypes in between.
The four archetypes we observe in production
After profiling request distributions across ML platform teams running 50–200 model fleets, we've converged on four archetypes that cover the vast majority of inference workload behavior:
Bursty. Characterized by a burst coefficient above 8× (ratio of peak to baseline RPS over a 24-hour window), sharp inter-arrival time variance, and a diurnal periodicity that correlates with human behavior cycles — morning session start, lunch, evening prime time. Recommendation systems and content ranking models almost universally fall into this category. The defining property: traffic onset is fast (seconds), sustain is moderate (minutes to hours), and drop-off is gradual. A scaling policy needs to pre-warm ahead of onset and hold capacity through the sustain phase.
Steady. Burst coefficient below 2×, low inter-arrival variance, no meaningful diurnal pattern. LLM serving endpoints often exhibit this profile when session concurrency is the primary driver rather than per-request volume. The challenge is not cold-start prevention but replica persistence — the worst thing you can do to a steady LLM endpoint is scale it to zero between low-traffic windows because the next session will incur an 8–12 second cold-start penalty for the user waiting on the first token.
Batch. Near-zero inter-arrival time variance within a job, but discrete job arrivals with long idle gaps between them. Periodic scoring pipelines, offline feature generation endpoints, and nightly ML jobs fit this profile. Burst coefficient can be very high during the job (all requests arrive simultaneously), but zero outside of it. Scale-to-zero is appropriate here — the key is detecting job arrival before the first request, not after, so that cold-start time doesn't eat into the job's SLA.
Hybrid. A combination of steady base load with burst overlays, or multiple diurnal cycles with different peak shapes. Fraud and risk scoring models often exhibit this — a steady baseline from continuous transaction scoring overlaid with sharp burst events during payment settlement windows. Each component needs a different scaling response, and a blended policy satisfies neither.
Why global scaling policies fail all four
The Kubernetes Horizontal Pod Autoscaler and most managed ML serving platforms expose a single scaling configuration per endpoint: a CPU or memory threshold at which scale-out triggers, a minimum replica count, and a maximum. This model encodes the assumption that traffic intensity correlates linearly with resource consumption — which is true for stateless web services and false for inference endpoints.
GPU inference workloads violate the CPU-scaling assumption in three ways:
First, GPU utilization does not scale linearly with request rate for most model architectures. Batching effects mean that a GPU can serve 10 requests at similar utilization to serving 1, until the batch saturates the compute units. CPU-based scaling signals see "low utilization" even when latency is already climbing because request queue depth is growing faster than the CPU metric reveals.
Second, cold-start time is asymmetric and large. A web service cold-start is typically sub-100ms. Loading a 13B parameter model into A100 GPU memory takes 40–90 seconds. This means that reactive scaling — triggering scale-out after traffic arrives — is always too late for any workload that has a fast-onset burst.
Third, a global min-replica setting is economically incoherent across a heterogeneous fleet. Setting min_replicas: 2 for every model in your fleet to prevent cold-starts means you're paying for 2 warm replicas of your overnight batch scorer that runs at 2am. Setting min_replicas: 0 everywhere to save cost means your 9am recommender cold-starts during the breakfast spike.
Traffic shape profiling in practice
The Traffic Shape Profiler in MLSrvyn runs a continuous analysis loop per endpoint. For each model, it maintains a rolling window of request inter-arrival times and computes the following features over 24-hour and 7-day windows:
- Burst coefficient: ratio of 95th percentile RPS to median RPS over a 24-hour window
- Diurnal periodicity score: autocorrelation of the RPS time series at 24-hour lag
- Inter-arrival variance: coefficient of variation of request inter-arrival times within 15-minute bins
- Peak onset rate: slope of RPS ramp-up at the start of each detected traffic burst
These four features are sufficient to classify endpoints into the four archetypes with high confidence after 5–7 days of observation. The profiler outputs a scaling archetype tag and a set of archetype-specific scaling parameters: minimum replica count derived from overnight baseline, scale-up threshold set at a percentage of the p99 SLO budget, pre-warm window calculated from peak onset rate, and maximum replicas bounded by the traffic peak estimate.
What changes when scaling is traffic-shape aware
The practical outcome is not subtle. For a Bursty recommendation model with a 10,000 RPS morning peak: pre-warming triggers 10 minutes before predicted onset, replicas are ready when the first users arrive, p99 stays inside the 30ms SLO through the spike, and capacity scales back down over the morning plateau without holding excess replicas through the afternoon. For a Steady LLM endpoint: minimum replicas maintain a warmed pool calculated from historical session concurrency, scale-to-zero is never triggered, and scale-out happens based on concurrency queue depth rather than CPU — 30–45 seconds before latency degrades, not after.
For a fleet of 100 models, the aggregate effect is a GPU utilization profile that tracks actual demand rather than the envelope of the worst-case model in the fleet — which is what a global policy forces you to provision for. That gap — between actual demand and worst-case envelope — is where your GPU budget goes when you run a single scaling policy across a heterogeneous fleet.
Traffic shape analysis isn't a new concept. It's what network capacity planners have been doing for decades. What's new is applying it at the per-model inference endpoint level, where the hardware cost of getting it wrong is measured in GPU-hours rather than bandwidth-minutes.