Your recommendation model cold-starts at 9:02am every morning. p99 spikes to 94ms against a 30ms SLO. Fallback responses run at 38% for 90 seconds until the new replicas are warm. You've seen this in your latency dashboards. You've filed the incident reports. The fix everyone reaches for is "add more replicas overnight so they're warm in the morning." That works, but it costs GPU-hours you don't need to spend.
The actual root cause is simpler and more fixable: the scaling policy has no memory of yesterday.
Why standard autoscalers forget
Kubernetes HPA and most managed serving platforms operate on current state: what is my current replica count, what is my current CPU/RPS, what are my scaling thresholds? The decision to scale is always made in the present tense. There is no mechanism to say "yesterday at 9:02am traffic went from 50 RPS to 8,000 RPS in 45 seconds — I should start warming replicas at 8:52am today."
This isn't a bug. It's an architectural choice that made sense for the web service use cases these tools were designed for. A stateless web service cold-start is 50–200ms. The first request after a scale-to-zero event is slightly slower, but not catastrophically so. Users won't notice a 100ms delay on a web page load.
A GPU inference cold-start for a recommendation model (two-tower architecture, 50M parameters on A100) takes 40–90 seconds. Every request that arrives during that window either waits — accumulating in a queue that will itself cause an SLO breach when the model finally comes online and processes them all at once — or receives a degraded fallback response from a simpler model that doesn't know the user's context.
The damage compounds. Fallback responses corrupt engagement metrics. A/B tests that run during cold-start windows record incorrect control and variant performance. Personalization pipelines that consume recommendation outputs see a 90-second gap and backfill with stale data.
Cold-start is a traffic-shape problem, not a hardware problem
The morning cold-start is a direct consequence of diurnal traffic shape: traffic drops to near-zero during a low-traffic window (typically 2am–7am for consumer products), the scaling policy reduces replicas to the minimum or to zero to save cost, and the burst onset at 9am outpaces the time it takes to warm new replicas.
The hardware can't solve this — a faster GPU reduces cold-start time from 90 seconds to 40 seconds, but 40 seconds is still 40 seconds of fallback responses during a morning spike. The network topology can't solve it — putting the model closer to users reduces serving latency but not cold-start time. The only thing that eliminates the cold-start is having warm replicas before the traffic arrives.
The question is: how do you know when to warm them without over-provisioning?
Pre-warming from traffic shape history
MLSrvyn's Traffic Shape Profiler maintains a historical request rate time series per endpoint. After 5–7 days of observation, it identifies the diurnal periodicity: the autocorrelation of the RPS series at 24-hour lag. For a consumer recommendation model, this autocorrelation is typically above 0.8 — meaning yesterday's traffic pattern is a good predictor of today's.
From this, the profiler derives a pre-warming schedule: for each day-of-week, what is the predicted time of onset of the morning traffic spike, and what is the predicted onset rate (how fast RPS climbs at spike start)? This schedule drives a pre-warming action: N minutes before the predicted onset, scale out to the predicted peak replica count so that all replicas are warm before the first user request arrives.
The pre-warming window (N minutes) is calculated from the measured cold-start time of the model plus a buffer. For a model with a 60-second cold-start time, the pre-warming window defaults to 8 minutes — enough time for 6 model load cycles with overlap, ensuring that all target replicas are warm even if one fails to load on the first attempt.
The pre-warming action doesn't hold those replicas forever. After the morning spike subsides and traffic settles into its midday plateau, the SLO engine begins scaling back — gradually, tracking the p99 against the SLO budget to confirm that fewer replicas can maintain compliance before releasing capacity. This is not the same as the reactive scale-down that a naive autoscaler would do.
Minimum replica count: the other half of the fix
Pre-warming from history handles the predictable cold-start. But traffic spikes aren't always predictable. A flash sale, a viral social media moment, a marketing email that goes out 30 minutes early — these are real production events that the historical pattern doesn't anticipate.
The second lever is minimum replica count. A properly set minimum replica count for a Bursty archetype model is not zero — it's the number of replicas needed to serve baseline traffic while keeping enough warm capacity to absorb a moderate unexpected spike without immediate cold-start. For a recommendation model with a daily mean of 200 RPS and a typical morning spike to 8,000 RPS, the minimum replica count is typically 1–2 replicas (enough to serve baseline, enough to absorb a 2× unexpected spike while the scaling engine brings up additional capacity).
The economic argument for non-zero minimum replicas is straightforward: 2 warm H100 replicas overnight costs far less than the engineering and business cost of 90 seconds of 38% fallback response rate every morning. The question is not whether to maintain minimum replicas — it's how many, and that calculation requires knowing the model's traffic shape and cold-start time, not just setting a number by feel.
The result
With pre-warming from historical traffic shape and a correctly calculated minimum replica count: zero cold-start events during the morning spike, p99 inside SLO throughout the burst onset, and GPU cost roughly equal to a well-optimized static provisioning setup — because you're paying for warm replicas only when you need them, not all night.
The incident report that says "cold-start SLO breach at 09:02am" is a symptom of a scaling policy that operates without memory of what happened at 09:02am yesterday. The fix is giving the policy that memory.