Use Case

Recommendation Systems — absorb morning spikes without cold-start failures

Recommendation models are the most latency-sensitive workload in most ML fleets. A cold-start at 9am during the first user session wave kills engagement metrics for hours.

Request Access

The problem: scale-to-zero at night = cold-start at 9am

Generic auto-scaling tools optimize for cost by scaling recommendation replicas to zero or near-zero during overnight low-traffic windows. The result is predictable: when morning traffic arrives, there are no warm replicas to serve it. Cold-start time for a typical two-tower recommendation model on GPU is 40–120 seconds — and every cold-start request either times out or receives a degraded fallback response.

The damage isn't just latency. Cold-start-induced fallbacks corrupt your engagement metrics, distort A/B test results, and ripple into downstream personalization pipelines that read from recommendation outputs.

inference-gateway.log — 09:02:31

WARN  model=rec-v3  replica_count=0  cold_start_triggered
WARN  p99_latency=94ms  slo_threshold=30ms  SLO_BREACH
INFO  scaling_up: 0→3 replicas  eta=85s
INFO  fallback_response_rate=38%  (09:02–09:03)

How MLSrvyn handles bursty recommendation traffic

MLSrvyn's Traffic Shape Profiler classifies your recommendation model as Bursty archetype within the first week of observation. It detects the diurnal pattern (low overnight, sharp morning spike, sustained midday plateau, evening tail) and builds a predictive pre-warming schedule.

Replicas are pre-warmed 8–12 minutes before the predicted traffic onset — enough time for model weights to load into GPU memory before the first real request arrives. The SLO engine then maintains p99 compliance throughout the spike by comparing real-time latency histograms against your configured threshold and adding replicas if the p99 approaches the SLO ceiling.

mlsrvyn.yaml — rec-v3 policy

model: rec-v3
archetype: bursty
slo:
  p99_ms: 30
  p50_ms: 12
scaling:
  min_replicas: 1  # never cold
  max_replicas: 12
  prewarm_minutes_before_peak: 10
  scale_up_slo_pct: 80  # scale at 80% of p99 budget

94ms → 11ms

p99 cold-start latency eliminated — morning spike

Fallback response rate during spike (was 38%)

41%

GPU cost reduction vs. over-provisioned baseline

Ready to eliminate cold-start failures?

Request access and we'll profile your recommendation fleet's traffic shape in the first week.

Request Access See Benchmark Numbers