Engineering Blog
Technical writing on GPU fleet ops, traffic shape analysis, latency SLOs, and the practical engineering of ML inference at scale.
Global scaling policies treat every model the same — but a bursty recommender and a steady LLM need completely different scaling strategies. Here's how traffic shape profiling changes the math.
Releasing a partially-used GPU node is easy. Deciding which workloads to rebalance first — without violating latency SLOs — is the hard part. A deep dive into the bin-packing allocator.
Every morning your recommendation model cold-starts during the breakfast traffic spike. We traced this to a root cause most teams miss: the scaling policy has no memory of yesterday.
We ran a 72-hour benchmark on three LLM serving configurations — static replicas, naive scale-to-zero, and MLSrvyn traffic-shape scaling. The gap between best and worst was 41%.
Most autoscalers watch CPU and memory. But inference endpoints care about p99 latency. We explain why CPU-driven scaling always lags, and how SLO-driven scaling closes the gap.
After a year working with ML platform teams managing 50–200 model fleets, we catalogued the failure modes nobody writes about: version drift, silent SLO decay, and heterogeneous GPU costs.
Recommendation models fail differently from other ML workloads. They have asymmetric read/write load, stale-feature sensitivity, and burst patterns tied to human behavior cycles.
