When we tell ML platform engineers that most fleets run at 30–40% average GPU utilization, they nod. They already know. What they often don't know is the precise mechanism producing that waste — and it's not what most people assume.
The common explanation is over-provisioning: teams allocate more GPU capacity than they need because latency SLOs are unforgiving and cold-starts are expensive. That's part of it. But there's a second, less visible driver: one-model-per-GPU allocation habits, reinforced by orchestration tooling that treats each model as needing exclusive GPU tenancy even when the model's memory footprint would allow co-location.
This is the problem that GPU bin-packing addresses. And it's harder than it sounds.
The naive case and why it fails
The naive approach to GPU sharing is straightforward: if Model A uses 20GB of an 80GB H100 and Model B uses 18GB, you can fit both on the same GPU with 42GB of headroom. Pack them together and you've doubled the density of that GPU.
This works until Model A and Model B both receive requests simultaneously. GPU compute units are shared, not just GPU memory. Under simultaneous load, each model competes for CUDA cores, tensor cores, and memory bandwidth. If both models are at peak load concurrently and the GPU is saturated, you haven't saved anything — you've created a resource contention problem where both models violate their SLOs.
The key insight is that co-location is only safe when the models' traffic shapes have low temporal correlation — that is, when their peaks don't overlap. A recommendation model that peaks at 9am and an LLM endpoint that peaks at 11pm can often be co-located on the same GPU safely, because their compute-heavy windows don't coincide.
How the bin-packing allocator works
MLSrvyn's GPU Bin-Packing Allocator starts from the traffic shape profiles computed by the Traffic Shape Profiler. For each pair of models under consideration for co-location, it computes a temporal overlap score: the Pearson correlation of their request rate time series over a 7-day window. Models with correlation below a configurable threshold (default: 0.3) are candidates for co-location.
The allocator then runs a constraint-satisfaction pass over the candidate pairs. Constraints include:
- Memory envelope: the sum of each model's maximum resident memory (weights + KV cache peak) must not exceed the GPU's available HBM with a safety margin (default: 15%). This is computed from actual measurement, not model parameter count alone.
- Compute headroom: at each model's individual peak load, GPU utilization must not exceed 85% to preserve headroom for the co-located model's baseline traffic.
- SLO isolation: the SLO tier of each model constrains the aggressiveness of co-location. A p99 = 20ms model will never be co-located with another model whose peak RPS would saturate GPU compute units.
When the allocator decides to scale down a fleet (because the SLO engine has determined that fewer replicas can satisfy demand), it doesn't simply release replicas in place. It first rebalances: it identifies which workloads can be migrated to existing nodes with capacity headroom, migrates them, and only then releases the now-empty nodes. This prevents the classic waste pattern of 5 models on 5 nodes at 25% utilization each, all too busy to release.
The rebalancing sequence
The rebalancing sequence matters. Migrate the wrong model first and you might temporarily overload the destination node, causing SLO violations during the migration window. The allocator uses the following priority order:
First, identify the emptiest source node (lowest current utilization across all co-located models). Then identify candidate destination nodes that can accept one or more of the source node's models without violating memory or compute constraints. Migrate models with the lowest current traffic load first (smallest disruption risk). Verify that post-migration utilization on the destination node stays within the compute headroom constraint. Release the source node only after all migrations complete and a 60-second stabilization window has passed.
This sequence is what distinguishes a proper bin-packing allocator from a naive node consolidator. The stability window prevents thrashing — the allocator won't release a node only to need it again 90 seconds later when traffic picks up.
What the numbers look like in practice
In our benchmark on a fleet of 12 models across 8× H100 GPUs: before bin-packing, average GPU utilization was 34% and all models ran on dedicated nodes. After bin-packing with MLSrvyn, the 12 models ran on 5 GPUs (with 3 GPUs released) at 81% average utilization. SLO compliance was maintained at 99.7% over the measurement window.
The cost implication depends on your GPU provider pricing. On standard cloud H100 rates, 3 released H100 GPUs over a 30-day window represents a material cost reduction — without any change to the serving infrastructure, model code, or traffic patterns.
What bin-packing doesn't solve
It's worth stating the limits. Bin-packing only helps when you have models whose traffic shapes are temporally uncorrelated. If your entire fleet of 100 models all peak simultaneously during a daily batch job, there's no bin-packing opportunity — every model needs its full capacity at the same time. In that case, the right optimization is traffic shaping or job staggering, not co-location.
Bin-packing also doesn't help with models that have memory footprints approaching single-GPU capacity. A Llama 2 70B in FP16 requires ~140GB of HBM — it won't fit on a single 80GB H100 regardless of temporal correlation with other models. The allocator will correctly identify this as a no-co-location case.
Where bin-packing delivers the most value is in fleets with significant heterogeneity: small classifier models (2–8B parameters), medium recommendation models (10–50B), and steady LLM endpoints all running on the same GPU pool, with different diurnal patterns. That's exactly the kind of fleet that most ML platform teams operating 50+ models in production actually have.