Operating a 100-model inference fleet is qualitatively different from operating a 10-model fleet. The problems are not 10× harder. Some are exponentially harder; some appear only above a certain fleet size threshold; and some are new failure modes that simply don't exist at smaller scale.
After working with ML platform teams running 50–200 model fleets in production, we've catalogued the failure patterns that come up repeatedly but rarely make it into blog posts or conference talks. This is that list.
Version drift: the silent SLO killer
Model version management sounds like a model registry problem. It is. But the inference infrastructure problem it creates is version drift: the state where different replicas of the same model are serving different model versions, and you don't know which ones.
This happens when a deployment update doesn't complete cleanly: the rollout starts, 3 of 8 replicas are updated to v2, then the deployment fails or is paused, and you end up with a 3:5 split between v1 and v2. Depending on your load balancer, some users see v1 responses and some see v2. Your A/B test results are corrupted. Your recommendation metrics show unexplained variance. Your oncall team spends two hours debugging a "latency regression" that turns out to be a model quality difference between versions, not a serving infrastructure problem at all.
At 100+ models, version drift happens constantly because the cadence of model updates is high. The operational practice that prevents it: per-endpoint version tracking in the fleet dashboard, automated alerts when replica version counts diverge within an endpoint, and a deployment policy that treats a partial rollout as a failure state requiring explicit operator action to complete or roll back — not a state that silently persists.
Silent SLO decay
A model that was within SLO when you deployed it may drift out of SLO over weeks as the input distribution shifts, model staleness increases, or traffic shape changes. This doesn't show up as a paging alert. The p99 doesn't spike above the SLO ceiling — it creeps upward by 2ms per week. After 6 weeks, the model is at 28ms against a 30ms SLO and any traffic spike will push it over.
Silent SLO decay is hard to detect without trending. A point-in-time dashboard check tells you whether you're currently in SLO. It doesn't tell you whether you were better 4 weeks ago. The operational practice: per-endpoint SLO trend monitoring with a 30-day rolling window, alerting on week-over-week p99 increase above a threshold (e.g., 5% per week), not just on current SLO breach.
At 100+ models, you cannot manually trend all endpoints. Automated per-endpoint SLO trend alerts are not optional — they're the difference between discovering a drifting endpoint before it pages you at 2am versus discovering it when it's already causing user impact.
Noisy-neighbor GPU contention
When you co-locate multiple models on the same GPU (via bin-packing or manual placement), you need to monitor for noisy-neighbor effects: one model's sudden traffic spike consuming GPU compute units and degrading the SLO of its co-located neighbors.
Noisy-neighbor detection requires per-model GPU utilization tracking, not just per-node GPU utilization. Most GPU monitoring stacks (DCGM, nvml) expose node-level metrics by default. Getting per-process GPU utilization requires configuring DCGM's per-process metrics exporter, which many teams skip because it adds metric cardinality.
At 100+ models across 40+ nodes, per-process GPU metrics add significant volume to your observability stack. The tradeoff is worth it: noisy-neighbor incidents at fleet scale cause cascading SLO breaches across multiple co-located endpoints simultaneously, and without per-model GPU utilization you can't distinguish "this model's traffic spiked" from "this node's GPU has a hardware issue."
Heterogeneous GPU type inventory drift
At the early stages of a production ML fleet, teams typically run on one GPU type. As the fleet grows, capacity constraints lead to purchasing or renting different GPU types: you started on A100 40GB PCIe, then added A100 80GB SXM, then provisioned H100 80GB SXM for the LLM endpoints. Now your model placement policy needs to account for the fact that your 70B model can only run on the H100 nodes, your older models were configured for A100 memory constraints, and your scaling policy min/max replica counts don't account for the performance difference between the two GPU generations.
GPU type inventory drift manifests as placement errors (a model assigned to a node that doesn't have enough HBM for its memory footprint) and performance surprises (a model calibrated on A100 performance characteristics showing unexpected latency on H100 because the compute-to-memory ratio is different). At 100+ models across heterogeneous GPU types, maintaining a consistent view of which model is placed on which GPU type, and what the SLO expectations are for each placement, becomes a first-class operational concern.
The alert fatigue problem
At 10 models, you can afford per-model p99 breach alerts that page oncall. At 100 models, if 5% of your fleet experiences a transient p99 breach simultaneously during a traffic spike, you've paged oncall 5 times in 30 seconds for an event that resolved itself without human intervention. The response to this is always the same: alert thresholds get raised, then raised again, until the alerts are no longer sensitive enough to catch real problems.
The operational practice that works at fleet scale: alert on sustained SLO breach (p99 above threshold for more than 90 seconds, not just a single measurement), alert on breach rate (more than 3 models breaching SLO simultaneously — which indicates a systemic issue, not individual model noise), and alert on breach trend (any model that has breached SLO more than N times in the past hour). This structure reduces alert volume while preserving sensitivity to real incidents.
The common thread
All of these failure modes share a structural property: they're invisible in per-model point-in-time dashboards and visible only in fleet-level time-series views. The ML platform teams that operate 100+ model fleets well are the ones that invested in fleet-level observability early — before the fleet was large enough to make operating it manually unsustainable. The ones who didn't are the ones we hear from after a 3am incident.