LLM serving cost is one of the most contested numbers in ML infrastructure. Every provider quotes cost-per-token differently, model size affects the denominator in non-obvious ways, and the gap between peak throughput (used in marketing benchmarks) and real-world sustained serving (what you actually pay for) is large.
This post documents what we measured on H100 SXM hardware over a 72-hour continuous window, across three model sizes (7B, 13B, and 70B parameters), and three serving configurations: static provisioning, naive scale-to-zero, and MLSrvyn's session-aware traffic-shape scaling. We're publishing the methodology alongside the numbers so you can reproduce it on your own workloads.
Test setup
Hardware: 8× NVIDIA H100 SXM5 (80GB HBM3) nodes on a standard cloud GPU pool. No dedicated or reserved hardware. OS: Ubuntu 22.04. Serving framework: vLLM 0.4.x with continuous batching enabled. Models: 7B (Mistral 7B Instruct, FP16), 13B (Llama 2 13B Chat, FP16), 70B (Llama 2 70B Chat, FP16 across 2× H100 with tensor parallelism).
Traffic: a 72-hour replay of sanitized production LLM serving logs. The session distribution had a clear diurnal pattern: peak concurrency of 40–60 active sessions between 10am–11pm, dropping to 3–8 sessions between 2am–7am. Average session length: 14 turns. Average prompt length: 380 tokens. Average completion length: 420 tokens.
Three serving configurations were tested in sequence, each for 72 hours on the same hardware pool:
- Static provisioning: replicas sized for 90th percentile peak concurrency, held constant throughout the 72-hour window. Never scale down overnight.
- Naive scale-to-zero: standard Kubernetes HPA with CPU-based scaling, minimum replicas = 0. Scales to zero when CPU drops below 10% for 5 minutes.
- MLSrvyn session-aware scaling: Steady archetype applied to each model endpoint. Minimum replicas calculated from overnight baseline concurrency (3–8 sessions → 1 warm replica per model). Scale-out triggered by concurrency queue depth trend, not CPU. Scale-down delayed until confirmed zero-session window.
Results: 7B model
The 7B model had the most dynamic utilization pattern — small enough to fit on a single H100 with room for concurrent requests, but the small per-request memory footprint meant that concurrency changes had an outsized effect on GPU utilization.
Static provisioning: average GPU utilization 41% over 72 hours. No cold-start events. Cost: measured in normalized GPU-hour units, this configuration set our baseline at 1.0×.
Naive scale-to-zero: average GPU utilization 68% during active hours (better than static). But: 31 cold-start events over 72 hours (average cold-start duration: 38 seconds). During cold-start windows, time-to-first-token (TTFT) for the first session in a new scaling period ranged from 42–65 seconds. Users waiting on the first token of a new session experienced a multi-second stall that in a production product would register as an unacceptable failure. Cost: 0.73× baseline (27% reduction), but only by accepting those cold-start events.
MLSrvyn session-aware scaling: average GPU utilization 62% during active hours. Zero cold-start events. The minimum warm replica was maintained at 1 throughout the overnight window, consuming GPU-hours at ~8% utilization during the 2am–7am low-traffic period. Scale-out from 1 replica to 3–4 replicas happened proactively as session concurrency queue depth climbed, 35–50 seconds before the arrival of the session that would have exceeded single-replica throughput capacity. Cost: 0.71× baseline (29% reduction vs. static). The cold-start avoidance cost roughly 2 additional GPU-hours per day of overnight warm replica maintenance; the proactive scale-down recaptured more than that.
Results: 70B model
The 70B model is the most interesting case because its cold-start penalty is severe. Loading a 70B FP16 model across 2× H100 with tensor parallelism initialization takes 8–12 minutes of wall-clock time. Naive scale-to-zero for a 70B model means any session arriving after a zero-replica period waits up to 12 minutes for the first token — an obviously unacceptable user experience.
Static provisioning: baseline at 1.0×. Maintaining 2× H100 24/7 is expensive.
Naive scale-to-zero: 14 cold-start events over 72 hours. Average cold-start duration: 9.4 minutes. Cost: 0.61× baseline. Cheap — but those 14 cold-starts represent 14 user sessions that experienced a 9-minute wait for the first token. In a product context, that's 14 support tickets or 14 silent churns.
MLSrvyn session-aware scaling: zero cold-start events. The Steady archetype kept 1 warm replica pair (2× H100) through overnight periods by detecting the confirmed zero-session window before releasing capacity, and immediately pre-warming a replacement before the next predicted session onset. The 72-hour window showed 3 confirmed zero-session gaps (between 3am–5:30am on each day). MLSrvyn drained to 0 replicas during each gap, confirmed zero sessions, and pre-warmed 8 minutes before the predicted next session arrival. Cost: 0.79× baseline. Overnight drain-and-pre-warm saved more than the naive scale-to-zero on a per-hour basis, and zero cold-starts.
The 41% spread
The gap between the most expensive configuration (static provisioning on the 7B model, 1.0× baseline) and the most cost-efficient (naive scale-to-zero on the 7B model, 0.73×) is 27%. But naive scale-to-zero achieves that by accepting cold-start events that are unacceptable in production.
When you compare static provisioning against MLSrvyn session-aware scaling across all three model sizes — 7B (0.71×), 13B (0.74×), 70B (0.79×) — the weighted average reduction is 25–28% depending on fleet composition. That's the cost of operating inference without cold-starts. The 41% figure in our benchmark headline is the gap between the worst-configured static provisioning setup and the best-case bin-packed MLSrvyn configuration on a mixed fleet — which we'll cover in a separate post.
Methodology notes
Latency measured at the gateway (wall clock from request receipt to last token delivery). GPU cost normalized to GPU-hours at 100% utilization (not actual cloud billing rates, which vary). Cold-start defined as any request that waits more than 5× the model's normal p99 TTFT for its first token. All traffic replayed from identical log data; the only variable was the serving configuration. Tests were not interleaved — each ran consecutively to avoid warm cache effects from prior runs. Raw data and replay tooling are available in our documentation.