Recommendation models fail differently from other ML workloads. This isn't a minor operational nuance — it's a structural fact about the workload that should drive every infrastructure decision, from GPU selection to scaling policy to cache architecture.
The three properties that make recommendation inference distinct: asymmetric read/write load between the candidate scoring phase and the retrieval phase, stale-feature sensitivity that creates latency-quality tradeoffs that don't exist in other model types, and burst patterns tied directly to human behavior cycles that are both highly predictable and non-negotiable from an SLO perspective.
Asymmetric load: two-tower architecture in production
Most production recommendation systems at scale use a two-tower architecture: a query tower that encodes the user context into an embedding vector, and an item tower that pre-computes embeddings for the candidate item catalog. At serving time, you compute the query embedding from the current user context (a real-time, latency-sensitive GPU operation) and then perform approximate nearest-neighbor (ANN) search over the pre-computed item embeddings (a CPU or specialized-hardware operation against a vector index).
The GPU load is on the query tower computation: per-request, proportional to request rate, and latency-sensitive. The item tower computation is largely offline — item embeddings are pre-computed and updated periodically, not per-request.
This asymmetry has a direct infrastructure implication: you're scaling the query tower computation to handle real-time request rate, and separately managing the item embedding index refresh pipeline. These two systems have completely different scaling characteristics and failure modes. Teams that treat recommendation serving as a single inference endpoint often discover this when the item embedding index becomes stale during a high-traffic period (because the refresh pipeline is competing for GPU resources with the query tower) or when the ANN index grows too large to serve from memory and latency spikes.
The operational practice: maintain explicit separation between the real-time query tower serving path (GPU, SLO-bound, per-request scaling) and the item embedding pipeline (batch, offline, not SLO-bound). Don't co-locate them on the same GPU pool under the same scaling policy.
Stale-feature sensitivity and the latency-quality tradeoff
Recommendation models depend on user feature freshness. A user who added 5 items to their cart 2 minutes ago should have those interactions reflected in their user embedding for the next recommendation request. If your user feature pipeline has a 10-minute lag, the recommendation model doesn't know about those cart adds.
The latency-quality tradeoff appears when feature pipelines are under load: if you're batching feature updates to reduce write load on the feature store, you're also increasing the staleness of the features that the recommendation model receives. Under high request rates, this can manifest as recommendation quality degradation — users seeing recommendations that ignore their recent activity — that your latency dashboards won't show because p99 looks fine.
The operational signal to watch: feature staleness percentile per model. A recommendation model with P95 feature freshness of 30 seconds is in a different operational state than one with P95 freshness of 5 minutes, even if their p99 inference latency is identical. This requires instrumenting the feature pipeline to publish staleness timestamps alongside feature values — not just inferring staleness from pipeline throughput metrics.
Burst patterns and the morning spike problem
Consumer recommendation models exhibit the strongest diurnal patterns of any ML workload we've observed. The autocorrelation of the RPS time series at 24-hour lag is typically above 0.85 — meaning yesterday is a very good predictor of today. The morning burst onset is fast: consumer products see RPS go from baseline (5–10% of daily peak) to 40–60% of daily peak within 8–12 minutes of the first session wave.
The GPU inference challenge is not the peak itself — it's the onset rate. A scaling policy that reacts to the RPS increase is already too late for a model with a 60-second cold-start. By the time the autoscaler observes a 3× increase in RPS, the cold-start-induced SLO breach has already been running for 45 seconds.
The solution is proactive pre-warming, as described in our earlier post on cold-start elimination. For recommendation specifically, there's an additional refinement: the morning burst pattern varies by day of week. Saturday and Sunday mornings on a consumer product have different onset times, onset rates, and peak durations than Tuesday mornings. A pre-warming schedule that uses a 7-day rolling average without day-of-week segmentation will consistently under-warm on weekend mornings and over-warm on Tuesday mornings.
Getting p99 under 30ms at scale
A common ML platform target for consumer recommendation models is p99 < 30ms at peak traffic. This is achievable at scale, but it requires stacking several optimizations together:
GPU selection: A100 80GB SXM (520 GB/s HBM bandwidth) is sufficient for most two-tower recommendation models up to ~500M parameters. For larger models or very high concurrency, H100 SXM (3.35 TB/s HBM3 bandwidth) provides a 6× memory bandwidth improvement that translates directly to faster query tower forward passes under concurrent load.
Batch size tuning: recommendation models are often served with sub-optimal batch sizes for latency. A batch of 1 request per forward pass minimizes per-request latency but wastes GPU compute. A batch of 32 maximizes throughput but adds queuing latency. For a p99 < 30ms target, the practical range is typically batch size 4–8, calibrated by measuring p99 vs. batch size at your actual request rate.
Query tower quantization: INT8 quantization of the query tower reduces memory footprint and typically improves throughput by 1.5–2× with less than 1% retrieval quality degradation on standard evaluation benchmarks. The quality degradation is workload-specific — measure on your own model and traffic distribution before deploying.
ANN index warm-up: ANN search latency spikes when the index isn't loaded into FAISS or ScaNN's in-memory structures. If you scale down the recommendation serving replicas overnight and bring them up fresh in the morning, the first 60–120 seconds of operation have elevated ANN latency while the index loads. Pre-warm the ANN index as part of the replica startup sequence, before the replica is added to the load balancer pool.
None of these optimizations is novel in isolation. What makes them effective together is applying them within a scaling policy that maintains warm replicas ahead of burst onset, not in the reaction period after burst has already hit.