Production LLM serving engines frequently crash due to CUDA Out-of-Memory errors under dynamic workloads. This systems-focused analysis explains how key-value cache fragmentation occurs and how modern inference engines use virtual-memory-inspired PagedAttention to eliminate…
Engineers have tackled KV‑cache fragmentation by discarding the traditional contiguous‑allocation model and adopting PagedAttention. In a typical serving pipeline the GPU reserves space for the longest possible sequence, which leaves 60–80 % of the memory idle as internal and external fragmentation and can provoke CUDA out‑of‑memory failures.
PagedAttention breaks the KV cache into uniform physical blocks—usually 16 tokens each—and uses a lookup table to bind those blocks to scattered memory locations. The approach trims wasted memory to less than 4 % and pushes inference throughput up roughly twofold.
The Anatomy of the KV Cache in LLM Inference
During autoregressive decoding, an LLM emits tokens one at a time, and each new token must attend to every token that has already been produced. To spare the model from recomputing the quadratic attention matrix at every step, inference engines store the Key and Value tensors of all previous tokens in GPU High‑Bandwidth Memory; this storage is referred to as the KV cache. Caching keys and values yields a dramatic speedup, but the cache’s memory demand is massive and grows linearly with both batch size and sequence length. A 70‑billion‑parameter model running in 16‑bit precision can consume several gigabytes of KV cache per request.
Allocating, tracking, and freeing that memory across many concurrent users is among the toughest engineering problems in production‑grade machine‑learning systems.
How Static Memory Allocation Triggers CUDA OOM
This flow outlines the sequence of events where dynamic request lengths combined with rigid memory allocation rules lead to system-wide crashes.
- 1Dynamic Request Arrival
Inference engine receives multiple concurrent user queries of highly variable lengths.
Inputs range from short prompts to multi-turn chat histories.
- 2Contiguous Pre-allocation
Engine reserves a contiguous GPU memory block for each request based on the maximum context window.
Memory is locked regardless of actual output length.
- 3Internal & External Fragmentation
Short responses leave allocated memory unused (internal), while finished requests leave scattered gaps (external).
Up to 80% of GPU memory becomes unusable but locked.
- 4Allocation Failure
A new request arrives requiring contiguous memory, but no single block of sufficient size exists.
Total free memory is high, but fragmented.
- 5CUDA OOM Crash
The GPU driver throws a fatal Out-of-Memory error, crashing the serving container.
Requires service restart and drops active requests.
How Static Allocation Triggers CUDA OOM Failures
Early LLM serving frameworks allocated memory for the KV cache contiguously. When a new request arrived, the engine pre-allocated a contiguous block of physical GPU memory sufficient to hold the KV cache for the maximum possible sequence length (for example, 2048 or 4096 tokens). This static allocation strategy was simple to implement but highly inefficient for dynamic production workloads. Because user queries and model outputs vary wildly in length, most requests finish long before reaching the maximum sequence limit. The unused pre-allocated memory remains locked and inaccessible to other requests, causing massive internal fragmentation.
Additionally, as requests of varying lifetimes start and finish, the physical memory space becomes peppered with small, non-contiguous free gaps. When a new request demands a large contiguous block, the system fails to find one despite having ample total free memory. This external fragmentation results in a fatal CUDA Out-of-Memory (OOM) crash, forcing the entire serving instance to restart.
The Three Dimensions of KV Cache Memory Waste
To design effective mitigation strategies, engineers categorize KV cache memory waste into three distinct behaviors. The first is internal fragmentation, which occurs because the system must reserve space for the maximum possible sequence length, even if the actual generation only uses a fraction of it. The second is reservation waste, which happens when the engine reserves memory for the next generation step before the token is actually computed. The third is external fragmentation, which arises from the dynamic creation and destruction of memory blocks of varying sizes. In typical production setups, these three factors combine to waste up to 60% to 80% of the total allocated GPU memory.
This artificial memory pressure severely limits the maximum batch size the engine can support, which in turn degrades system throughput and increases the cost per token.
Static Allocation vs. PagedAttention Memory Architectures
A comparison of memory management strategies in LLM serving engines.
| Factor | Engineering view | Why it matters |
|---|---|---|
| Memory Layout | Contiguous blocks per sequence | PagedAttention: Non-contiguous fixed-size physical blocks |
| Internal Fragmentation | High (wasted space up to max sequence length) | PagedAttention: Extremely low (limited to the last active block) |
| External Fragmentation | Severe (creates unusable memory gaps) | PagedAttention: Zero (all blocks are uniform in size) |
| Typical Memory Waste | 60% to 80% of total GPU memory | PagedAttention: Under 4% of total GPU memory |
| Addressing Overhead | Zero (direct physical addressing) | PagedAttention: Low (requires virtual block table lookup) |
| Throughput Capability | Low (small batch sizes due to memory limits) | PagedAttention: High (enables 2-4x larger batch sizes) |
PagedAttention: Applying Virtual Memory to GPU Tensors
To eliminate the limitations of contiguous allocation, modern LLM engines use PagedAttention, an architecture inspired by the virtual memory paging systems found in operating systems. Instead of allocating a single contiguous block of memory for each sequence, PagedAttention divides the KV cache of each request into fixed-size blocks. Each block contains the key and value vectors for a fixed number of tokens, typically 8, 16, or 32. These blocks do not need to be stored contiguously in physical GPU memory. The engine maintains a virtual block table that maps logical token sequences to physical memory blocks.
When the model generates a new token, the engine allocates a new physical block from a global pool only when the current block is fully occupied. This design eliminates external fragmentation entirely because all allocations are of a uniform, block-aligned size. Internal fragmentation is limited strictly to the final block of a sequence, reducing total memory waste to less than 4%.
Architectural Trade-offs of Paged Memory Management
While PagedAttention solves the fragmentation problem, it introduces new architectural trade-offs that system engineers must manage. The most immediate trade-off is the overhead of block table lookups. Because physical memory is non-contiguous, the GPU cannot perform simple contiguous memory reads during the attention computation. Instead, it must resolve physical addresses dynamically using the block table. This address resolution adds a small latency penalty. However, because LLM inference is highly memory-bandwidth bound rather than compute-bound, the performance gains from batching larger numbers of concurrent requests far outweigh the address lookup overhead. Furthermore, non-contiguous memory access can hinder memory coalescing on the GPU.
Modern implementations mitigate this by choosing block sizes (such as 16 tokens) that align well with GPU warp sizes and memory bus widths, ensuring high hardware utilization.
Production KV Cache Optimization Checklist
Use these configurations and architectural checks to prevent CUDA OOM errors in enterprise LLM deployments.
- Block Size Selection: Set block size to 16 tokens to align with GPU warp execution and memory coalescing patterns. (Avoid block sizes of 4 or 8 unless dealing with extremely short sequences.)
- GPU Memory Limit: Configure the GPU memory utilization threshold to 0.90 to reserve 10% headroom for dynamic activations. (Adjust down to 0.85 if running complex multi-modal or high-concurrency workloads.)
- Preemption Policy: Define a fallback strategy (Recompute vs. CPU Swap) for when physical block pools are completely exhausted. (CPU swapping introduces latency; recomputation is preferred for low-latency APIs.)
- Monitoring Metrics: Expose block pool utilization, preemption rates, and queue times to your Prometheus observability stack. (Set alerts when block pool utilization consistently exceeds 95%.)
Production Engineering Best Practices for KV Cache Management
When deploying LLM serving engines like vLLM or Hugging Face TGI in production, engineers should tune several parameters to optimize KV cache performance. First, select the block size carefully. A block size of 16 tokens is generally the sweet spot for modern NVIDIA GPUs (such as the A100 or H100), balancing internal fragmentation against memory access efficiency. Second, configure the GPU memory utilization parameter. This setting dictates what fraction of total GPU memory is dedicated to the KV cache pool versus model weights and activation memory. Setting this value too high (e.g., above 0.95) leaves insufficient headroom for dynamic activations during peak loads, risking runtime OOMs.
A conservative starting point of 0.90 is recommended. Finally, implement request preemption strategies. When the block pool is completely exhausted, the engine must either swap active blocks to CPU RAM or preempt (evict) low-priority requests to prevent system-wide failure.
Key takeaways
- The KV cache is a major memory bottleneck in LLM serving, scaling linearly with batch size and sequence length.
- Traditional contiguous memory allocation pre-allocates space for the maximum sequence length, wasting up to 80% of GPU memory.
- External fragmentation prevents the allocation of contiguous blocks even when total free memory is theoretically sufficient, causing CUDA OOMs.
- PagedAttention solves this failure mode by partitioning the KV cache into small, non-contiguous physical blocks managed via a virtual lookup table.
- By reducing memory waste to less than 4%, PagedAttention allows for significantly larger batch sizes and doubles production throughput.
- Production tuning requires balancing block size, reserving memory headroom for activations, and defining clear preemption policies.
Questions engineers often ask
What is the primary cause of CUDA OOM errors during LLM inference?
CUDA OOM errors during inference are primarily caused by the static and contiguous allocation of memory for the KV cache. Because the engine pre-allocates memory based on the maximum sequence length, much of the GPU High Bandwidth Memory remains locked and unused by shorter requests. This leads to internal and external fragmentation, preventing new requests from finding contiguous free memory blocks.
How does PagedAttention prevent memory fragmentation?
PagedAttention prevents fragmentation by breaking the KV cache of a request into small, fixed-size blocks (typically 16 tokens) that do not need to be stored contiguously in physical memory. A virtual block table maps logical sequences to these physical blocks, eliminating external fragmentation entirely and restricting internal fragmentation to only the last block of a sequence.
What is the optimal block size for PagedAttention in production?
For most modern NVIDIA GPUs, such as the A100 or H100, a block size of 16 tokens is optimal. This size balances the trade-off between memory waste (internal fragmentation within the block) and memory access efficiency (ensuring coalesced memory reads across GPU warps).
What happens when the PagedAttention block pool runs out of memory?
When the physical block pool is fully exhausted, the serving engine must preempt active requests to protect system stability. It does this either by swapping the KV cache blocks of lower-priority requests to CPU RAM (which introduces latency when swapped back) or by dropping the request and recomputing its tokens later.
Explore our deep-dive engineering guides to master high-performance AI infrastructure and database systems.
