You are paying for a Ferrari but driving it in first gear. That is what happens when you run Large Language Models with naive serving strategies. The hardware sits idle, waiting for the slowest request in a batch to finish, while your cloud bill climbs. If you want to squeeze every drop of performance out of your GPUs, you need to understand two specific mechanisms: KV Caching and Continuous Batching. These aren't just buzzwords; they are the engineering reasons why modern inference servers like vLLM can handle ten times more traffic than older systems on the same hardware.
Let's break down why this matters before we get into the code or architecture. In traditional static batching, you collect a group of requests, process them together, and wait until all of them are done before starting the next group. Imagine one user asks for a short summary (50 tokens) and another wants a full essay (2,000 tokens). Your GPU finishes the summary instantly but has to keep processing the essay for thousands of steps. During those extra steps, the resources allocated to the finished summary sit empty. Continuous batching fixes this by swapping completed requests out and slotting new ones in immediately. It keeps the GPU saturated. But to do this efficiently, the system needs to remember what it already computed so it doesn't redo work. That is where KV caching comes in.
The Math Behind KV Caching
To grasp why memory management is the bottleneck, you have to look at how Transformers work. When a model generates text, it uses an attention mechanism that looks back at all previous tokens to decide what to output next. Without optimization, generating the 100th token requires recalculating attention scores for all 99 previous tokens. This is computationally expensive and scales poorly as sequences get longer.
Key-Value (KV) Caching is a technique that stores the intermediate "key" and "value" vectors from these attention layers. Instead of recomputing them for every new token, the model retrieves them from memory. This changes the computational complexity of generating a single token from O(n²) to O(n), where n is the sequence length. It is a massive win for speed.
However, this speed comes at a cost: memory. For every token in the context window, the model must store its key and value vectors across every layer and every attention head. The formula for the cache size per token is roughly 2 * Layers * Heads * Head_Dimension. For a large model like Llama-3-70B, this adds up fast. If you are running multiple users simultaneously, the GPU's VRAM fills up with cached history rather than active computation. Managing this memory efficiently is no longer optional; it is the primary constraint on throughput.
Why Static Batching Fails
Before continuous batching, most serving systems used static batching. You would set a batch size, say 32, and wait until you had 32 requests. Then you would run them. The problem is variance. User prompts vary wildly in length. Some are questions, some are documents. Some outputs are short answers, others are long stories.
In a static batch, the entire group waits for the longest request to finish. If 31 requests generate 100 tokens and one generates 1,000 tokens, the GPU spends 90% of its time computing only one useful stream while 31 slots sit idle. This leads to terrible GPU utilization, often dropping below 30% in production environments. Since GPUs are expensive, low utilization means high cost per token. You are effectively burning money on idle silicon.
How Continuous Batching Works
Continuous batching, also known as iteration-level scheduling, changes the unit of work from the "request" to the "token." The scheduler runs at every step of the generation process. Here is the loop:
- Admit: New requests enter the queue whenever there is free memory.
- Decode: The GPU processes one token for every active request in the current batch.
- Evict: As soon as a request hits its stop condition (like an end-of-sequence token), it is removed from the batch immediately.
- Refill: A new waiting request takes the vacated spot for the very next token step.
This dynamic approach ensures that the batch size fluctuates based on actual demand and memory availability, not arbitrary limits. If many short requests come in, the batch grows. If long requests dominate, it shrinks. The goal is always to keep the GPU cores busy. By eliminating the "wait for the slowest runner" penalty, systems using continuous batching report throughput improvements ranging from 2x to over 20x compared to static baselines.
PagedAttention: Solving the Memory Fragmentation Problem
Continuous batching introduces a new challenge: memory fragmentation. If you allocate a contiguous block of memory for each request's KV cache, you waste space. Why? Because you don't know exactly how long a user's response will be. You might reserve space for 4,000 tokens, but the user only uses 500. The remaining 3,500 slots are wasted because other requests cannot use that fragmented gap.
PagedAttention, introduced by the team behind vLLM, solves this by borrowing a concept from operating systems: virtual memory paging. Instead of storing KV caches in contiguous blocks, PagedAttention splits them into fixed-size pages (e.g., 16 tokens per page). These pages can be stored anywhere in GPU memory. The system maintains a table mapping logical token positions to physical memory addresses.
| Feature | Static Batching | Continuous Batching + PagedAttention |
|---|---|---|
| Scheduling Granularity | Request Level | Token Level |
| Memory Allocation | Contiguous, Pre-reserved | Paged, On-demand |
| Handling Variable Lengths | Poor (Idle GPU) | Excellent (High Utilization) |
| Fragmentation | High Internal Fragmentation | Negligible |
| Throughput Gain | Baseline | 2x - 20x Improvement |
This allows the system to share memory blocks between requests if they have common prefixes (prefix caching) and eliminates internal fragmentation. You only pay for the memory you actually use, allowing you to fit significantly more concurrent requests onto a single GPU.
Chunked Prefill: Balancing Latency and Throughput
There is a catch with continuous batching. When a new request arrives, it has a prompt that needs to be processed before any tokens can be generated. This is called the "prefill" phase. Processing a long prompt (e.g., a 10,000-token document) takes a long time. If the scheduler stops everything to process this prefill, the existing requests waiting for their next token suffer increased latency. This is known as Time To First Token (TTFT) degradation.
To fix this, modern systems implement Chunked Prefill. Instead of processing the entire prompt in one giant step, the system breaks it into smaller chunks. These chunks are interleaved with the decoding steps of ongoing requests. The GPU alternates between processing small bits of the new prompt and generating tokens for existing conversations. This smooths out the load and prevents long prompts from blocking the pipeline, keeping latency predictable even under heavy load.
Implementation Strategies and Tools
If you are building your own inference stack, you likely won't write the CUDA kernels for PagedAttention from scratch. Several open-source frameworks have abstracted these complexities:
- vLLM: The pioneer of PagedAttention. It offers high throughput and is widely adopted for production deployments requiring maximum GPU efficiency.
- TensorRT-LLM: NVIDIA’s optimized runtime. It supports continuous batching and paged attention, often providing lower latency for specific hardware configurations due to deep integration with CUDA.
- Hugging Face TGI (Text Generation Inference): Also implements continuous batching and flash attention, making it a strong choice for teams already embedded in the Hugging Face ecosystem.
When choosing between these, consider your operational constraints. vLLM is excellent for maximizing throughput on standard GPUs. TensorRT-LLM might offer better performance if you are willing to deal with compilation overhead and strict version compatibility. Managed providers (like Anyscale or Together AI) handle these optimizations for you, but understanding the underlying mechanics helps you diagnose issues when latency spikes occur.
Common Pitfalls and Optimization Tips
Even with continuous batching enabled, you can still hit bottlenecks. Here are three things to watch out for:
- Memory Pressure: If your prompts are extremely long, the KV cache will consume all available VRAM. Monitor your GPU memory usage. If you see frequent evictions or errors, consider reducing the maximum context length or limiting the number of concurrent requests.
- Cold Starts: Loading the model weights into GPU memory takes time. Ensure your infrastructure keeps models warm or uses fast loading techniques if you scale instances dynamically.
- Batch Size Tuning: While continuous batching is dynamic, setting a hard cap on the maximum batch size can help stabilize latency. Start with a moderate limit and increase it only if TTFT remains within acceptable bounds.
Another advanced technique gaining traction is prefix sharing. If multiple users ask questions about the same document, their initial prompt tokens are identical. Systems like BatchLLM build a global prefix tree to reuse the KV cache for these shared prefixes. This saves both computation time and memory, further boosting throughput for specific use cases like RAG (Retrieval-Augmented Generation).
Does KV caching increase memory usage?
Yes, KV caching trades memory for speed. It stores key and value vectors for every token in the context window across all layers. This linearly increases memory consumption relative to sequence length, which can become a bottleneck for long-context models if not managed via techniques like PagedAttention.
What is the difference between static and continuous batching?
Static batching collects requests into a fixed group and processes them until the longest request finishes, leaving GPU resources idle for shorter requests. Continuous batching operates at the token level, removing finished requests immediately and admitting new ones, ensuring the GPU stays fully utilized regardless of request length variance.
How does PagedAttention improve efficiency?
PagedAttention divides the KV cache into fixed-size pages that can be stored non-contiguously in GPU memory. This eliminates internal fragmentation caused by reserving large contiguous blocks for variable-length responses, allowing more concurrent requests to fit into the same amount of VRAM.
What is Chunked Prefill?
Chunked Prefill is a technique where long input prompts are split into smaller segments. These segments are processed interleaved with the decoding steps of ongoing requests. This prevents long prompts from blocking the generation pipeline, thereby reducing Time To First Token (TTFT) latency for other users.
Which frameworks support continuous batching?
Major open-source frameworks supporting continuous batching include vLLM, TensorRT-LLM, and Hugging Face Text Generation Inference (TGI). These tools implement token-level scheduling and memory management optimizations like PagedAttention to maximize GPU throughput.