Continuous Batching in LLM Inference: What AI Product Managers Need to Know

The Problem: How Static Batching Wastes GPU Time

To understand continuous batching, you first need to understand the problem it solves. LLM inference is autoregressive: the model generates one token at a time, feeding each token back as input to generate the next. This means a single response of 500 tokens requires 500 forward passes through the model.

The naive approach to serving multiple users is static batching: group several requests into a batch, process all of them together, and release the batch when every sequence has finished. This approach has a critical inefficiency. If user A asks a short question (20 tokens of output) and user B asks a complex question (400 tokens of output), the GPU still holds user A's slot for the full 400-step duration of user B's generation, even though user A's request finished at step 20. The GPU is doing nothing useful for that slot for 380 steps.

Output length is unpredictable

Unlike a database query or an image resize, you cannot know in advance how many tokens an LLM will generate. Different users send different prompts, ask for different amounts of detail, and get responses that vary by 10x or more in length. Static batching means the longest response in the batch determines when all slots free up.

GPU memory is expensive and idle

A GPU slot held open for a finished sequence is pure waste. GPU time is the dominant cost in LLM inference. Idle GPU capacity during the wait-for-the-longest-sequence period is money and throughput thrown away.

Latency spikes under load

When a batch fills up and new requests arrive, those requests must wait for the entire current batch to complete before they are processed. A user who arrives just as a slow batch starts could wait the full duration of the longest sequence in that batch before their request even begins. At scale, this creates unpredictable tail latency.

How Continuous Batching Works

Continuous batching, first described in the 2022 paper "Orca: A Distributed Serving System for Transformer-Based Generative Models" from the University of Washington, solves the static batching problem by scheduling at the iteration level rather than the request level.

Instead of grouping requests into fixed batches and releasing all slots together, the inference server inspects the state of every sequence after each forward pass (each token generation step). When a sequence finishes (hits the end-of-sequence token or reaches the maximum length), its slot is immediately freed and filled with the next waiting request. The batch composition changes dynamically at every iteration.

What happens at each step

After each forward pass, the server checks every sequence in the batch. Finished sequences are evicted. New sequences from the queue are inserted to fill those slots. The batch for the next forward pass is assembled from ongoing sequences and newly inserted ones.

The KV cache connection

Continuous batching works because modern inference frameworks maintain a KV (key-value) cache that stores the attention computations for each token already generated. New sequences can join mid-batch because they start from scratch in the KV cache, independent of other sequences.

GPU utilization improvement

In practice, continuous batching increases GPU utilization from roughly 20-40% (for static batching on variable-length workloads) to 70-90%. This is why inference costs dropped dramatically between 2023 and 2025 without hardware changes: the same GPU simply served more users per hour.

The role of paged attention

PagedAttention (from vLLM, also 2022) is a complementary technique that manages KV cache memory more efficiently by storing it in non-contiguous blocks, similar to virtual memory paging in an OS. Most production serving systems combine continuous batching with paged attention.

The inference frameworks you are most likely to encounter in production, vLLM, TGI (Text Generation Inference from Hugging Face), TensorRT-LLM from NVIDIA, and SGLang, all implement continuous batching as a core feature. When an API provider quotes you a price per million tokens, their cost structure is built on this optimization.

The Latency vs Throughput Tradeoff

Continuous batching improves throughput substantially, but it creates a specific latency tradeoff that product managers need to understand. When the server is fully loaded and every GPU slot is occupied, a new request must wait for an existing sequence to finish before it can be inserted. The time-to-first-token (TTFT) for a new request under high load is therefore determined by the length of the shortest remaining sequence in the batch, not by a fixed queue time.

Time to First Token (TTFT)

What it measures: How long from request submission until the first output token arrives. Affected by queue depth and the number of prefill (prompt processing) tokens.

PM implication: For interactive chat features, TTFT is the primary latency metric users perceive. Target under 500ms for conversational interfaces. For document analysis with long prompts, TTFT can be seconds and users accept it if the total experience is fast.

Time Between Tokens (TBT) / Inter-token Latency

What it measures: How long between each successive output token. Determined by the forward pass time, which scales with model size and batch size.

PM implication: TBT determines whether streaming responses feel natural. At 50ms per token (20 tokens/second), streaming text feels like a fast typist. At 200ms per token (5 tokens/second), it feels slow. Most modern serving setups achieve 20-80 tokens/second per request depending on load.

Throughput (tokens per second per GPU)

What it measures: The aggregate output rate of the server, across all concurrent users. Continuous batching maximizes this by minimizing idle GPU time.

PM implication: Throughput determines your cost per million tokens. Higher throughput per GPU = lower serving cost. This is the metric your ML infra team optimizes. As a PM, you need to know what throughput your SLA requires and what it costs.

Build Technical Depth That Changes How You Ship

The AI PM Masterclass covers LLM inference, evaluation, and cost architecture at the level that actually improves your product decisions. Taught live by a Salesforce Sr. Director PM.

What This Means for Your Product Architecture Decisions

Understanding continuous batching changes how you think about several common AI product decisions.

Setting latency SLAs for AI features

Under continuous batching, latency is not constant. It varies with server load. An SLA that works at 10% load may break at 80% load if your TTFT budget is tight. When you commit to a latency SLA, you need to know the load at which the SLA holds and what autoscaling triggers keep it there. 'P99 under 2 seconds at 500 concurrent users' is a testable SLA. 'Fast responses' is not.

Choosing between streaming and non-streaming responses

Streaming (receiving tokens as they are generated) and non-streaming (waiting for the full response) have different UX and infrastructure tradeoffs. Streaming reduces perceived latency for long responses and is better for interactive chat. Non-streaming is simpler to implement and may be preferable for batch document processing. Continuous batching makes streaming more reliable under load because slots are released as sequences finish, reducing the queue depth for new requests.

Prompt engineering and output length

Shorter outputs generated faster means more requests served per GPU per hour. If your product prompts the model to always write detailed multi-paragraph responses when a single sentence would serve the user, you are paying a real cost in throughput and latency. Prompt design that minimizes unnecessary output tokens is both a UX improvement and a cost optimization.

Maximum output token limits

Setting a max_tokens parameter on your API calls limits how long any single sequence can run, which reduces tail latency under load. For use cases where occasional long outputs are acceptable (code generation, document drafting), a high max_tokens limit is appropriate. For use cases where consistent low latency matters more (real-time chat), a tighter limit flattens your latency distribution.

Self-hosted vs API provider tradeoffs

If you are consuming a managed API (OpenAI, Anthropic, Google), the provider handles continuous batching, autoscaling, and GPU fleet management. You see a per-token price that reflects their efficiency. If you are self-hosting, you are responsible for selecting and configuring the inference framework. The operational complexity is real, but the unit economics at high volume can justify it if your traffic is consistent and predictable.

The Business Case: Capacity Planning and Cost Implications

Continuous batching is the primary reason that the cost per million tokens for frontier models dropped approximately 90% between 2023 and 2025, even as model quality improved. The GPU hardware improved somewhat, but the efficiency gains from better serving techniques, including continuous batching, paged attention, and speculative decoding, were the dominant factor.

Capacity planning implications

Your peak concurrent user count determines your GPU provisioning requirement. Under continuous batching, a single A100 GPU can handle roughly 50-200 concurrent users for a 70B parameter model at acceptable latency, depending on average output length. For a 7B model, the number is 5-20x higher. Use these rough figures to sanity-check your infra sizing.

Cost per request vs cost per token

Because continuous batching improves throughput, your effective cost per request is lower than a naive calculation would suggest. But your cost is still fundamentally driven by token volume. A feature that generates 2000-token responses costs roughly 10x more to serve than one that generates 200-token responses, regardless of how efficient the serving system is.

The fixed vs variable cost split

With continuous batching and autoscaling, your inference cost is largely variable (scales with usage). But GPU reservation and committed use discounts introduce fixed cost components. Understanding this split matters for your unit economics model and your pricing strategy.

Autoscaling lag and cold start

Adding GPU capacity takes time (seconds to minutes depending on the provider). Continuous batching improves steady-state efficiency but does not eliminate the need to provision capacity ahead of demand spikes. For products with predictable traffic patterns, scheduled scaling reduces the latency impact of traffic ramps.

The serving stack in 2026

The most widely deployed open-source continuous batching frameworks are vLLM (from the UC Berkeley Sky Computing Lab), TGI (Hugging Face), TensorRT-LLM (NVIDIA), and SGLang. Each has different tradeoffs in supported model architectures, quantization support, and operational maturity. All major managed inference providers (Together AI, Fireworks, Replicate, Groq, Cerebras) implement continuous batching internally. When you see 2000 tokens per second throughput claims from inference providers, continuous batching is a significant contributor to that number.