Chunked prefill

Take Llama-3-8B on one H100 SXM. Thirty-two requests are decoding, each around 2k tokens of context. A thirty-third request arrives with a 32,768-token prompt — a code file, a long document, a stuffed RAG context. The scheduler admits it. What happens to the thirty-two decoders?

Work the two iteration costs. Symbols follow FORMULAS: $L=32$, $h=32$, $h_{kv}=8$, $d_h=128$, $P = 8.03\times10^{9}$, $b = 2$ bytes (bf16). Two hardware numbers are assumed and stated up front, because everything below scales with them: an achieved $\pi_{\text{eff}} = 500$ TFLOP/s in bf16 (about half of the H100's 989 TFLOP/s dense peak) and an achieved $\beta_{\text{eff}} = 2.6$ TB/s (about 78% of HBM3's 3.35 TB/s) — measure yours in §0.4 and substitute.

Cost of the prefill iteration (derived). Prefill FLOPs for a prompt of $S$ tokens are the weight GEMMs plus causal attention:

The second term is the $\text{FLOPs}_{\text{tok}}$ attention term of FORMULAS summed over positions $1..S$, which halves the $4$ to a $2$. At $S = 32768$: $2PS = 5.26\times10^{14} = 526$ TFLOP, and $2\cdot32\cdot32\cdot128\cdot S^2 = 2.82\times10^{14} = 281$ TFLOP. Total 807 TFLOP, so $t_{\text{prefill}} = 807/500 = \mathbf{1.61\ s}$.

Cost of a decode iteration (derived). One decode step must stream all weights plus every resident KV element once. Per-token KV bytes across all layers are $2 \cdot L \cdot h_{kv} \cdot d_h \cdot b = 2\cdot32\cdot8\cdot128\cdot2 = 131{,}072$ B — exactly 128 KiB per token, a number worth memorising for this model. Weights are $2P = 16.06$ GB; the 32 sequences at 2048 tokens hold $65{,}536$ tokens $\times$ 128 KiB $= 8.59$ GB. Total 24.65 GB, so $t_{\text{decode}} = 24.65/2600 = \mathbf{9.5\ ms}$.

That is the symptom. Every decoding request sees one inter-token gap of 1.61 s instead of 9.5 ms — a 170× spike — and the maximum gap exceeds 50 ms by a factor of 32. Whether p99 breaches depends on the fraction of affected gaps in the measurement window. Nothing is broken and nothing is misconfigured; one long prompt simply owns the GPU for a second and a half. Continuous batching does not help: it removed the wait for a batch to finish, not the cost of a single iteration.

Stop treating a prefill as an atomic unit of work. A request in flight is nothing but a pair (num_computed_tokens, num_tokens), and the KV cache is the record of how far along it is: if the cache holds K and V for positions $[0, k)$, the next forward pass can pick up at position $k$ with no state beyond that cache. The scheduler is therefore free to hand the model any number of new positions per iteration. Chunked prefill is the policy of handing it a bounded number.

Two things in Figure 1 deserve to be uncomfortable. Total prefill wall-clock is essentially unchanged — seventeen chunks summing to the same 807 TFLOP still take 1.61 s, plus about 56 ms for the decode tokens now riding along — because chunking does not change the FLOP count. And ITL does not return to 9.5 ms; it lands at one chunk's compute time, which is not even constant: chunk 17 attends to 32k keys where chunk 1 attended to 2k, so it costs 134 ms against the first chunk's 70 ms. Chunked prefill converts an unbounded latency spike into a bounded one whose size you choose, and that choice is the whole engineering content of this chapter.

Why splitting a prefill is correct, for free

The non-obvious part is that no compensation is needed. Consider query position $i$ in chunk $j$, which covers positions $[jc, (j{+}1)c)$ for chunk size $c$. Causal attention says $i$ attends only to keys at positions $0 \ldots i$. Positions $< jc$ had their K and V written to the cache by chunks $0 \ldots j{-}1$; positions in $[jc, i]$ are in the current chunk's own K and V. So the softmax denominator for position $i$ ranges over exactly the same key set it would have in a single 32k forward pass. No rescaling, no stitching of partial softmax statistics, no recomputation. The causal mask is what makes it free — a bidirectional encoder cannot be chunked this way, which is why vLLM disables the feature for encoder-decoder models.

    def __post_init__(self, max_model_len: int, is_encoder_decoder: bool) -> None:
        if is_encoder_decoder:
            # Chunked prefill should be disabled for encoder-decoder models.
            self.disable_chunked_mm_input = True
            self.enable_chunked_prefill = False
            self.long_prefill_token_threshold = 0
            logger.info(
                "Encoder-decoder models do not support chunked prefill nor"
                " prefix caching; disabling both."
            )

The one piece of bookkeeping that is required: a non-final chunk's last position is not the end of the prompt, so the token sampled there is garbage. vLLM samples it anyway and discards it, which is cheaper than special-casing the sampler. The two blocks below are the V1 model runner, whose NumPy version of the predicate is the readable one; the V2 runner a dense model actually gets computes the same fact as is_prefilling_np = num_computed_prefill_tokens_np < prefill_len_np (vllm/v1/worker/gpu/model_runner.py:L1089) and carries it through the batch state (§11.4):

        use_spec_decode = len(scheduler_output.scheduled_spec_decode_tokens) > 0
        if not use_spec_decode:
            # NOTE(woosuk): Due to chunked prefills, the batch may contain
            # partial requests. While we should not sample any token
            # from these partial requests, we do so for simplicity.
            # We will ignore the sampled tokens from the partial requests.
            # TODO: Support prompt logprobs.
            logits_indices = query_start_loc[1:] - 1
            spec_decode_metadata = None
            num_sampled_tokens = np.ones(num_reqs, dtype=np.int32)

The discard is a one-line predicate — partial means sequence length still short of prompt:

        # Record which requests should not be sampled,
        # so that we could clear the sampled tokens before returning
        self.discard_request_mask.np[:num_reqs] = (
            self.optimistic_seq_lens_cpu[:num_reqs].numpy() < num_tokens_np
        )
        self.discard_request_mask.copy_to_gpu(num_reqs)

The token budget, and why one slot is not one unit of work

Neither engine has a "chunk size" in the primitive sense. Both schedule against a per-iteration token budget: an integer count of query positions the next forward pass may contain. Requests draw from it in priority order until it is empty. Chunked prefill is simply what happens when a prefill request wants more positions than the budget has left — it takes what is left and comes back next iteration.

Here is the asymmetry that causes most tuning confusion. A budget slot is a row of the batched GEMM, and prefill and decode tokens each occupy one row for the same $2P$ FLOPs of weight math. But attention cost is proportional to the number of keys the query attends to: a decode token at 32k context reads 32,768 keys while a fresh prefill token at position 500 reads 500. And a decode-only pass has too few rows to cover the 16.06 GB weight stream, so it is bandwidth-bound and its per-slot cost explodes. Price a few iteration shapes:

One budget slot buys 33 µs of prefill or 2 ms of long-context decode. The budget bounds batch width, not iteration duration, and it proxies for duration only when the batch is prefill-dominated. That is why "just lower max_num_batched_tokens until ITL is fine" stops working once resident contexts get long: the decode half of the iteration, which the budget barely constrains, becomes the term that sets ITL.

Why chunk size cannot go to 1

Chunking is FLOP-neutral. It is emphatically not byte-neutral. Split $S$ tokens into $N = S/c$ chunks and two costs appear, both scaling as $1/c$:

where $k = 2 L h_{kv} d_h b$ is the per-token KV footprint (128 KiB here). Every chunk is a separate set of kernel launches, so every chunk re-streams the full 16.06 GB of weights; and chunk $j$ must read the K and V that chunks $0 \ldots j{-}1$ wrote, which is $k \cdot jc$ bytes. At 32k the KV term is $kS/2 = 2.15$ GB against a weight term of 16.06 GB, so weight re-streaming dominates; the two cross at $S = 2Pb_w/k \approx 245$k tokens, and at 128k the KV re-read is already 53% of the weight cost.

The transition between rows 4 and 5 is the rule to remember. A chunk's weight GEMMs have arithmetic intensity $I \approx c$ FLOP/byte — each weight byte is read once and used for $c$ rows — and the roofline ridge point is $I^{*} = \pi/\beta$: $989/3.35 = 295$ on H100 spec sheets, $500/2.6 = 192$ on the achieved numbers assumed here. A chunk stops paying for its own weight stream at $c \approx 200$. Below that you buy lower ITL with pure waste.

vLLM: clamp against a shared budget

In vLLM V1 there is no prefill phase. Scheduler.schedule() opens with a comment that is the clearest statement of the design in either codebase:

    def schedule(self, throttle_prefills: bool = False) -> SchedulerOutput:
        self.current_step += 1
        # NOTE(woosuk) on the scheduling algorithm:
        # There's no "decoding phase" nor "prefill phase" in the scheduler.
        # Each request just has the num_computed_tokens and
        # num_tokens_with_spec. num_tokens_with_spec =
        # len(prompt_token_ids) + len(output_token_ids) + len(spec_token_ids).
        # At each step, the scheduler tries to assign tokens to the requests
        # so that each request's num_computed_tokens can catch up its
        # num_tokens_with_spec. This is general enough to cover
        # chunked prefills, prefix caching, speculative decoding,
        # and the "jump decoding" optimization in the future.

The budget is initialised once per call and drawn down by the running queue before the waiting queue is touched — that ordering is what "decode is prioritised" means concretely:

        num_scheduled_tokens: dict[str, int] = {}
        token_budget = self.max_num_scheduled_tokens
        spec = self.vllm_config.speculative_config
        draft_slots = spec.max_num_new_slots_for_drafting if spec is not None else 0
        input_budget = self.scheduler_config.max_num_batched_tokens

And this — five lines in the waiting-queue loop — is chunked prefill. There is no other implementation of it in vLLM:

                    request_token_budget = min(token_budget, input_budget - draft_slots)
                    # Number of tokens to be scheduled.
                    # We use `request.num_tokens` instead of
                    # `request.num_prompt_tokens` to consider the resumed
                    # requests, which have output tokens.
                    num_new_tokens = request.num_tokens - num_computed_tokens

                    threshold = self.scheduler_config.long_prefill_token_threshold
                    if 0 < threshold < num_new_tokens:
                        num_new_tokens = threshold

                    # chunked prefill has to be enabled explicitly to allow
                    # pooling requests to be chunked
                    if (
                        not self.scheduler_config.enable_chunked_prefill
                        and num_new_tokens > request_token_budget
                    ):
                        # If chunked_prefill is disabled,
                        # we can stop the scheduling here.
                        break

                    num_new_tokens = min(num_new_tokens, request_token_budget)
                    assert num_new_tokens > 0

Read the disabled path carefully: with enable_chunked_prefill=False the scheduler does not run a smaller prefill, it breaks out of admission entirely and the request waits for an iteration with a big enough budget — §1's head-of-line block, restored on purpose.

An in-flight prefill returns to the running queue, where the identical clamp appears at vllm/v1/core/sched/scheduler.py:L566-L575. One predicate then marks the request as mid-prefill, which downstream code uses to suppress structured-output advancement and speculative drafting:

        num_scheduled_tokens = scheduler_output.num_scheduled_tokens
        for req_id, num_scheduled_token in num_scheduled_tokens.items():
            request = self.requests[req_id]
            request.num_computed_tokens += num_scheduled_token
            request.num_in_flight_tokens += num_scheduled_token
            if self.defer_block_free:
                # Record the in-flight step, to fence deferred block freeing.
                request.last_sched_seq = self.sched_step_seq
            request.is_prefill_chunk = request.num_computed_tokens < (
                request.num_tokens + request.num_output_placeholders
            )

Defaults at a556f3f. enable_chunked_prefill is True in SchedulerConfig (vllm/config/scheduler.py:L74-L76), resolved per model from model_config.is_chunked_prefill_supported (vllm/engine/arg_utils.py:L2678-L2683). The budget default is not the DEFAULT_MAX_NUM_BATCHED_TOKENS = 2048 class constant at vllm/config/scheduler.py:L41-L43, whose own docstring calls it "mainly for convenience when testing". The real default is chosen by device memory and usage context:

        # NOTE(Kuntai): Setting large `max_num_batched_tokens` for A100 reduces
        # throughput, see PR #17885 for more details.
        # So here we do an extra device name check to prevent such regression.
        if device_memory >= 160 * GiB_bytes:
            # for GPUs like B200/B300 with >= 160GB memory, use the largest defaults
            default_max_num_batched_tokens = {
                UsageContext.LLM_CLASS: 16384,
                UsageContext.OPENAI_API_SERVER: 16384,
            }
            default_max_num_seqs = {
                UsageContext.LLM_CLASS: 1024,
                UsageContext.OPENAI_API_SERVER: 1024,
            }
        elif device_memory >= 70 * GiB_bytes and "a100" not in device_name:
            # For GPUs like H100 and H200, use larger offline defaults.
            default_max_num_batched_tokens = {
                UsageContext.LLM_CLASS: 16384,
                UsageContext.OPENAI_API_SERVER: 8192,
            }

So an H100 serving over the OpenAI-compatible endpoint defaults to a budget of 8192, the same H100 driven from the offline LLM class defaults to 16384, and anything below 70 GB — or any A100, by explicit name check — falls to 2048 for the API server and 8192 offline (vllm/engine/arg_utils.py:L2629-L2639).

long_prefill_token_threshold is a second, separate control. It defaults to 0, meaning disabled (vllm/config/scheduler.py:L70-L72), and it is applied before the budget clamp in both loops. The difference matters: the budget clamp is opportunistic, so a 32k prompt arriving at an idle server with a 16384 budget gets all 16384 tokens and produces a 400 ms iteration. The threshold is absolute — set it to 2048 and no request ever gets more than 2048 prefill positions in one pass, however empty the budget is. It is the knob for "bound my worst-case ITL" as distinct from "bound my batch width".

SGLang: one in-flight chunked request, carried by hand

SGLang reaches the same outcome through a visibly different structure. Its loop picks a batch kind per iteration rather than filling one unified batch:

        if new_batch is not None:
            # Run prefill first if possible
            ret = new_batch
        else:
            # Run decode (skip for prefill-only batches)
            if not running_batch.is_empty() and not running_batch.is_prefill_only:
                running_batch = self.update_running_batch(running_batch)
                ret = running_batch if not running_batch.is_empty() else None
            else:

The budget lives in PrefillAdder, which holds two counters: rem_input_tokens (from --max-prefill-tokens, default 16384) and rem_chunk_tokens (from --chunked-prefill-size). None for the latter means chunking is off. Decode tokens that will ride along are subtracted from both up front:

        self.rem_input_tokens = rem_input_tokens - num_mixed_decode_tokens
        self.rem_chunk_tokens = rem_chunk_tokens
        self.dllm_config = dllm_config

        if self.dllm_config is not None:
            self._init_dllm_meta(dllm_config)

        if self.rem_chunk_tokens is not None:
            self.rem_chunk_tokens -= num_mixed_decode_tokens
        self.rem_total_token_offset = num_mixed_decode_tokens

The split itself is the else branch of add_one_req. Note the page alignment — SGLang truncates the chunk down to a multiple of page_size in two places, once on the chunk length and once on the resulting absolute prefix length, so that a chunk boundary is always a KV page boundary:

            else:
                # Make sure at least one page is available
                trunc_len = chunk_tokens_limit // self.page_size * self.page_size

                if trunc_len <= 0:
                    return AddReqResult.OTHER
# ...
                now_input_len = trunc_len + len(req.prefix_indices)
                now_input_len = now_input_len // self.page_size * self.page_size
                trunc_len = now_input_len - len(req.prefix_indices)

                if trunc_len <= 0:
                    return AddReqResult.OTHER
# ...
                # Chunked prefill
                req.set_extend_range(
                    len(req.prefix_indices), len(req.prefix_indices) + trunc_len
                )

                self.can_run_list.append(req)
                self.new_chunked_req = req

And here is the real design difference. SGLang tracks exactly one in-flight chunked request on the scheduler object, and the invariant is enforced by an assertion:

        if adder.new_chunked_req is not None:
            # Update chunked prefill
            assert self.chunked_req is None
            self.chunked_req = adder.new_chunked_req

        if self.chunked_req is not None:
            self.chunked_req.inflight_middle_chunks += 1

Next iteration, that request is re-entered explicitly at the head of the adder, before any waiting request is considered:

        if self.chunked_req is not None:
            self.chunked_req.init_next_round_input()
            self.chunked_req = adder.add_chunked_req(self.chunked_req)

add_chunked_req returns the request if it is still truncated and None when the last chunk fits, which is how the field clears itself (python/sglang/srt/managers/schedule_policy.py:L1036-L1053).

Why the difference? A mid-chunk request holds a lock on its radix-cache tree node, and permitting several simultaneous partial prefills would multiply the pinned, half-written subtrees the cache has to reason about; the single-slot invariant keeps that analysable. The cost is that a second long prompt cannot start until the first finishes chunking, so under a burst of long prompts SGLang serialises where vLLM interleaves — and vLLM pays for that flexibility with an _inflight_prefills set and per-request state on every path that cares.

Defaults at 7d89325. chunked_prefill_size is None in ServerArgs (python/sglang/srt/server_args.py:L811-L815) and auto-tuned from GPU memory, together with the decode CUDA-graph batch size, in _handle_gpu_memory_settings:

The two split values are tp_size < 4 versus otherwise. The same docstring explains why this is not purely a latency knob: mem_fraction_static — SGLang's single knob for how much device memory the weights and KV pool may claim, owned by §2.6 — is derived from it, since "the activation memory is proportional to the chunked_prefill_size", via reserved_mem = chunked_prefill_size * 1.5 + max_bs * 2 in the heuristic's MiB-scale accounting, not GB. Raising the chunk size on SGLang silently shrinks the KV pool.

Piggybacking: a chunk and a decode in one pass

vLLM mixes unconditionally — the running loop and the waiting loop write into the same num_scheduled_tokens dict, so a chunk and thirty-two decodes become one forward pass with 2048 rows. SGLang gates it behind a flag that is off by default:

    enable_mixed_chunk: A[
        bool,
        "Enabling mixing prefill and decode in a batch when using chunked prefill.",
        NS("schedule"),
    ] = False

When on, ScheduleBatch.mix_with_running sets ForwardMode.MIXED and appends each running request as a one-token extend (python/sglang/srt/managers/schedule_batch.py:L2758-L2788). Price the 32 decodes both ways (derived): standalone, 16.06 GB of weights plus 8.59 GB of KV, i.e. 9.5 ms. Folded into a 2016-token chunk the prefill has already paid for the weight stream, so their marginal cost is 32 extra GEMM rows (about 1.0 ms) plus their own 8.59 GB of KV attention (3.3 ms) — 4.3 ms instead of 9.5 ms, a 2.2× reduction. The saving shrinks as the decode batch grows, because a wide decode batch is compute-bound on its own and no longer needs to borrow the weight read.

One 32,768-token request through vLLM V1 at a556f3f, budget 2048, 32 requests already decoding. Function names in call order.

Seventeen iterations, not sixteen, because the decodes take 32 slots off the top of each: $\lceil 32768/2016 \rceil = 17$. That is also why max_num_batched_tokens < max_num_seqs is rejected; equality is allowed — at max_num_seqs = 2048 with a budget of 2048, decodes would take every slot and no prefill would ever advance.

Two debugging entry points. vLLM logs its budget once at startup: Chunked prefill is enabled with max_num_batched_tokens=%d. (vllm/config/scheduler.py:L241-L246) — if that number is not what you passed, something in _set_default_max_num_seqs_and_batched_tokens_args clamped it, most likely min(max_num_seqs * max_model_len, ...) at vllm/engine/arg_utils.py:L2851-L2856. On SGLang, an in-flight chunk increments inflight_middle_chunks every iteration (python/sglang/srt/managers/scheduler.py:L3466-L3467); a counter that keeps climbing means chunks are being truncated far below the configured size, usually by KV pressure in add_chunked_req.

Pick a chunk size from an ITL SLO, don't guess it. The arithmetic is one line: a chunk of $c$ tokens costs roughly $2Pc/\pi_{\text{eff}}$ once $c$ is above the ridge point, so

For a 60 ms p99 ITL target on Llama-3-8B/H100 with $t_{\text{decode}} = 9.5$ ms: $c \approx 500\times10^{12}\times0.0505 / 1.606\times10^{10} = 1572$, so 1536 after page alignment — which is why 2048 feels slightly coarse for a chat SLO and 512 leaves throughput on the table. This is a lower bound on chunk cost: it counts only the weight GEMMs, and the attention term adds $4Lh d_h c \cdot s_{\text{prefix}}/\pi_{\text{eff}}$ on top, which is what makes the last chunk of a long prompt roughly twice the first. Budget for the most expensive full chunk, not the average; a shorter remainder can be cheaper than the preceding full chunk.

# vLLM: bound the batch width, then bound the worst-case chunk independently
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 65536 --max-num-seqs 64 \
  --max-num-batched-tokens 2048 \
  --long-prefill-token-threshold 1536

# vLLM: the pathological baseline this chapter opened on
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 65536 --no-enable-chunked-prefill \
  --max-num-batched-tokens 65536

# SGLang: chunk size and the mixed batch are separate decisions
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
  --context-length 65536 --chunked-prefill-size 2048 --enable-mixed-chunk

# SGLang: chunking off
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
  --context-length 65536 --chunked-prefill-size -1

The workload that exposes the effect is one long prompt injected into a stream of short ones — exactly what Lab 05 runs. Watch p99 ITL, p99 TTFT, and output tokens/s together; any two can be made to look good by sacrificing the third. Methodology (open-loop arrivals, warmup, goodput rather than throughput) is §10.3.

• SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — the original chunk-plus-piggyback formulation. Linked directly from vLLM's own tuning guide at docs/configuration/optimization.md:L80.
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI '24) — the stall-free-batching scheduler and the ITL/throughput curves this chapter's trade-off section restates. Also linked from the same vLLM doc line.
• docs/configuration/optimization.md:L49-L81 in the vLLM tree — the maintainers' own tuning advice, including "smaller values (e.g. 2048) achieve better ITL" and "for optimal throughput, set max_num_batched_tokens > 8192".
• vLLM PR #17885, named in the comment at vllm/engine/arg_utils.py:L2606-L2608, is the source of the A100-specific default. Worth reading for how a "bigger is better" default was found to regress one card.
• Next: §1.6 takes the same problem to its conclusion — if prefill and decode want different schedulers, give them different machines. Chunked prefill is the single-node answer; disaggregation is the multi-node one, and they are not mutually exclusive.