Prefill and decode are two different computers

Consider the Llama-3-8B tensor shapes on an H100, but use Llama-3.1-8B for the runnable 32k-context experiment: original Llama-3 has an 8k native limit. Assume at least 33 active-request slots. Thirty-two streams are decoding when a 32,768-token prompt arrives; an unchunked prefill can create a roughly two-second stall under the illustrative arithmetic model below.

Total throughput can increase while a long inter-token stall appears. Whether that stall changes p99 depends on how many gaps are affected and the measurement window; it is not automatically a 222-fold p99 increase.

The two numbers above are not two points on a spectrum. They come from two workloads with opposite bottlenecks, opposite ideal batch shapes, and opposite latency SLOs, and they are multiplexed onto the same silicon by the same scheduler loop. Once you see that split, every design decision in both vLLM and SGLang — chunking, budgets, preemption, disaggregation — reads as an answer to the same question: who gets this step?

A transformer forward pass is the same sequence of operators regardless of phase. What changes is one number: how many token rows go in. Prefill hands the block a matrix with $T$ rows — 2048, 8192, 32768 — and every weight matrix is used $T$ times per byte fetched. Decode hands the same block a matrix with one row per sequence — 32 rows for 32 sequences — and every weight matrix is used once or twice per byte fetched. The linear layers do not change; they change regime. A GEMM with $T=2048$ is a tensor-core problem. A GEMM with $T=1$ is a memory controller problem with a tensor core attached to it for decoration.

The attention operator is the exception and it moves the opposite way. In prefill, attention is a dense $T \times T$ causal score matrix — quadratic FLOPs, small state. In decode, attention is a single query row against the entire accumulated KV cache — linear FLOPs, and a read of every byte of cache the sequence owns. Prefill's attention grows the compute; decode's attention grows the traffic.

Take a single weight GEMM inside the block: activations $X$ of shape $[T, d]$ times weights $W$ of shape $[d, d']$, in bf16. Symbols as in FORMULAS: $T$ is the number of token rows in this forward pass, $d$ the hidden size, $d'$ the output width, and $b=2$ bytes per element.

The first byte term is the weight matrix, read once. The other two are the input and output activations. Arithmetic intensity, $I = \text{FLOPs} / \text{bytes}$:

The arithmetic intensity of a weight GEMM is, to first order, the number of tokens in the batch. That single line explains the entire chapter. Prefill has $T$ in the thousands. Decode has $T$ equal to the number of concurrent sequences, which is bounded by how much KV cache fits in VRAM (§2.1).

Now place both on the H100 roofline from §0.4. Using the NVIDIA SXM5 spec figures $\pi = 989$ TFLOP/s dense bf16 and $\beta = 3.35$ TB/s, the ridge point is $I^\star = \pi/\beta = 295$ FLOP/byte. Setting $I = I^\star$ with $d = d' = 4096$ and solving for $T$:

Below 345 tokens in flight you are on the bandwidth roof and the only levers that help are moving fewer bytes — quantization, MLA, KV compression. Above it you are on the compute roof and the only levers are more FLOP/s — better kernels, lower-precision tensor cores, more SMs. Prefill lives thousands of tokens above the ridge. Decode at any batch size a single H100 can hold lives below it.

Working the numbers for Llama-3-8B

Model FLOPs per token, from FORMULAS, are $2P + 4\,L\,h\,d_h\,s$ with $P = 8.03 \times 10^9$. KV bytes per token are $2 \, L \, h_{kv} \, d_h \, b = 2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072$ bytes, i.e. 128 KiB per token across all 32 layers. Weights in bf16 are 16.06 GB.

Read the last row against the roofline. Achievable rate is $\min(\pi, \beta I) = 3.35 \times 10^{12} \times 22.2 = 74$ TFLOP/s — 7.5% of the card's bf16 peak. That is not a kernel bug. It is what the roofline says a batch-32 decode step is allowed to reach on this hardware. The tensor cores are idle by construction, and no amount of kernel tuning changes it; only raising $T$ (bigger batch, or speculative decoding, which is §6.2) or lowering bytes does.

Read the first row the same way: $I = 2082$ is seven times past the ridge, so prefill is pinned to the compute roof and the useful lever is MFU, not bandwidth. Note also that at $T = 2048$ the weight GEMMs contribute 32.9 of the 34.0 TFLOP and attention only 1.1; by $T = 32768$ attention has grown to 282 of 808 TFLOP, because the causal score matrix is quadratic. Long-prompt prefill is not just "more prefill" — its cost curve bends upward.

Why they fight

And they share one GPU, one CUDA stream, one set of weights. The scheduler cannot run them concurrently on separate resources; on a single device it can only interleave them in time, or merge them into one forward pass. Those are the only two moves, and both engines make both.

The most instructive difference between the two engines is not an algorithm. It is that one of them has a first-class name for the phase and the other has deliberately deleted it.

SGLang: the phase is an enum

SGLang carries a ForwardMode on every batch, all the way from the scheduler into the attention backend. It is the cleanest statement of the phase split in either codebase.

class ForwardMode(IntEnum):
    # Extend a sequence. The KV cache of the beginning part of the sequence is already computed (e.g., system prompt).
    # It is also called "prefill" in common terminology.
    EXTEND = auto()
    # Decode one token.
    DECODE = auto()
    # Contains both EXTEND and DECODE when doing chunked prefill.
    MIXED = auto()
    # No sequence to forward. For data parallel attention, some workers will be IDLE if no sequence are allocated.
    IDLE = auto()

    # Used in speculative decoding: verify a batch in the target model.
    TARGET_VERIFY = auto()
    # Used in speculative decoding: extend a batch in the draft model.
    DRAFT_EXTEND_V2 = auto()

    # Used in disaggregated decode worker
    # Represent a batch of requests having their KV cache ready to start decoding
    PREBUILT = auto()

    # Split Prefill for PD multiplexing
    SPLIT_PREFILL = auto()

    # Used in dLLM
    DLLM_EXTEND = auto()

Three things worth extracting. First, the primary name is EXTEND, not PREFILL — because with prefix caching a "prefill" usually only computes the suffix that is not already in the radix tree, so extending a sequence by $n$ tokens is the general operation and a cold prefill is the special case $n = $ prompt length. Second, MIXED exists as its own mode: prefill and decode tokens in the same forward pass. Third, the phase is not a bookkeeping label — downstream code branches on it:

    def is_decode(self):
        return self == ForwardMode.DECODE

    def is_mixed(self):
        return self == ForwardMode.MIXED

# ...

    def is_cuda_graph(self):
        return (
            self == ForwardMode.DECODE
            or self == ForwardMode.TARGET_VERIFY
            or self == ForwardMode.IDLE
            or self == ForwardMode.DLLM_EXTEND
        )

is_cuda_graph() is the phase split leaking into the runtime. Decode steps have a fixed, small, enumerable shape, so they can be captured into CUDA graphs and replayed with almost no launch overhead; this particular decode-graph predicate excludes EXTEND; general or piecewise graph capture can still support prefill (§8.1). At 9.2 ms per decode step and dozens of kernel launches per layer, launch overhead is a real fraction of a decode step and generally a smaller fraction of a sufficiently large prefill step.

The scheduler loop consumes the phase as a hard branch. Each iteration asks for a prefill batch first; only if there is none does it run the decode batch:

        if new_batch is not None:
            # Run prefill first if possible
            ret = new_batch
        else:
            # Run decode (skip for prefill-only batches)
            if not running_batch.is_empty() and not running_batch.is_prefill_only:
                running_batch = self.update_running_batch(running_batch)
                ret = running_batch if not running_batch.is_empty() else None
            else:
                ret = None

That is the whole policy in nine lines, as of 7d89325: prefill has priority; decode is what runs when there is nothing to prefill. The consequence — a long prefill starves every decoding stream — is exactly the symptom in §1, and the mitigations are chunking and mixing, below.

vLLM V1: there is no phase

vLLM's V1 scheduler opens with a comment that is the single most quotable line in either repository on this topic:

    def schedule(self, throttle_prefills: bool = False) -> SchedulerOutput:
        self.current_step += 1
        # NOTE(woosuk) on the scheduling algorithm:
        # There's no "decoding phase" nor "prefill phase" in the scheduler.
        # Each request just has the num_computed_tokens and
        # num_tokens_with_spec. num_tokens_with_spec =
        # len(prompt_token_ids) + len(output_token_ids) + len(spec_token_ids).
        # At each step, the scheduler tries to assign tokens to the requests
        # so that each request's num_computed_tokens can catch up its
        # num_tokens_with_spec. This is general enough to cover
        # chunked prefills, prefix caching, speculative decoding,
        # and the "jump decoding" optimization in the future.

The scheduler's state per request is a pair of integers: how many tokens have been computed, and how many exist. Scheduling is closing that gap under a global token budget. The core of the running loop is:

            num_new_tokens = (
                request.num_tokens_with_spec
                + request.num_output_placeholders
                - request.num_computed_tokens
            )
            if 0 < self.scheduler_config.long_prefill_token_threshold < num_new_tokens:
                num_new_tokens = self.scheduler_config.long_prefill_token_threshold
            num_new_tokens = min(
                num_new_tokens, token_budget, input_budget - draft_slots

For a decoding request, num_tokens_with_spec - num_computed_tokens is 1 and the min is a no-op. For a request whose prompt has not been computed, it is thousands and the min clamps it to what is left of max_num_batched_tokens. Same code path, same variable, different magnitude. The phase is an emergent property of the number, not a flag.

The artefact handed to the worker says the same thing. SchedulerOutput has no phase field at all — it has a dictionary of token counts:

@dataclass
class SchedulerOutput:
    # list of the requests that are scheduled for the first time.
    # We cache the request's data in each worker process, so that we don't
    # need to re-send it every scheduling step.
    scheduled_new_reqs: list[NewRequestData]
    # list of the requests that have been scheduled before.
    # Since the request's data is already cached in the worker processes,
    # we only send the diff to minimize the communication cost.
    scheduled_cached_reqs: CachedRequestData

    # req_id -> num_scheduled_tokens
    # Number of tokens scheduled for each request.
    num_scheduled_tokens: dict[str, int]
    # Total number of tokens scheduled for all requests.
    # Equal to sum(num_scheduled_tokens.values())
    total_num_scheduled_tokens: int

A step where num_scheduled_tokens is {"a": 1, "b": 1, "c": 1} is a decode step. One where it is {"d": 2048} is a prefill step. One where it is {"a": 1, "b": 1, "d": 2046} is a mixed step. The scheduler never named any of them.

The runner then re-derives the phase when it needs it — for CUDA graph dispatch, since that is the one place the distinction is load-bearing. Both runners do; the V1 one below spells the predicate out, while the V2 runner that a dense model such as Llama-3-8B actually gets calls the shared get_uniform_decode_token_count (vllm/v1/worker/utils.py:L601-L607), which answers the same question by returning the per-request token count or None (§11.4 on which runner you are on):

    def _is_uniform_decode(
        max_num_scheduled_tokens: int,
        uniform_decode_query_len: int,
        num_tokens: int,
        num_reqs: int,
        force_uniform_decode: bool | None = None,
    ) -> bool:
        """
        Checks if it's a decode batch with same amount scheduled tokens
        across all requests.
        """
        return (
            (
                (max_num_scheduled_tokens == uniform_decode_query_len)
                and (num_tokens == max_num_scheduled_tokens * num_reqs)
            )
            if force_uniform_decode is None
            else force_uniform_decode
        )

Why the difference matters

Neither is obviously better, and they buy different things. vLLM's token-budget formulation makes chunked prefill, prefix caching and speculative decoding the same mechanism — every one of them is "this request has $n$ uncomputed tokens, schedule some of them" — which is what the NOTE claims and what the code delivers. The cost is that phase is invisible in the scheduler, so anything that genuinely needs to know (graph capture, backend metadata, DP synchronisation) must reconstruct it from token counts and can get it subtly wrong.

SGLang's explicit mode makes phase-specific behaviour trivial to express and to read, and gives speculative decoding and disaggregation their own first-class modes (TARGET_VERIFY, PREBUILT, SPLIT_PREFILL) rather than overloading a token count. The cost is combinatorial: nine modes and a dozen predicate helpers, and every new feature must decide which mode it is and update the predicates.

Both engines can put prefill and decode tokens in one forward pass. Why that is even possible is worth being precise about, because only attention cares about which request a token belongs to. Every other operator in a decoder block — the QKV projection, the output projection, the MLP, RMSNorm, the residual adds — is either a per-row pointwise function or a GEMM against a shared weight matrix. Both are invariant to how you group the rows. So you flatten: concatenate all scheduled tokens from all requests into one $[N_{\text{tokens}}, d]$ matrix, run the whole block on it, and hand attention a side-channel describing where each request's rows start and how long its cached prefix is.

In vLLM that flattening is the first thing _prepare_inputs does — quoting the V1 runner, whose NumPy version of the arithmetic is legible; the V2 runner does the identical flattening inside Triton kernels — and the worked example is in the comment:

        # Get request indices.
        # E.g., [2, 5, 3] -> [0, 0, 1, 1, 1, 1, 1, 2, 2, 2]
        req_indices = np.repeat(self.arange_np[:num_reqs], num_scheduled_tokens)

        # cu_num_tokens: [2, 5, 3] -> [2, 7, 10]
        # self.query_pos.np[:10]: [0, 1, 0, 1, 2, 3, 4, 0, 1, 2]
        cu_num_tokens = self._get_cumsum_and_arange(
            num_scheduled_tokens, self.query_pos.np
        )

Three requests scheduled for 2, 5 and 3 tokens become a flat run of 10 rows plus a cumulative end-offset array [2, 7, 10]; prepend zero to obtain query_start_loc=[0, 2, 7, 10] for the attention backend. Substitute [1, 1, 2046] and you have a mixed batch: two decodes and one prefill chunk, indistinguishable to every kernel except attention, which reads the offsets and the per-request sequence lengths and does the right thing per segment.

SGLang builds the same thing but has to say so explicitly, because its batches are typed. When mixed chunking is on, the decode batch is converted into an extend batch of length 1 per request and concatenated onto the prefill batch:

    def mix_with_running(self, running_batch: ScheduleBatch):
        self.forward_mode = ForwardMode.MIXED
        running_bs = running_batch.batch_size()

        for req in running_batch.reqs:
            req._refresh_fill_ids()
            full_len = len(req.full_untruncated_fill_ids)
            req.set_extend_range(full_len - 1, full_len)

# ...

        self.merge_batch(running_batch)
        self.out_cache_loc = out_cache_loc

# ...

        # NOTE: prefix_indices is what has been cached, but we don't cache each decode step
        self.prefix_lens = self.prefix_lens + [
            len(r.origin_input_ids) + len(r.output_ids) + delta
            for r in running_batch.reqs
        ]
        self.extend_lens = self.extend_lens + [1] * running_bs
        self.extend_num_tokens = self.extend_num_tokens + running_bs

self.extend_lens + [1] * running_bs is the whole idea in one expression: a decode step is an extend of length one. Its prefix length is everything generated so far; its extend length is 1. That is why the MIXED mode can exist at all, and it is the same insight vLLM encodes by never distinguishing the cases.

The gate is opt-in and carries real restrictions:

        # Mixed-style chunked prefill
        if (
            self.is_mixed_chunk
            and not running_batch.is_empty()
            and not (new_batch.return_logprob or running_batch.return_logprob)
            # mix_with_running cats input_ids but not input_embeds — shapes would mismatch
            and new_batch.input_embeds is None
        ):
            # TODO (lianmin): support return_logprob + mixed chunked prefill
            running_batch.filter_batch()
            if not running_batch.is_empty():
                running_batch.prepare_for_decode()
                new_batch.mix_with_running(running_batch)
                new_batch.decoding_reqs = running_batch.reqs

Note self.is_mixed_chunk is only true when chunked prefill is enabled and --enable-mixed-chunk was passed, and that requesting logprobs on any request in either half silently disables mixing for that step. That is a live source of "why did my tail latency get worse when I turned on logprobs" reports.

Now put the arithmetic and the scheduling policy together. SGLang's loop prefers prefill; vLLM's loop will happily spend its whole token budget on one long prompt. Either way, if a 32,768-token prefill is admitted as a single step, every decoding request waits for it.

Chunking caps each prefill contribution and lets decode rows run alongside it. Sixteen chunks of 2048 prefill tokens require a budget of at least 2080 when 32 decode rows share the step; with a total budget of 2048 the prompt receives 2016 rows per full step and needs 17 chunks. Attention to the growing prefix changes chunk cost, and the final shorter chunk need not be the most expensive.

Both engines ship chunking on by default (enable_chunked_prefill: bool = True at vllm/config/scheduler.py:L74), with the cap expressed differently. vLLM's is the scheduler's token budget, whose class-level fallback is 2,048:

    DEFAULT_MAX_NUM_BATCHED_TOKENS: ClassVar[int] = 2048
    DEFAULT_MAX_NUM_BATCHED_TOKENS_FOR_BATCHED_DP: ClassVar[int] = 256
    DEFAULT_MAX_NUM_SEQS: ClassVar[int] = 128

That 2,048 is not what an H100 server actually runs on. max_num_batched_tokens is left as None on EngineArgs and resolved by EngineArgs.get_batch_defaults (vllm/engine/arg_utils.py:L2580-L2671), which returns 8,192 for UsageContext.OPENAI_API_SERVER and 16,384 for UsageContext.LLM_CLASS on any device with at least 70 GiB that is not an A100. The 2,048 fallback is what a smaller card — or a 4090 — gets. SGLang's cap is a separate knob, auto-tuned from device memory the same way:

            elif gpu_mem < 90 * 1024:
                # H100, A100
                # (chunked_prefill_size 8k, max_bs 256 if tp < 4 else 512)
                if self.chunked_prefill_size is None:
                    self.chunked_prefill_size = 8192
                if decode_cuda_graph_config.max_bs is None:

On an H100 the two engines therefore land on the same number, 8,192, by two different routes — and it is worth knowing that neither of them is the number written in either config dataclass. The tradeoff the knob controls runs both ways: a smaller cap gives smoother ITL and worse prefill MFU (2,048 and 8,192 both sit far above the ridge, but the smaller step pays more per-step fixed cost), a larger cap gives better TTFT and a fatter ITL tail. This is the first knob in the book you cannot set without knowing your own SLO — which is why §1.2 comes next.

Watch each engine's own log. SGLang prints the phase directly, because it has one:

        msg = (
            f"Prefill batch{iter_msg}, "
            f"#new-seq: {prefill_stats.num_new_seqs}, "
            f"#new-token: {prefill_stats.log_input_tokens}, "
            f"#cached-token: {prefill_stats.log_hit_tokens}, "

# ... L811:
        msg = f"Decode batch{iter_msg}, #running-req: {num_running_reqs}, {token_usage_msg}"

vLLM cannot print a phase, so it prints two throughputs and lets you infer which one was busy:

        # Format and print output.
        log_parts = [
            "Avg prompt throughput: %.1f tokens/s",
            "Avg generation throughput: %.1f tokens/s",
            "Running: %d reqs",
            "Waiting: %d reqs",

Then reproduce the stall on one GPU: start each server with chunking effectively disabled, fire a steady stream of short chat requests, inject one long prompt, and watch the short requests' token streams pause.

# vLLM: raise the budget so a 32k prompt lands in one step
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-num-batched-tokens 32768 --max-model-len 65536 --max-num-seqs 64

# then the same with an explicitly configured 2048 budget and compare the ITL tail
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-num-batched-tokens 2048 --max-model-len 65536 --max-num-seqs 64

# SGLang: disable chunking entirely, then enable it and mix
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
  --chunked-prefill-size -1
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
  --chunked-prefill-size 2048 --enable-mixed-chunk

The measurement harness for the ITL tail, and the lab that walks this end to end, are Lab 05 — Chunked prefill and ITL jitter, which belongs to §1.5.

• Sarathi: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills (Agrawal et al., 2023) — the paper that named chunked prefill and the piggybacking idea, with the prefill/decode intensity argument worked out.
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (Agrawal et al., OSDI 2024) — the stall-free-batching formulation, and the clearest published treatment of why prefill and decode fight.
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (Zhong et al., OSDI 2024) — the other answer: give each phase its own GPUs. Covered in §1.6.
• Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., SOSP 2023) — the vLLM paper; §3 is the original statement of the two-phase workload for this line of systems.
• SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al., 2023) — the RadixAttention paper, and the origin of the extend-versus-decode framing that ForwardMode encodes.
• In the vLLM checkout, docs/design/arch_overview.md for the V1 engine layering, and the NOTE(woosuk) block at vllm/v1/core/sched/scheduler.py:L484-L495 — the closest thing to a design document the scheduler has.