Chunked prefill and ITL jitter

The workload is one long prompt injected into a stream of short ones, and the instrument is a timestamp on every token arrival:

• ITL percentiles across the whole run — p50, p90, p99, max.
• The same percentiles in three windows: before the injection (a clean baseline), during the long prefill, and after it completes.
• The worst single gap, and how many seconds it sat from the injection. That is the headline.
• The long request's TTFT, which is what chunking costs you and which should barely move.

Percentiles alone would hide the effect. A 32k prefill blocking a batch of sixteen decoders produces roughly one enormous gap per decoder, which is a handful of samples out of thousands — it does not reach p99, and it is exactly the thing your users complain about. Hence the max, and hence the timestamps: a spike whose offset from the injection matches the long prompt's TTFT is the long prefill; a spike anywhere else is something else, and you should find out what.

Three ways to measure nothing

Each of these silently produces a plausible, wrong answer.

1. The server bundles tokens into one SSE frame. ITL is measured client-side, one timestamp per frame. If a frame carries four tokens, every gap you record is a four-token gap and the spike is smeared. SGLang exposes this directly and defaults it to 1 — leave it there:

    stream_interval: A[
        int,
        "The interval (or buffer size) for streaming in terms of the token length. A smaller value makes streaming smoother, while a larger value makes the throughput higher",
        NS("serving"),
    ] = 1

2. The prompt is served from the prefix cache. run.py builds every prompt from distinct pseudo-random integers so nothing is shared; a repeated long prompt would prefill once and then cost nothing, and you would conclude chunking is unnecessary. Prefix caching is lab 04's subject.

3. Responses stop early. Both engines accept ignore_eos on their OpenAI-compatible routes as an extension field, and the script sets it, so every short request generates exactly --short-output-len tokens and the decode load is constant across the injection:

    stop_token_ids: list[int] | None = []
    include_stop_str_in_output: bool = False
    ignore_eos: bool = False
    min_tokens: int = 0
    skip_special_tokens: bool = True

    repetition_penalty: float = 1.0
    stop_token_ids: Optional[List[int]] = None
    stop_regex: Optional[Union[str, List[str]]] = None
    no_stop_trim: bool = False
    ignore_eos: bool = False
    skip_special_tokens: bool = True

Two servers, one at a time, then a comparison.

# A. chunked prefill on (the default), batch width bounded at 2048
$ vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --max-model-len 32768 --max-num-batched-tokens 2048 --port 8000
$ python3 run.py --port 8000 --label chunked --json chunked.json

# B. off — the pathological baseline. The budget must now cover the whole prompt.
$ vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --max-model-len 32768 --no-enable-chunked-prefill \
    --max-num-batched-tokens 32768 --port 8000
$ python3 run.py --port 8000 --label unchunked --json unchunked.json

$ python3 run.py --compare chunked.json unchunked.json

Note what B forces you to do. With chunking off, the token budget becomes a hard cap on prompt length, and vLLM refuses to start if it is smaller than the context window:

        if (
            self.max_num_batched_tokens < max_model_len
            and not self.enable_chunked_prefill
        ):
            raise ValueError(
                f"max_num_batched_tokens ({self.max_num_batched_tokens}) is "
                f"smaller than max_model_len ({max_model_len}). "
                "This effectively limits the maximum sequence length to "
                "max_num_batched_tokens and makes vLLM reject longer "
                "sequences. Please increase max_num_batched_tokens or "
                "decrease max_model_len."
            )

That error is the whole chapter in one exception: without chunking, the widest iteration you are willing to run is also the longest prompt you can accept. Confirm which mode you actually got from the startup log — the line is emitted once and carries the resolved budget, not the one you passed:

        if self.enable_chunked_prefill:
            logger.info_once(
                "Chunked prefill is enabled with max_num_batched_tokens=%d.",
                self.max_num_batched_tokens,
            )

The third and fourth runs: SGLang, and the two knobs it separates

$ python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
    --context-length 32768 --chunked-prefill-size 2048 --port 30000
$ python3 run.py --port 30000 --label sgl-2048 --json sgl-2048.json

# chunking off
$ python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
    --context-length 32768 --chunked-prefill-size -1 --port 30000
$ python3 run.py --port 30000 --label sgl-off --json sgl-off.json

# and with decodes riding along inside the chunk's iteration
$ python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
    --context-length 32768 --chunked-prefill-size 2048 --enable-mixed-chunk --port 30000

    chunked_prefill_size: A[
        Optional[int],
        "The maximum number of tokens in a chunk for the chunked prefill. Setting this to -1 means disabling chunked prefill.",
        NS("schedule"),
    ] = None

    enable_mixed_chunk: A[
        bool,
        "Enabling mixing prefill and decode in a batch when using chunked prefill.",
        NS("schedule"),
    ] = False

This is a real difference from vLLM and it shows up in this exact measurement. vLLM mixes prefill and decode tokens in one flat batch unconditionally — its running and waiting loops write into the same per-request token dict, traced in §1.5. SGLang gates the same behaviour behind --enable-mixed-chunk, off by default. With it off, an iteration is either a chunk or a decode step, so the short requests still stall for each chunk's duration — bounded, but not overlapped. Turning it on should reduce the residual ITL elevation during the prefill window without changing the worst gap much.

        # Skip validation if chunked prefill is disabled (i.e., size <= 0).
        # Skip validation if disaggregation mode is decode.
        if self.chunked_prefill_size > 0 and self.disaggregation_mode != "decode":
            assert (
                self.chunked_prefill_size % self.page_size == 0
            ), "chunked_prefill_size must be divisible by page_size"

Predict first

1. The unchunked worst gap. Compute it: a 32k prefill on Llama-3-8B is about 807 TFLOP (§1.5 derives it), so divide by the π you measured in lab 01.
2. The chunked worst gap, from the same arithmetic applied to one chunk plus the decode half of that iteration. Remember the last chunk attends to 32k keys where the first attends to 2k, so budget for the last one.
3. The ratio of the two long-prompt TTFTs. This one should be close to 1, and if your prediction is not, re-read why chunking is FLOP-neutral.

Chunking should move the worst gap by orders of magnitude and the long prompt's TTFT hardly at all. If either of those fails, something specific is wrong, and the failures are enumerable:

The last-but-one row is the most instructive experiment in this lab. Run it with --long-count 2 against both engines. SGLang tracks exactly one in-flight chunked request, and enforces it:

        if adder.new_chunked_req is not None:
            # Update chunked prefill
            assert self.chunked_req is None
            self.chunked_req = adder.new_chunked_req

        if self.chunked_req is not None:
            self.chunked_req.inflight_middle_chunks += 1

so the second long prompt's TTFT should be roughly double the first's — head-of-line blocking, by design, to keep radix-cache locking analysable. vLLM has no such invariant: both sit in the running queue and the same clamp fires per request, so with a large enough budget they genuinely interleave and both TTFTs land in between. Neither behaviour is better; they price the same tradeoff differently, and this is the measurement that shows it.

    long_prefill_token_threshold: int = Field(default=0, ge=0)
    """For chunked prefill, a request is considered long if the prompt is
    longer than this number of tokens. 0 disables the cap (default)."""