Reproducibility under batch invariance

        # Generate baseline
        print("Generating baseline (warmup)...")
        baseline_out = llm.generate([needle_prompt], sampling)
        assert len(baseline_out) == 1
        baseline_text = baseline_out[0].outputs[0].text
            # Verify needle output still matches
            needle_output = outputs[needle_pos]
            assert needle_output.prompt == needle_prompt

baseline_text is computed and then never used. The assertion compares the prompt it sent against the prompt it got back — a check that the engine did not shuffle the batch, which is true by construction. It is a performance harness: it times the same workload with VLLM_BATCH_INVARIANT at 0 and 1 and prints an overhead percentage. Read its green output as a determinism result and you will conclude the property holds when you have not tested it. The correctness harnesses are elsewhere, and this lab drives those.

Three measurements, in order:

1. Unique completions across batch sizes, flag off. Greater than one is the bug, reproduced.
2. The same count with the flag on. It should be exactly one.
3. The overhead the flag cost, for your model, backend and TP degree — measured, because there is no meaningful number to cite.

There is also a fourth thing worth watching, and run.py reports it separately: the maximum ULP distance between the baseline's per-step logprobs and each batch's. Token equality is the weaker claim. Two runs can emit identical tokens while every logprob differs in the last few bits — the argmax simply survived the perturbation this time. The logprob column is what tells you the arithmetic moved.

Part A — the upstream harnesses, verbatim

Start here. run.py --mode harness prints these with no server and no GPU, so you can read them before installing anything.

vLLM's suite has a detail that will confuse you if you skip it. A directory-scoped fixture turns invariance on for every test in it:

@pytest.fixture(autouse=True)
def enable_batch_invariant_mode(monkeypatch: pytest.MonkeyPatch):
    """Automatically enable batch invariant kernel overrides for all tests."""
    monkeypatch.setattr(envs, "VLLM_BATCH_INVARIANT", True)
    monkeypatch.setenv("VLLM_BATCH_INVARIANT", "1")

So the test that proves the effect exists has to switch it back off itself, and its docstring is the clearest statement of the whole idea in either repository:

def test_logprobs_without_batch_invariance_should_fail(
    backend, monkeypatch: pytest.MonkeyPatch
):
    """
    This test is the inverse of test_logprobs_bitwise_batch_invariance_bs1_vs_bsN.
    It DISABLES batch invariance mode and expects to see non-deterministic behavior
    between BS=1 and BS=N runs. This demonstrates that batch invariance is actually
    doing something useful.

    The test will PASS if we detect differences (proving batch invariance matters).
    The test will FAIL if everything matches (suggesting batch invariance isn't needed).
    """
    # CRITICAL: Disable batch invariance for this test

The model is an environment variable, not a flag, so you can point the whole suite at your own weights:

DEFAULT_MODEL = "Qwen/Qwen3-1.7B"
TEST_MODEL = os.getenv("VLLM_TEST_MODEL", DEFAULT_MODEL)

SGLang's is a driver rather than a test: you start a server and run it against the live endpoint. Its single mode is the experiment in eleven lines — the same prompt at batch 1 through n, and a count of distinct completions:

def test_deterministic(args):
    if args.test_mode == "single":
        # In single mode, we test the deterministic behavior by sending the same prompt in batch sizes ranging from 1 to n_trials.
        texts = []
        for i in range(1, args.n_trials + 1):
            batch_size = i
            text = send_single(args, args.profile, prompt=[PROMPT_1] * batch_size)
            text = text.replace("\n", " ")
            print(f"Trial {i} with batch size {batch_size}: {text}")
            texts.append(text)
        print(f"Total samples: {len(texts)}, Unique samples: {len(set(texts))}")
        return [len(set(texts))]

Its in-tree test class fixes the server arguments, and one of them is not obvious:

DEFAULT_MODEL = "Qwen/Qwen3-8B"
COMMON_SERVER_ARGS = [
    "--trust-remote-code",
    "--cuda-graph-max-bs-decode",
    "32",
    "--enable-deterministic-inference",
]

--cuda-graph-max-bs-decode 32 caps the largest decode batch SGLang captures a graph for (python/sglang/srt/server_args.py:L1906-L1910, resolved at :L4561-L4562). The source does not say why the test class sets it, and I am not going to guess; what matters here is the consequence. It fixes the bucket ladder the sweep runs against, and above the cap there is no graph to pad into. That changes what a clean result means — see What to expect.

$ python3 run.py --mode harness          # prints all of the below, offline

# arm 1: flag off. Expect "Unique samples" > 1.
$ python3 -m sglang.launch_server --model-path Qwen/Qwen3-8B \
    --attention-backend triton --cuda-graph-max-bs-decode 32
$ python3 -m sglang.test.test_deterministic --n-trials 50 --test-mode single

# arm 2: flag on. Expect exactly 1.
$ python3 -m sglang.launch_server --model-path Qwen/Qwen3-8B \
    --attention-backend triton --cuda-graph-max-bs-decode 32 \
    --enable-deterministic-inference
$ python3 -m sglang.test.test_deterministic --n-trials 50 --test-mode single

        logger.warning(
            f"Attention backend not specified. Falling back to '{backend}' for deterministic inference. "
            f"You can explicitly set --attention-backend to one of {DETERMINISTIC_ATTENTION_BACKEND_CHOICES}."
        )
        return {"attention_backend": backend}
    elif view.attention_backend not in DETERMINISTIC_ATTENTION_BACKEND_CHOICES:
        # User explicitly specified an incompatible attention backend
        raise ValueError(
            f"Currently only {DETERMINISTIC_ATTENTION_BACKEND_CHOICES} attention backends are supported for deterministic inference, "
            f"but you explicitly specified '{view.attention_backend}'."
        )
    return {}

That is a comparison hazard, not a convenience: if the two arms of your experiment ran different attention backends, the difference you measured is not the difference you meant to measure. Pin --attention-backend explicitly in both arms.

Part B — the sweep, over plain HTTP

The upstream harnesses want pytest and a checkout. run.py --mode sweep does the same comparison against either engine over the wire, with nothing but the standard library. It sends n copies of one prompt in a single request — which is what the online test does, and what makes the engine schedule them together:

    sp_kwargs: dict[str, Any] = {
        "temperature": 0.6,
        "top_p": 1.0,
        "max_tokens": 8,
        "seed": 42,
        "logprobs": 5,
    }

    tp_size = os.getenv("VLLM_TP_SIZE", "1")
    server_args: list[str] = [
        "--max-model-len=8192",
        "--max-num-seqs=32",
        f"--attention-backend={backend}",
    ]
    if tp_size:
        server_args += ["-tp", tp_size]

    with RemoteOpenAIServer(TEST_MODEL, server_args) as server:

Note that the upstream test does not use temperature=0. It uses 0.6 with a fixed seed, which exercises more of the distribution while remaining reproducible by seed. Greedy is the sharper demonstration and is run.py's default, because a greedy divergence is unarguable; seeded sampling is the more thorough test. Run both. And the comparison it makes is not on text:

        for t, (a, b) in enumerate(zip(logprobs_bs1, logprobs_bsN)):
            if a != b:
                diff = abs(a - b)
                raise AssertionError(
                    f"Prompt {i} Step {t}: Bitwise mismatch "
                    f"(abs diff={diff:.6e}). "
                    f"BS=1 tokens: {tokens_bs1} BS=N tokens: {tokens_bsN}"
                )

$ python3 run.py --mode sweep --engine vllm --url http://localhost:8000 \
    --model Qwen/Qwen3-1.7B --max-batch 32 --save off.json

$ python3 run.py --mode sweep --engine sglang --url http://localhost:30000 \
    --sizes 1,15,16,17,23,24,25 --temperature 0.0

$ python3 run.py --mode cost --url http://localhost:8000 \
    --invariant http://localhost:8001 --batch 32 --trials 20

$ python3 run.py --help          # every knob, all four modes

The default batch ladder deliberately straddles vLLM's CUDA-graph bucket boundaries — 15, 16, 17 and 23, 24, 25 — rather than sampling round numbers. The reason is in What to expect.

Flag off, a long sweep: several unique completions. Not all of them different from each other; divergence is sporadic, because it needs a position where the top two logits are within a few ULP. Chapter §10.4 shows four fp32 partials rearranged differing by 1,746 ULP, which is a relative error of $1.5 \times 10^{-4}$ — enormous next to a near-tie and invisible next to a confident one. One flipped argmax at position 40 rewrites everything after it, which is why the effect looks bimodal: completions are either identical or wildly different, rarely close.

Flag on: exactly one. If it is not, you have found either a coverage gap or a bug, and the gaps are documented. §10.4's coverage map names three: vLLM's fused-MoE kernels default to unsupported and must opt in; SGLang's routing top-k stays unsorted unless the model has fused shared experts; Mamba backends raise rather than degrade. Check your model against that list before filing anything.

The result that surprises people: no divergence at all, flag off. This is common and it is not a failed experiment. Three causes, in order of likelihood:

The cost. This is the number nobody can hand you, and the reason is instructive. §10.4 argues the intuitive ranking is wrong: forcing one attention split and one fixed GEMM tile is nearly free at small batch, because those layers are bandwidth-bound and the padded arithmetic rides along in time the machine was already spending on memory. What you actually pay for is single-channel NCCL — NCCL_MIN_NCHANNELS and NCCL_MAX_NCHANNELS pinned to 1, with NCCL_ALGO=allreduce:tree and custom all-reduce off — and disabled prefix caching. Neither of those appears at TP=1 on a cold workload, which is exactly the configuration a quick benchmark uses. So:

• Run --mode cost at TP=1 and at your real TP degree. The split between attention, GEMM and collective cost inverts between them.
• Run it on a shared-prefix workload and a cold one. Turning cache hits into full prefills can cost more than every kernel change combined.
• Report all four cells, or report none. One number with no configuration attached is what §10.3 spends a chapter arguing against.