Batch invariance and reproducible inference

A support ticket, roughly verbatim from every serving team that has ever run an eval suite against a live endpoint:

This is not a bug in either engine but a property of both, and vLLM has a test that asserts the property holds — one that fails if the outputs happen to match:

def test_logprobs_without_batch_invariance_should_fail(
    backend, monkeypatch: pytest.MonkeyPatch
):
    """
    This test is the inverse of test_logprobs_bitwise_batch_invariance_bs1_vs_bsN.
    It DISABLES batch invariance mode and expects to see non-deterministic behavior
    between BS=1 and BS=N runs. This demonstrates that batch invariance is actually
    doing something useful.

    The test will PASS if we detect differences (proving batch invariance matters).
    The test will FAIL if everything matches (suggesting batch invariance isn't needed).
    """
    # CRITICAL: Disable batch invariance for this test

SGLang ships the same experiment as a driver against a live server: its "single" mode sends one identical prompt at batch sizes 1, 2, … n and prints how many distinct completions came back (python/sglang/test/test_deterministic.py:L461-L471). That counter — "unique samples" — is the subject of this chapter. Two properties are being conflated in the ticket, and separating them is the first step:

A GPU reduction — a dot product, a softmax denominator, an attention numerator — is never a single serial sum. The work is cut into pieces, each summed independently, then merged. How many pieces is a scheduling decision: the kernel picks a split count that fills the machine, and "fills the machine" depends on how much other work is present. Change the batch, change the split count, change the shape of the tree that adds the partials. Floating-point addition is not associative, so a different tree is a different number.

Both answers are wrong, and neither more wrong than a kernel author would accept. But if those are two candidate logits differing by less than 1.5 × 10^-4 relative, the argmax flips, a different token is emitted, and every later token is conditioned on a different prefix. One divergence at position 40 rewrites the rest of the completion.

Non-associativity, concretely

IEEE-754 binary32 has a 24-bit significand. Every addition rounds its exact result to the nearest representable value. Rounding is not distributive over grouping, so for $a, b, c, d \in \mathbb{F}$, in general

where $\mathrm{fl}(\cdot)$ is round-to-nearest-even into binary32. Figure 1 is a witness. Serially, $p_0 + p_1 = 3.7184176$, then $+p_2$ cancels almost every significant bit and leaves $0.00041771$; adding $p_3$ gives 0x3a2e392e. Pairwise, $p_2 + p_3 = -3.7177529$ is formed first, so the cancellation happens once instead of twice and the result is 0x3a2e4000 — 1746 units in the last place away. The exact sum is 0.00066451075; both answers exceed it, by different amounts; their errors have the same sign.

From batch size to reduction tree

The link from "who else is in the batch" to "which tree" runs through occupancy. A decode attention kernel launches one thread block per (query block, KV head); at batch 1 for Llama-3-8B (L=32, d=4096, h=32, $h_{kv}$=8, $d_h$=128) that is 8 blocks on an H100's 132 SMs, so the kernel splits the KV axis to manufacture parallelism (§3.3). SGLang's split policy states the dependency outright, taking batch size as an explicit input:

    # NOTE: this is a hack to let num_kv_split grows up with seqlen gradually
    ext_seq_len = tl.cast(max_seq_len, tl.float32) / 64.0
    ext_device_core_count = tl.cast(
        device_core_count * tl.maximum(tl.log2(ext_seq_len), 1.0), tl.int32
    )
    block_h, num_kv_group = 16, num_head // num_kv_head
    if num_kv_group == 1:
        token_grid = num_seq * num_group * num_head
    else:
        # from triton_ops/decode_attention.py:_decode_grouped_att_m_fwd
        block_h = tl.minimum(block_h, num_kv_group)
        token_grid = num_seq * num_group * tl.cdiv(num_head, block_h)
    max_kv_splits_2 = tl.minimum(
        tl.cdiv(ext_device_core_count, token_grid), max_kv_splits
    )
    kv_chunk_size_2 = tl.cdiv(max_seq_len, max_kv_splits_2)

    num_kv_splits = tl.maximum(
        tl.cdiv(seq_lens, kv_chunk_size_1), tl.cdiv(seq_lens, kv_chunk_size_2)
    )

num_seq is the batch size. It enters token_grid, which divides into the core budget, which sets max_kv_splits_2, the chunk size, and finally how many fp32 partials the merge kernel adds. Nothing about the request changed.

The same numbers, framed as occupancy rather than invariance, are §3.3's table. There they are a feature; here they are the bug.

1 · Split-K and segmented attention reductions

The primary source, and the one §3.3 hands off here. vLLM removes the split axis entirely: is_batch_invariant is a disqualifier in the unified Triton kernel's use_3d predicate (vllm/v1/attention/ops/triton_unified_attention.py:L1035-L1049), and FlashAttention's num_splits is pinned to 1 at metadata build (vllm/v1/attention/backends/flash_attn.py:L592-L593) and in both cascade calls (:L1765, :L1790). SGLang does the opposite in form and the same in effect — keep the split, make its count a function of sequence length only:

        if self.split_tile_size is not None and self.enable_deterministic:
            if num_group > 1:
                expanded_seq_lens = seq_lens.repeat_interleave(num_group)
            else:
                expanded_seq_lens = seq_lens

            num_kv_splits[:] = (
                expanded_seq_lens + self.split_tile_size - 1
            ) // self.split_tile_size
            return

split_tile_size defaults to 256 for Triton (python/sglang/srt/layers/attention/triton_backend.py:L286-L296) and 2048 for FlashInfer decode, prefill 4096, with CUDA-graph KV splitting disabled outright (python/sglang/srt/layers/attention/flashinfer_backend.py:L405-L419). The early return is the point: it bypasses Figure 2's occupancy kernel, so a request at $s = 2560$ gets $\lceil 2560/256 \rceil = 10$ splits alone or in a batch of 200.

2 · Cross-block accumulation: split-K in GEMMs, and the workspace that enables it

Split-K in a matmul partitions the contraction dimension across thread blocks and merges via atomics or a workspace; which block lands first is undefined, so the order is undefined. On Hopper and Blackwell vLLM does not replace the GEMM at all — it removes cuBLAS's ability to choose split-K by starving the workspace:

        else:
            # Hopper (SM90) and Blackwell (SM100): the only source of batch
            # variance is split-k, which we disable via the cuBLAS workspace
            # config.
            os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":16:8"
            os.environ["CUBLASLT_WORKSPACE_SIZE"] = "1"

On SM8x — Ampere and Ada, since the predicate is is_device_capability_family(80) — that trick is not trusted, so it installs a Triton persistent matmul instead (:L917-L923) whose K loop accumulates in one fixed order per output tile. Note the dispatcher registration is the only SM8x-gated part: UnquantizedLinearMethod.apply and the LM-head method both call linear_batch_invariant directly whenever the flag is set on any CUDA-alike device (vllm/model_executor/layers/linear.py:L216-L218, vllm/model_executor/layers/vocab_parallel_embedding.py:L73-L75), so on Hopper the unquantized linears still run the Triton kernel and the workspace starvation covers only the GEMMs that still reach cuBLAS. The MoE kernel gets a fixed config with the split axis explicitly set to one:

) -> dict[str, int]:
    if envs.VLLM_BATCH_INVARIANT:
        return {
            "BLOCK_SIZE_M": 64,
            "BLOCK_SIZE_N": 64,
            "BLOCK_SIZE_K": 32,
            "GROUP_SIZE_M": 8,
            "SPLIT_K": 1,
        }

The same argument applies to collectives, where the "blocks" are GPUs. NCCL picks an algorithm and a channel count from the message size, and the message size is the token count (§5.4). Both engines pin every free variable; SGLang says why in the comments:

                else:
                    # CUDA: use NCCL tree algorithm
                    os.environ["NCCL_ALGO"] = "allreduce:tree"
                    self.disable_custom_all_reduce = True
                    # should_torch_symm_mem_allreduce() takes the
                    # symmetric-memory path only below a byte threshold, so
                    # which reduce runs would follow the token count.
                    self.enable_torch_symm_mem = False
                    # Each channel carries a differently shaped tree and the
                    # channel count is picked from the message size, so a
                    # token's reduction order would follow the token count.
                    nchannels = str(envs.SGLANG_DETERMINISTIC_NCCL_NCHANNELS.get())
                    os.environ["NCCL_MIN_NCHANNELS"] = nchannels
                    os.environ["NCCL_MAX_NCHANNELS"] = nchannels

3 · Kernel selection changing with shape

A different tile size is a different reduction tree even when nothing is "split". Quantized GEMM dispatchers pick a CUTLASS config from M — §4.3's M-bucketed dispatch — and M is the token count. vLLM's fix is a second dispatcher that reads N and ignores M:

  // keep the CUTLASS config independent of M for batch invariance
  uint32_t const n = b.size(1);
  if (n <= 1280) {
    return cutlass_gemm_caller_sm90_fp8<Cutlass3xGemmM64_N1280>(
        out, a, b, b_scales, a_scales, std::forward<EpilogueArgs>(args)...);
  }
  return cutlass_gemm_caller_sm90_fp8<Cutlass3xGemmM64_N8192>(
      out, a, b, b_scales, a_scales, std::forward<EpilogueArgs>(args)...);

The default dispatcher immediately above it (:L318-L349) branches on m <= 16, m <= 64, m <= 128, and m >= 8192 && k >= 6144. Under invariance every M runs the M=64 config. Kernels that cannot offer a shape-independent path are removed outright: FP8 Marlin returns "FP8 Marlin not supported for batch invariant execution." (vllm/model_executor/kernels/linear/scaled_mm/marlin.py:L44-L45), and SGLang rejects its CuTe DSL BF16 GEMM with "--bf16-gemm-backend cutedsl is batch-size dependent and cannot be combined with --enable-deterministic-inference" (python/sglang/srt/layers/quantization/unquant.py:L120-L125).

CUDA graphs make this discontinuous. Replay pads the batch up to the nearest captured bucket (§8.1), and vLLM's ladder is [1, 2, 4, 8, 16, 24, …]. Batches of 9 and 13 both replay the batch-16 graph and share padded launch geometry; this alone does not guarantee bitwise agreement. A batch of 17 pads to 24 and can select another numerical path. Bucket boundaries can therefore produce discontinuities, which is why "it only reproduces at batch 17" is a real bug report and not a red herring.

4 · Unsorted top-k in MoE routing

The subtlest one, because it is not a floating-point issue at all. Routing selects the top-$k$ experts per token with torch.topk, and both engines pass sorted=False by default. Neither sorted=False nor sorted=True guarantees stable tied indices in PyTorch; sorting orders returned values, not a deterministic secondary index key. See the topk contract. The quoted source comment below expresses implementation intent, not that API guarantee. Tied router logits can send a token to a different expert on a different run, which changes its arithmetic wholesale. §7.1 owns the routing mechanics; the gates are:

    # For batch invariance, use sorted=True to ensure deterministic expert selection
    use_sorted = envs.VLLM_BATCH_INVARIANT
    group_idx = torch.topk(group_scores, k=topk_group, dim=-1, sorted=use_sorted)[
        1
    ]  # [n, top_k_group]

    topk_weights, topk_ids = torch.topk(
        tmp_scores,
        k=topk,
        dim=-1,
        sorted=(True if num_fused_shared_experts > 0 else False),
    )

Note how the flag is reached. vLLM ties sorting to the invariance flag directly; SGLang ties it to num_fused_shared_experts, because that trick needs a known slot — not because anyone asked for determinism. On a model without fused shared experts, SGLang's routing top-k stays unsorted even under --enable-deterministic-inference. Check that against your model before trusting a bitwise claim on an MoE.

Both implementations descend from the same reference — SGLang's header says so: "# Adapted from https://github.com/thinking-machines-lab/batch_invariant_ops/…" (python/sglang/srt/batch_invariant_ops/__init__.py:L1) — and both work the same way: override ATen operators at the PyTorch dispatcher with Triton kernels whose reduction order does not depend on input shape.

    _batch_invariant_LIB.impl("aten::_log_softmax", _log_softmax_batch_invariant, key)
    _batch_invariant_LIB.impl("aten::softmax", softmax_batch_invariant, key)
    _batch_invariant_LIB.impl("aten::_softmax", softmax_batch_invariant, key)
    _batch_invariant_LIB.impl("aten::mean.dim", mean_batch_invariant, key)
    # torch 2.12+ registers a built-in Triton bmm kernel for CUDA
    # (torch._native.ops.bmm_outer_product), so we need allow_override
    # to replace it at the dispatcher level.
    _batch_invariant_LIB.impl(
        "aten::bmm", bmm_batch_invariant, key, allow_override=True
    )
    torch.bmm = bmm_batch_invariant

    reduced_precision_val = (
        (False, False) if is_torch_equal_or_newer("2.10.0") else False
    )
    torch.backends.cuda.matmul.allow_fp16_reduced_precision_reduction = (
        reduced_precision_val
    )
    torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction = (
        reduced_precision_val
    )
    if current_platform.is_cuda():
        torch.backends.cuda.preferred_blas_library(backend="cublaslt")

    if not _is_npu:
        # Register for detected device
        _batch_invariant_LIB.impl("aten::mm", mm_batch_invariant, dispatch_key)
        _batch_invariant_LIB.impl("aten::addmm", addmm_batch_invariant, dispatch_key)
        _batch_invariant_LIB.impl(
            "aten::_log_softmax", _log_softmax_batch_invariant, dispatch_key
        )
        _batch_invariant_LIB.impl("aten::mean.dim", mean_batch_invariant, dispatch_key)
        _batch_invariant_LIB.impl("aten::rms_norm", _rms_norm_aten_compat, dispatch_key)
        _batch_invariant_LIB.impl("aten::mm.dtype", _mm_dtype_compat, dispatch_key)

        if enable_bmm:
            _batch_invariant_LIB.impl("aten::bmm", bmm_batch_invariant, dispatch_key)
            # Also monkeypatch torch.bmm directly as a fallback
            _original_torch_bmm = torch.bmm
            torch.bmm = bmm_batch_invariant

The replacement kernels are deliberately dull. log_softmax runs one thread block per row and loops the row serially in BLOCK_SIZE=1024 chunks — no cross-block reduction, so no order to get wrong (vllm/model_executor/layers/batch_invariant.py:L341-L445); mean_dim (:L448-L585) and rms_norm (:L774-L886) do the same. matmul_persistent uses one hard-coded tile config per dtype instead of an autotuned table (:L158-L183), and bmm_kernel's docstring states the contract: "Each program computes one (batch_idx, tile_m, tile_n) tile, accumulating along K in a fixed order to preserve batch invariance" (:L233-L236).

Around that core sits a larger surface of configuration. The C++ side reads the same variable through a cached getter, so compiled kernels branch on it without a Python round-trip (csrc/core/batch_invariant.hpp:L9-L16), and a block of process environment is rewritten before anything is initialised:

def override_envs_for_invariance():
    os.environ["VLLM_ALLREDUCE_USE_SYMM_MEM"] = "0"

    os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"

    # NCCL determinism settings
    os.environ["NCCL_LAUNCH_MODE"] = "GROUP"
    os.environ["NCCL_COLLNET_ENABLE"] = "0"
    os.environ["NCCL_NVLS_ENABLE"] = "0"
    os.environ["NCCL_P2P_NET_DISABLE"] = "1"
    os.environ["NCCL_MIN_NCHANNELS"] = "1"
    os.environ["NCCL_MAX_NCHANNELS"] = "1"
    os.environ["NCCL_PROTO"] = "Simple"
    os.environ["NCCL_ALGO"] = "allreduce:tree"
    os.environ["NCCL_NTHREADS"] = "1"
    os.environ["NCCL_SOCKET_NTHREADS"] = "1"

    # torch.compile settings
    os.environ["VLLM_USE_AOT_COMPILE"] = "0"

VLLM_USE_AOT_COMPILE=0 is the §8.2 handshake: fusion rewrites arithmetic, so ahead-of-time compilation is switched off rather than audited. TF32 goes too (:L1002-L1006) — a 10-bit multiply is a different rounding.

The gaps deserve as much attention as the coverage. A fused-MoE kernel in vLLM is excluded unless it opts in, and the base class opts out:

    @staticmethod
    def _supports_batch_invariance() -> bool:
        """
        Whether the kernel supports batch invariance, i.e. the output does not
        depend on the order of the tokens in the input batch. This is useful
        for determining if the kernel can used with VLLM_BATCH_INVARIANT=1.
        """
        return False

Failing closed is the right default, but it narrows your MoE kernel choices to whatever has been audited, per kernel. Mamba is harder still: _cached_get_mamba_attn_backend raises "VLLM batch_invariant mode is not supported for {backend}." rather than degrading (vllm/v1/attention/selector.py:L243-L247).

1. Process start. init_batch_invariance() is the first thing init_worker_distributed_environment does, ahead of set_custom_all_reduce (vllm/v1/worker/gpu_worker.py:L1406-L1410) — before the distributed environment exists, which is what makes the NCCL variables effective.
2. init_batch_invariance() (vllm/model_executor/layers/batch_invariant.py:L997-L1006) reads envs.VLLM_BATCH_INVARIANT — whose comment two lines up (vllm/envs.py:L627-L628) claims it "Requires NVIDIA GPU with compute capability >= 9.0" and is stale: the SM8x branch below installs real overrides and the determinism suite gates on has_device_capability(80) (tests/v1/determinism/utils.py:L26-L36), so an A100 qualifies — (vllm/envs.py:L629, default "0"), calls override_envs_for_invariance() and enable_batch_invariant_mode(), then sets torch.backends.cuda.matmul.fp32_precision = "ieee".
3. enable_batch_invariant_mode() (:L904-L973) opens a torch.library.Library("aten", "IMPL") and branches on compute capability — SM80 gets four Triton matmul overrides, SM90/SM100 the cuBLAS workspace starvation — then registers the five reduction ops unconditionally.
4. Config validation. VllmConfig sets disable_cascade_attn = True (vllm/config/vllm.py:L1740-L1748); ParallelConfig._verify_args sets disable_custom_all_reduce = True (vllm/config/parallel.py:L1027-L1029); Attention.__init__ disables prefix caching for FLASHINFER and TRITON_MLA (vllm/model_executor/layers/attention/attention.py:L363-L378).
5. Backend selection. get_attn_backend passes use_batch_invariant=envs.VLLM_BATCH_INVARIANT into the selector (vllm/v1/attention/selector.py:L169), which demotes FA4 (vllm/v1/attention/backends/fa_utils.py:L164) — the same demotion table §3.2 reads.
6. Every step. FlashAttentionMetadataBuilder.build sets max_num_splits = 1 (vllm/v1/attention/backends/flash_attn.py:L592-L593); the Triton path's use_3d evaluates false. Neither depends on num_seqs. UnquantizedLinearMethod.apply routes to linear_batch_invariant (vllm/model_executor/layers/linear.py:L216-L218), and so does the LM head through vocab_parallel_embedding.py:L73-L75 — the last GEMM before the argmax.

SGLang's path is shorter and later: ModelRunner.maybe_enable_batch_invariant_mode runs after weight loading (python/sglang/srt/model_executor/model_runner.py:L761-L765, called from :L664), while the environment and backend decisions happen earlier in ServerArgs._handle_deterministic_inference (python/sglang/srt/server_args.py:L8260-L8366).

SGLang also constrains the scheduler, which vLLM does not. Chunked prefill would otherwise cut a prompt wherever the leftover token budget falls, moving the prefill split boundaries; so the chunk length is snapped to a multiple of the attention split tile:

                # When truncation align size is set, we want to assert that the prefill prefix length is multiple of truncation align size
                # A typical use case is when deterministic inference is enabled with flashinfer attention backend,
                # we need the prefill prefix length to be multiple of attention split size
                if truncation_align_size is not None:
                    if trunc_len < truncation_align_size:
                        return AddReqResult.OTHER
                    else:
                        trunc_len = truncation_align_size * (
                            trunc_len // truncation_align_size
                        )

The value is 4096 for both FlashInfer and Triton prefill, from Scheduler.init_deterministic_inference_config (python/sglang/srt/managers/scheduler.py:L1541-L1557). This is the deepest consequence in either repo: batch invariance is not only a kernel property, it is a scheduling constraint.

Every mechanism above trades parallelism or kernel choice for order, and the intuitive ranking of what that costs is wrong.

Attention splits: cheapest where it looks worst. At batch 1 with $s = 2560$, forcing $S = 1$ drops vLLM's decode attention grid from 128 blocks to $G_{2D} = 8$ — 6.1% of an H100's 132 SMs against 97% (§3.3, Figure 1), a 16× loss of memory-level parallelism. But the KV read is $2 \cdot 2560 \cdot 8 \cdot 128 \cdot 2 = 10.5$ MB per layer, 336 MB over 32 layers, against 15.0 GB of weights on the same step, whose 15.0/3.35 = 4.48 ms is the batch-1 decode step. Attention's 336 MB is 0.10 ms of that; a three times slower unsplit kernel therefore adds 0.20 ms, or 4.5%. And where attention does dominate the step, the adaptive policy of Figure 2 has already decayed to $S = 1$ or $2$, so pinning it changes almost nothing.

Fixed GEMM tiles: also cheap at batch 1, for a reason worth internalising. vLLM's persistent matmul uses BLOCK_SIZE_M = 128 for every shape, so a batch-1 qkv projection ($M=1$, $K=4096$, $N=6144$) computes 128× the FLOPs it needs. It does not take 128× the time: the padded GEMM's arithmetic intensity is $2MKN/(2KN) = M = 128$ FLOP/byte, below the H100 ridge $I^{*} = 295$, so it stays bandwidth-bound — 50.3 MB of weights at 3.35 TB/s is 15.0 µs, while 6.44 GFLOP at 989.4 TFLOP/s is 6.5 µs. The padding is free because the machine was waiting on memory anyway. The real loss is that one config serves every shape where an autotuned table would have used dozens — including MoE, where the fixed 64/64/32 replaces exactly the tuned JSON files that exist because the default is bad (vllm/model_executor/layers/fused_moe/fused_moe.py:L1126-L1129).

Collectives and features: where the money actually goes. vLLM's NCCL_MIN_NCHANNELS=NCCL_MAX_NCHANNELS=1 with NCCL_NTHREADS=1, NCCL_ALGO=allreduce:tree and custom all-reduce off is not a tuning tweak; it removes the parallelism that makes NVLink fast. SGLang pins the same two channel variables but to SGLANG_DETERMINISTIC_NCCL_NCHANNELS, which defaults to 8 (python/sglang/srt/environ.py:L1115), and sets neither NCCL_NTHREADS nor NCCL_PROTO — a materially less aggressive setting than vLLM's. And disabling prefix caching — vLLM for two backends, SGLang for the radix cache on unsupported backends — converts cache hits into full prefills, which on a shared-prefix workload can cost more than every kernel change combined.

The weaker guarantees, which are usually enough

Two properties sit between "nothing" and "batch invariant", and both are cheap:

• Determinism at fixed batch. If the harness always sends the same requests in the same order at the same concurrency, the batches match and the results repeat. This is why running the failing test alone passes: batch 1 is a perfectly reproducible batch. It breaks the moment arrival timing shifts.
• Per-request sampling determinism. A different axis: making the sampler independent of the batch without touching the logits. SGLang's multinomial_with_seed derives Gumbel noise from murmur_hash32(seed, positions, col_indices) in float64 rather than a global RNG stream (python/sglang/srt/layers/sampler.py:L688-L720), so a request's draws depend only on its own seed and position (§6.1).

seed buys the second, not the first. §9.1 is blunt: neither engine promises the same seed yields the same tokens across runs, because the batch changes the reduction order. A seed fixes the draws, not the probabilities they are applied to.

"temperature=0 means deterministic"

It means the sampler is deterministic. Greedy decoding is $\arg\max$ over a logit vector that is itself batch-dependent, and when the top two logits are within a few ULP — routine at high-entropy positions — the argmax is a coin flip decided by reduction order. Say "greedy, not deterministic" out loud and half the ticket queue disappears.

Comparing eval scores across engine versions

A version bump can change the attention backend, the CUTLASS config table, the graph bucket ladder, or the torch.compile fusion set — all of which move logits without moving weights. A 0.3-point MMLU delta needs controlled paired evaluation and uncertainty analysis. Bitwise invariance helps isolate numerical causes but is not a prerequisite for all valid statistical evidence. Record the SHA, backend, and flag alongside the score, per §10.3.

The regression that only reproduces at one batch size

Bucket padding can create a discontinuity: batches 9 through 16 may share a graph, while 17 selects another. Sharing a bucket is not sufficient for bitwise identity; scheduling, other operators, reduction order, sampling state, and backend eligibility also matter. Re-run with --cuda-graph-max-bs-decode 32 (SGLang's determinism tests do exactly this, python/sglang/test/test_deterministic_utils.py:L13-L18) or with graphs off and see whether the discontinuity survives.

Greedy speculative decode diverging from greedy plain decode

The motivating case, handed here by §6.2. Greedy speculation is output-equivalent to greedy decoding under exact arithmetic. In practice the target verifies $k+1$ draft positions in one pass — a different sequence length, kernel path, and reduction order than one-token-at-a-time — so the verification logits are not bitwise equal to plain-decode logits, and an argmax near a tie can reject a token the plain path would accept. SGLang forbids the combination where the sampler is the offender: "--speculative-use-rejection-sampling is incompatible with --enable-deterministic-inference; the sampling kernel draws coins from the global RNG and is not batch-invariant." (python/sglang/srt/arg_groups/speculative_hook.py:L652-L657).

Mathematically equivalent is not bitwise equivalent

The best-documented instance of this in either repo is SGLang refusing to compute logprobs as $\log(\mathrm{softmax}(x))$ even though that is the definition:

                # Deterministic inference must derive the returned logprobs
                # from F.log_softmax — the same kernel prefill rescoring uses —
                # not log(softmax(x)) below: the two disagree at ~1e-6 despite
                # being mathematically equivalent, which breaks bitwise
                # prefill/decode logprob alignment.

Same reason the fast input-logprob path is disabled under determinism (python/sglang/srt/layers/logprob_processor.py:L453-L458). Logprobs from a prefill rescore and logprobs from generation must come from the same kernel or they will not match.

Reproduce the effect before you fix it. SGLang's driver is the shortest path — start a server without determinism, then run the sweep:

python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --attention-backend triton

# in another shell
python3 -m sglang.test.test_deterministic --n-trials 50 --test-mode single

It sends one identical prompt at batch sizes 1..50 and prints Total samples: 50, Unique samples: N (python/sglang/test/test_deterministic.py:L461-L471). Restart with --enable-deterministic-inference and rerun; N should become 1. --test-mode prefix is the harder test — it varies shared-prefix length across 1, 511, 2048 and 4097 characters of a fixed long prompt, exercising the chunked-prefill alignment of schedule_policy.py:L1393-L1402.

On vLLM, the equivalent pair is the two tests that assert opposite things:

pytest -s tests/v1/determinism/test_batch_invariance.py -k "without_batch_invariance_should_fail"
VLLM_BATCH_INVARIANT=1 pytest -s tests/v1/determinism/test_batch_invariance.py -k "bitwise_batch_invariance_bs1_vs_bsN"

# then price it (needs an H100 or newer; the script gates on SM90)
VLLM_BENCH_MODEL=Qwen/Qwen3-1.7B VLLM_BENCH_NUM_TRIALS=5 python3 benchmarks/benchmark_batch_invariance.py

The benchmark reports the overhead percentage this chapter declines to guess at. Run it at your own TP size: the split between attention, GEMM and collective cost shifts completely between TP=1 and TP=8, and the NCCL pinning only bites in the latter.

• Thinking Machines, "Defeating Nondeterminism in LLM Inference" (2025) — thinkingmachines.ai. The argument this chapter formalises, and the direct ancestor of both implementations: SGLang's module credits thinking-machines-lab/batch_invariant_ops in its header.
• thinking-machines-lab/batch_invariant_ops — the reference op set, from which matmul_persistent, log_softmax, mean_dim and the RMSNorm kernel in both repos descend nearly verbatim.
• FlashInfer PR #1675 — the fixed split-size and disable_split_kv plumbing that SGLang's FlashInfer backend calls into; cited by name in python/sglang/srt/layers/attention/flashinfer_backend.py:L402-L404.
• vLLM's tests/v1/determinism/ — eight files, 2,413 lines, with op-level tests for matmul, RMSNorm, NVFP4 and CUTLASS, plus test_online_batch_invariance.py, the one that exercises a running server. Read test_decode_logprobs_match_prefill_logprobs for the prefill/decode alignment problem SGLang's sampler comment describes from the other side. SGLang's counterparts are test/registered/attention/test_deterministic.py (the flashinfer/fa3/triton CI matrix) and test/registered/sampling/test_deterministic_gumbel_u1.py.
• §3.3 for the split-K mechanism and its occupancy arithmetic, §7.1 for routing, §8.1 for bucket padding, §5.4 for what the NCCL pinning gives up.