FlashInfer and the attention-backend abstraction

Here is the entire attention kernel source tree in vLLM at this SHA:

attention_dtypes.h   attention_generic.cuh  dtype_bfloat16.cuh
dtype_float16.cuh    dtype_float32.cuh      dtype_fp8.cuh

Headers. No .cu. The kernel that gave §2.2 its name — the one the PagedAttention paper benchmarked — is gone from the tree. In its place is vllm/v1/attention/backends/, 23 Python modules plus an mla/ subdirectory of 21 more, and not one of them contains a kernel. They contain adapters: classes that translate vLLM's batch state into the argument list some library wants, and then call it. The kernels vLLM does still write live one directory over, in vllm/v1/attention/ops/ — 25 modules of Triton, including the unified attention kernel of §3.3 and the merge of §3.1. Triton, not CUDA C++, is where vLLM's own attention code now lives.

This makes the backend abstraction the whole story on CUDA, and it makes the selection logic load-bearing in a way it never was when there was a house kernel to fall back on. Two failure modes follow directly.

The first is loud. Ask for a backend the configuration cannot support and vLLM refuses to start:

        if selected_backend is not None:
            try:
                backend_class = _get_attn_backend_class(selected_backend)
                invalid_reasons = backend_class.validate_configuration(
                    device_capability=device_capability,
                    **attn_selector_config._asdict(),
                )
            except (ImportError, OSError) as e:
                raise ValueError(
                    f"Selected backend {selected_backend} is not valid for "
                    f"this configuration. Reason: [{type(e).__name__}: {e}]"
                ) from e
            if invalid_reasons:
                raise ValueError(
                    f"Selected backend {selected_backend} is not valid for "
                    f"this configuration. Reason: {invalid_reasons}"
                )
            else:
                logger.info("Using %s backend.", selected_backend)
                return _backend_cls_path(backend_class)

The reason strings come from a fixed vocabulary — "head_size not supported", "kv_cache_dtype not supported", "block_size not supported", "MLA not supported" — assembled in AttentionBackend.validate_configuration (vllm/v1/attention/backend.py:L353-L440). Version-stamp this: at a556f3f an explicitly selected backend that fails validation raises. It does not silently degrade.

The second failure mode is quiet, and it is the one that ruins benchmarks. Selecting FLASHINFER does not select a kernel. It selects a dispatcher that will, at metadata-build time, choose between FlashInfer's native FA2/FA3 paths, NVIDIA's trtllm-gen cubins, and the XQA decode kernel — per batch, based on token count, KV dtype, and head ratio. Two runs with identical flags can execute different kernels because one had 200 decode tokens and the other had 300. §5 reads that dispatch table. If you are comparing "FlashInfer versus FlashAttention" without reading the logger.warning_once("Using TRTLLM %s attention (auto-detected).") line in your log, you do not know what you measured.

A transformer forward pass calls attention $L$ times — 32 for Llama-3-8B, 80 for Llama-3-70B. Every one of those calls needs the same description of the batch: where each request's query tokens start, how long each sequence is, which KV blocks belong to which request. That description does not change between layer 3 and layer 47. Computing it 80 times would be 80× the host work for one copy's worth of information.

So both engines build it exactly once per forward step, before the model runs, and hand every layer a reference to the same object. This is the single design decision the entire abstraction exists to enable, and it is why the interface is split into a builder (runs once per step, on the host, sees the whole batch) and an impl (runs once per layer, on the device, sees only tensors).

The block table those 80 layers read may not be the one the KV cache manager allocated: as §2.2 found, vLLM decouples kernel_block_size from block_size, so what reaches the kernel can be a subdivided view of the logical table. The metadata builder is where that subdivision is materialised.

What a backend class must provide

Four abstract static methods, and nothing else is mandatory:

class AttentionBackend(ABC):
    """Abstract class for attention backends."""

    supported_dtypes: ClassVar[list[torch.dtype]] = [torch.float16, torch.bfloat16]
    supported_kv_cache_dtypes: ClassVar[list["CacheDType"]] = [
        "auto",
        "float16",
        "bfloat16",
    ]
# ...
    @staticmethod
    @abstractmethod
    def get_name() -> str:
        raise NotImplementedError

    @staticmethod
    @abstractmethod
    def get_impl_cls() -> type["AttentionImplBase"]:
        raise NotImplementedError

    @staticmethod
    @abstractmethod
    def get_builder_cls():  # -> Type["AttentionMetadataBuilder"]:
        raise NotImplementedError

    @staticmethod
    @abstractmethod
    def get_kv_cache_shape(
        num_blocks: int,
        block_size: int,
        num_kv_heads: int,
# ...

A name, an impl class, a builder class, and the KV cache shape this backend's kernels want. Everything else on AttentionBackend is a capability predicate with a conservative default: supports_sink(), supports_sliding_window(), is_mla(), is_sparse(), supports_compute_capability(), get_supported_head_sizes(), and a dozen more, all returning False or [] unless overridden (vllm/v1/attention/backend.py:L250-L335). A new backend that overrides nothing is treated as: dense decoder attention, bf16/fp16 only, no sinks, no sliding window, no CUDA graphs. It will be rejected for anything else, which is the correct default.

What the builder must produce

class AttentionMetadataBuilder(ABC, Generic[M]):
    # Does this backend/builder support CUDA Graphs for attention (default: no).
    # Do not access directly. Call get_cudagraph_support() instead.
    _cudagraph_support: ClassVar[AttentionCGSupport] = AttentionCGSupport.NEVER
    # Does this backend/builder reorder the batch?
    # If not, set this to None. Otherwise set it to the query
    # length that will be pulled into the front of the batch.
    reorder_batch_threshold: int | None = None
    # Does this backend/builder support updating the block table in existing
    # metadata
    supports_update_block_table: bool = False
    # Whether the builder constructor requires the block-table width.
    requires_block_table_width: ClassVar[bool] = False
    # Whether all step-dependent draft decode metadata can be updated in place,
    # allowing one metadata build to be reused across autoregressive draft steps.
    supports_draft_decode_metadata_update: bool = False

One abstract method, build(common_prefix_len, common_attn_metadata, fast_build=False) (vllm/v1/attention/backend.py:L743-L762), returning the backend's own metadata type M. Its input is the engine-neutral CommonAttentionMetadata, whose first fields are exactly the tensors §1.3 identified as the concrete form of Orca's selective batching:

    query_start_loc: torch.Tensor
    query_start_loc_cpu: torch.Tensor
    """(batch_size + 1,), the start location of each request in query Tensor"""

    seq_lens: torch.Tensor
    """(batch_size,), the number of computed tokens for each request"""

    num_reqs: int
    """Number of requests"""
    # TODO(lucas): rename to num_tokens since it may be padded and this is misleading
    num_actual_tokens: int
    """Total number of tokens in batch"""
    max_query_len: int
    """Longest query in batch"""
    max_seq_len: int
    """Longest context length (may be an upper bound)"""

    block_table_tensor: torch.Tensor
    slot_mapping: torch.Tensor

The arithmetic of building once

Let $c$ be the host-side cost of one build() call and $L$ the layer count. Per-layer building costs $L \cdot c$ per step; per-step building costs $c$. The saving is a factor of $L$ — but the number that matters is $L \cdot c$ against the step's device time, because host work that does not overlap is dead time on the GPU.

Anchor it on Llama-3-8B in bf16 on the H100 SXM of §0.4 (3.35 TB/s). A decode step must read all 16.06 GB of weights, so its bandwidth floor is $16.06/3350 = 4.79$ ms. With $L = 32$:

A build costing $c = 50\ \mu$s — plausible for a call that does several cumulative sums and a handful of host-to-device copies over a batch of 256, though unmeasured here — would consume 1.05% of the step if built once and 33.4% if built per layer. For Llama-3-70B under tensor parallelism the device time per rank falls while $L$ rises to 80, so the per-layer version gets worse, not better. That ratio is the entire design rationale, and it is why build() takes the whole batch rather than a layer index.

The sharing is literal — one Python object, aliased into every layer's slot:

            if ubid is None:
                assert isinstance(attn_metadata, dict)
                attn_metadata_dict = attn_metadata
            else:
                assert isinstance(attn_metadata, list)
                attn_metadata_dict = attn_metadata[ubid]

            for layer_name in attn_group.layer_names:
                attn_metadata_dict[layer_name] = attn_metadata_i

The layer retrieves it by name from the forward context (vllm/model_executor/layers/attention/attention.py:L680-L688), then calls self.impl.forward(...) (:L744-L770). Note the granularity: one build per attention group, where a group is layers sharing a backend class, a KV cache spec, and a per-rank Q-head count (vllm/v1/worker/gpu_model_runner.py:L7160-L7175). A Gemma-style model interleaving sliding-window and full attention gets two groups and two builds; a plain Llama gets one.

SGLang's base class (python/sglang/srt/layers/attention/base_attn_backend.py:L36-L57) documents the same lifecycle with different names — init_forward_metadata is the once-per-step build, and its result is stored on the backend instance rather than in a per-layer dict. Because there is one backend object for the whole model, SGLang does not need the layer-name indirection: forward_decode reads self.forward_metadata.decode_wrappers[...] directly (python/sglang/srt/layers/attention/flashinfer_backend.py:L1414-L1416). The cost is that per-layer variation must be handled by a wrapper index (_get_wrapper_idx, dispatching on layer.sliding_window_size == -1) rather than by grouping.

vLLM enumerates its backends in a single enum whose values are import paths, resolved lazily:

    FLASH_ATTN = "vllm.v1.attention.backends.flash_attn.FlashAttentionBackend"
    FLASH_ATTN_DIFFKV = (
        "vllm.v1.attention.backends.flash_attn_diffkv.FlashAttentionDiffKVBackend"
    )
    TRITON_ATTN = "vllm.v1.attention.backends.triton_attn.TritonAttentionBackend"
    TRITON_ATTN_DIFFKV = (
        "vllm.v1.attention.backends.triton_attn_diffkv.TritonAttentionDiffKVBackend"
    )
    ROCM_ATTN = "vllm.v1.attention.backends.rocm_attn.RocmAttentionBackend"
    ROCM_AITER_MLA = "vllm.v1.attention.backends.mla.rocm_aiter_mla.AiterMLABackend"
    ROCM_AITER_TRITON_MLA = (
        "vllm.v1.attention.backends.mla.aiter_triton_mla.AiterTritonMLABackend"
    )
    ROCM_AITER_FA = (
        "vllm.v1.attention.backends.rocm_aiter_fa.AiterFlashAttentionBackend"
    )
    ROCM_AITER_MLA_SPARSE = (
        "vllm.v1.attention.backends.mla.rocm_aiter_mla_sparse.ROCMAiterMLASparseBackend"
    )
    XPU_MLA_SPARSE = "vllm.v1.attention.backends.mla.xpu_mla_sparse.XPUMLASparseBackend"
    TORCH_SDPA = ""  # this tag is only used for ViT
    FLASHINFER = "vllm.v1.attention.backends.flashinfer.FlashInferBackend"
    FLASHINFER_MLA = (
        "vllm.v1.attention.backends.mla.flashinfer_mla.FlashInferMLABackend"
    )

The enum runs to registry.py:L130 and carries a large family of named backends — 19 of them MLA variants — plus a CUSTOM = None placeholder for out-of-tree registration via the register_backend decorator (registry.py:L243-L295). A second enum, MambaAttentionBackendEnum (registry.py:L176-L193), holds six members: five real state-space backends — MAMBA1, MAMBA2, SHORT_CONV, LINEAR, GDN_ATTN — plus the same CUSTOM = None out-of-tree placeholder the main enum carries. They are selected by model architecture rather than by capability, and covered in §7.3.

One enum entry, three kernel generations: FLASH_ATTN and FA4

FLASH_ATTN is a single enum entry that resolves to one of three kernel families at import time. §3.2 covers the FA1 → FA2 → FA3 algorithmic progression; what belongs here is the dispatch, because it is the cleanest capability-driven selection in the tree and because a fourth generation exists at this SHA that the earlier chapters do not cover:

        # 1. default version depending on platform
        if device_capability.major == 9 and is_fa_version_supported(3):
            # Hopper (SM90): prefer FA3
            fa_version = 3
        elif device_capability.major == 10 and is_fa_version_supported(4):
            # Blackwell (SM100+, restrict to SM100 for now): prefer FA4
            fa_version = 4
        else:
            # Fallback to FA2
            fa_version = 2

Four more passes follow (fa_utils.py:L100-L175): a config override from attention_config.flash_attn_version; a hard demotion of FA3 → FA4 on Blackwell ("Cannot use FA version 3 on Blackwell platform, defaulting to FA version 4 if supported, otherwise FA2.", :L113-L118); version-specific demotions for ALiBi and batch-invariant mode; do not assume the latter universally demotes FA3; and — going the other way — an upgrade FA3 → FA4 on SM90 when FA3 cannot handle the shape, with reasons f"FA3 does not support head_size={head_size} on SM90", "Diff-KV with sinks", and "Per-sequence causal (dynamic_causal) requires FA4" (:L132-L159). One backend name, six branches, and the version you get depends on head size and whether the model has sinks.

The eligibility functions show why FA4 is different in kind:

try:
    from . import _vllm_fa3_C  # type: ignore[attr-defined]  # noqa: F401

    FA3_UNAVAILABLE_REASON = None
    FA3_AVAILABLE = True
except ImportError as e:
    FA3_UNAVAILABLE_REASON = str(e)
    FA3_AVAILABLE = False


try:
    import os

    _cute_interface_path = os.path.join(
        os.path.dirname(__file__), "cute", "interface.py"
    )
    if not os.path.exists(_cute_interface_path):
        raise ImportError("vllm.vllm_flash_attn.cute.interface not found")

    FA4_UNAVAILABLE_REASON = None
    FA4_AVAILABLE = True
except (ImportError, ModuleNotFoundError) as e:
    FA4_UNAVAILABLE_REASON = str(e)
    FA4_AVAILABLE = False

FA2 and FA3 are compiled C extensions — _vllm_fa2_C, _vllm_fa3_C — and availability is an import. FA4 is a file existence check for cute/interface.py, a Python module, and the dispatch at flash_attn_interface.py:L388-L402 ends in from vllm.vllm_flash_attn.cute.interface import _flash_attn_fwd. FA4 is written in CuteDSL — CUTLASS's Python DSL — and JIT-compiled, not shipped as a prebuilt .so. Practically: it is gated by whether the build populated that directory, it participates in the JIT warmup of §5 rather than in the C++ build, and its supported-capability set (_is_fa4_supported, :L72-L86) is 9.x, 10.x, or 11.x — wider than FA3's 9.x-only. SGLang reaches the same implementation from sglang.kernels.ops.attention.flash_attn.cute.

SGLang models the same three generations as two separate registry entries, "fa3" and "fa4", both constructing the same class with a version argument (attention_registry.py:L235-L241: return FlashAttentionBackend(runner, fa_impl_ver=4)). And it has no FA2 path at all:

        # Select version
        self.fa_impl_ver = fa_impl_ver
        device_capability = get_device_capability()
        if self.fa_impl_ver == 3:
            from sgl_kernel.flash_attn import (
                flash_attn_varlen_func,
                flash_attn_with_kvcache,
                get_scheduler_metadata,
            )
# ...
        elif self.fa_impl_ver == 4:
            if device_capability[0] == 12:
                from sglang.kernels.ops.attention.flash_attention_v4_sm120 import (
# ...
        else:
            raise ValueError(f"Invalid version: {self.fa_impl_ver=}")

This is the fallback philosophy split in one line. vLLM's FLASH_ATTN degrades: FA3 → FA2 on ALiBi, on batch invariance, on any unsupported combination, so the backend runs on every SM80+ device. SGLang's FlashAttention backend refuses: anything that is not 3 or 4 raises, and off Hopper/Blackwell the default-selection tree never names it — you get flashinfer or triton instead. Neither is wrong; they are different bets about whether a slow-but-correct path is worth carrying. It is also why FA4 is a visible --attention-backend fa4 flag in SGLang and an invisible --attention-config.flash_attn_version=4 sub-field in vLLM.

SGLang's inventory is a flat list of strings, validated as argparse choices:

ATTENTION_BACKEND_CHOICES = [
    # Common
    "triton",
    "torch_native",
    "flex_attention",
    "dsa",
    "nsa",  # Deprecated alias for "dsa"
    "dsv4",
    "compressed",  # Deprecated alias for "dsv4"
    # NVIDIA specific
    "cutlass_mla",
    "fa3",
    "fa4",
    "flashinfer",
    "flashmla",
    "trtllm_mla",
    "cutedsl_mla",
    "tokenspeed_mla",
    "trtllm_mha",
    "dual_chunk_flash_attn",
    "hpc_ops",  # HPC-Ops (https://github.com/Tencent/hpc-ops), Hopper (SM90) only, requires --page-size 64
    # AMD specific
    "aiter",
    "wave",
    # Other platforms
    "intel_amx",
    "ascend",
    "intel_xpu",
]

Twenty-three choices (two are deprecated aliases), backed by 22 factory functions in a plain dict:

ATTENTION_BACKENDS = {}


def register_attention_backend(name):
    def decorator(fn):
        ATTENTION_BACKENDS[name] = fn
        return fn

    return decorator

The structural difference is worth naming. vLLM's enum entry FLASHINFER maps to one class that internally handles MLA-vs-dense by rejecting the wrong case (use_mla != cls.is_mla() is a validation failure); SGLang's string "flashinfer" maps to a factory that branches (attention_registry.py:L42-L66: if not runner.use_mla_backend → FlashInferAttnBackend, else FlashInferMLAAttnBackend). One user-visible name, two classes. That is why SGLang's list is shorter without covering less.

The field is typed as the enum itself, with "auto" mapped to None:

    backend: AttentionBackendEnum | None = None
    """Attention backend to use. Use "auto" or None for automatic selection."""
# ...
    @field_validator("backend", mode="before")
    @classmethod
    def validate_backend_before(cls, value: Any) -> Any:
        """Enable parsing of the `backend` enum type from string.

        The special value "auto" is treated as None, which triggers
        automatic backend selection.
        """
        if isinstance(value, str):
            if value.lower() == "auto":
                return None
            return AttentionBackendEnum[value.upper()]
        return value

Selection then runs per attention layer, at layer construction. get_attn_backend (vllm/v1/attention/selector.py:L105-L192) assembles an AttentionSelectorConfig of 17 fields — head size, dtype, KV dtype, block size, MLA, sinks, sparse, attention type, sliding window, batch invariance, KV connector, PCP, DCP — then applies a per-KV-group override before dispatching to the platform:

    # A per-KV-group override (keyed by KVCacheSpecKind) takes precedence over
    # the global backend; kinds not present in the map fall back to it.
    attention_config = vllm_config.attention_config
    backend = attention_config.backend
    if attention_config.backend_per_kind:
        kind = get_attn_spec_kind(
            use_mla=use_mla,
            has_sliding_window=has_sliding_window,
            attn_type=attn_type,
        )
        backend = attention_config.backend_per_kind.get(kind.value, backend)

    return _cached_get_attn_backend(
        backend=backend,
        attn_selector_config=attn_selector_config,
        num_heads=num_heads,
    )

The result is @cached on the config tuple, so 80 identical layers pay for one resolution. When no backend is named, CudaPlatform walks a hardcoded priority list and takes the first that validates:

    else:
        # SM100f defaults to FlashInfer for TRTLLM causal attention, but its non-causal
        # cutlass path (used for dflash attention) is known to have problems.
        # So prefer FlashAttention when non-causal on SM100f.
        if device_capability.major == 10 and not use_non_causal:
            return [
                AttentionBackendEnum.FLASHINFER,
                AttentionBackendEnum.FLASH_ATTN,
                AttentionBackendEnum.TRITON_ATTN,
                AttentionBackendEnum.FLEX_ATTENTION,
                AttentionBackendEnum.TURBOQUANT,
            ]
        else:
            return [
                AttentionBackendEnum.FLASH_ATTN,
                AttentionBackendEnum.FLASHINFER,
                AttentionBackendEnum.TRITON_ATTN,
                AttentionBackendEnum.FLEX_ATTENTION,
                AttentionBackendEnum.TURBOQUANT,
            ]

On H100 (SM90) the order is FlashAttention first, FlashInfer second. On B200 (SM100) it flips. If every candidate is rejected the engine raises "No valid attention backend found for {device} with {config}. Reasons: {...}" (vllm/platforms/cuda.py:L455-L459) with the full per-backend reason map — which is the single most useful error string in this subsystem, because it tells you exactly which predicate each backend failed.

SGLang chooses differently: not by capability predicates but by a decision tree over hardware and model, written out longhand in ServerArgs._get_default_attn_backend:

        if not use_mla_backend:
            # MHA architecture

            if is_hopper_with_cuda_12_3() and is_no_spec_infer_or_topk_one(
                resolved_view(self)
            ):
                # Note: flashinfer 0.6.1 caused performance regression on Hopper attention kernel
                # Before the kernel is fixed, we choose fa3 as the default backend on Hopper MHA
                # ref: https://github.com/sgl-project/sglang/issues/17411
                return "fa3"
            elif (
                is_sm100_supported()
                and is_no_spec_infer_or_topk_one(resolved_view(self))
                and (
                    self.speculative_algorithm is None
                    or self.speculative_eagle_topk is not None
                )
            ):
                # trtllm_mha requires equal K/V row widths; fa4 carries
                # v_head_dim through.
                if model_config.has_asymmetric_kv:
                    return "fa4"
                return "trtllm_mha"
            elif is_hip():
                return "aiter"
            elif is_mps():
                return "torch_native"
            else:
                # FlashInfer does not support attention sinks.
                if is_flashinfer_available() and not model_config.has_attention_sinks:
                    return "flashinfer"
                return "triton"

Three things are different from vLLM in kind, not degree. First, there is no fallback chain: this returns one string and the factory either constructs or asserts (create_flashattention_v3_backend at attention_registry.py:L209-L232 literally does assert (major == 8 and not runner.use_mla_backend) or major == 9). Second, capability checks live inside each factory and each backend's __init__, not in a shared predicate table — which is why SGLang's failures surface as assertions with prose rather than as a reason map. Third, SGLang lets you split the choice by phase: --prefill-attention-backend and --decode-attention-backend take priority over --attention-backend, and when they differ the two backends are composed:

    elif resolved.decode != resolved.prefill:
        from sglang.srt.layers.attention.hybrid_attn_backend import (
            HybridAttnBackend,
        )

        # Compose the two full-attention backends first, then apply model-level
        # wrappers once.  Wrapping each child independently duplicates the
        # linear/sparse side backend for hybrid models (for example, two GDN
        # dispatchers for Qwen3.5 when prefill and decode use different MHA
        # backends), duplicating initialization and associated state while only
        # one side backend can be active in a forward pass.
        attn_backend = attn_backend_wrapper(
            model_runner,
            HybridAttnBackend(
                model_runner=model_runner,
                decode_backend=_build_full_attention_backend_from_str(
                    model_runner=model_runner,
                    backend_str=resolved.decode,
                    init_new_workspace=init_new_workspace,
                ),

vLLM reaches the same place from the other direction: backend_per_kind splits by KV-cache group kind (mla_attention, sliding_window, …), not by phase. Both are answers to "one model, two kernels", chosen along different axes — SGLang along the prefill/decode axis because its backends are phase-dispatched objects, vLLM along the KV-spec axis because its metadata builders are already grouped that way.

FlashInfer is the cleanest expression of the whole idea, because its API is explicitly two-phase. A wrapper object owns a scratch workspace; plan() does host-side layout and scheduling work against a batch description; run() launches the kernel with the plan already resident. vLLM calls plan() inside build() — once per step — and run() inside impl.forward() — once per layer.

What plan() actually computes

Before the call, the builder converts vLLM's block table into FlashInfer's CSR page layout. That is _compute_flashinfer_kv_metadata (vllm/v1/attention/backends/flashinfer.py:L1255-L1310): a np.cumsum of per-request page counts into paged_kv_indptr, a Triton kernel (_copy_page_indices_kernel, :L2656-L2678) that gathers each request's row of the block table into the flat paged_kv_indices array, and a modulo to fill paged_kv_last_page_len. Then:

                    prefill_wrapper.plan(
                        qo_indptr=qo_indptr_prefill_cpu,
                        paged_kv_indptr=paged_kv_indptr_prefill_cpu,
                        paged_kv_indices=paged_kv_indices,
                        paged_kv_last_page_len=paged_kv_last_page_len_prefill_cpu,
                        num_qo_heads=self.num_qo_heads,
                        num_kv_heads=self.num_kv_heads,
                        head_dim_qk=self.head_dim,
                        page_size=self.page_size,
                        causal=attn_metadata.causal,
                        sm_scale=self.sm_scale,
                        window_left=self.window_left,
                        logits_soft_cap=self.logits_soft_cap,
                        q_data_type=self.q_data_type_prefill,
                        kv_data_type=self.kv_cache_dtype,
                        o_data_type=o_dtype,
                        fixed_split_size=self.prefill_fixed_split_size,
                        disable_split_kv=self.disable_split_kv,
                    )
                attn_metadata.prefill = FIPrefill(wrapper=prefill_wrapper)

Note what is in that argument list: no tensors of data, only shape and policy. Sequence layout, head counts, head dim, page size, causality, softmax scale, window, soft cap, dtypes, and the split-K policy from §3.3. Inside, FlashInfer uses this to pick a kernel specialisation, compute the CTA-to-work-tile assignment for the ragged batch, and size its split-K temporaries in the workspace buffer. SGLang's monkey-patched fast_prefill_plan is explicit about the last step — self._plan_info = self._cached_module.plan(*args) (python/sglang/srt/layers/attention/flashinfer_backend.py:L286) — and about why it exists:

    """Sync-free ``BatchPrefillWithPagedKVCacheWrapper.plan`` for the EAGLE
    draft-extend CUDA graph (FlashInfer fa2, cuda-graph mode only).

    Upstream plan() always does qo/paged_kv/last_page_len ``.to("cpu")`` to build
    its host scheduling metadata, a blocking D2H that drains the GPU queue every
    replay. The caller passes host-known qo/kv layout in, so we call the underlying
    ``_cached_module.plan`` directly with no readback; the ``_plan_info`` produced
    is identical to plan()'s.

The quoted EAGLE path performs a blocking device-to-host copy during planning. Other paths, including specialized trtllm decode, have different metadata work and may avoid plan(). Even one synchronization per short step can be material; measure it separately rather than calling it a rounding error.

The wrappers

vLLM imports five wrapper classes; SGLang instantiates four kinds:

The dispatch inside FLASHINFER

Choosing FLASHINFER starts a second, per-batch selection. use_trtllm_attention (vllm/utils/flashinfer.py:L445-L530) decides per phase and per step, and its auto-detection rules are specific enough to be surprising:

    if force_use_trtllm is None:
        # CLI argument not set - use auto-detection
        if is_prefill:
            # Prefill auto-detection
            use_trtllm = kv_cache_dtype == "auto"
        elif (
            current_platform.is_device_capability(90)
            or current_platform.is_device_capability_family(120)
        ) and kv_cache_dtype.startswith("fp8"):
            # SM90/SM12x + FP8 KV cache: prefer the XQA decode kernel.
            use_trtllm = True
        else:
            # Decode auto-detection
            use_trtllm = num_tokens <= 256 and kv_cache_dtype == "auto"

num_tokens <= 256. A decode batch of 256 tokens uses trtllm-gen; 257 uses FlashInfer's native decode kernel. If you sweep batch size across that boundary you will see a discontinuity in your latency curve that has nothing to do with occupancy. Earlier in the same function, FP8 queries, attention sinks, and speculative decoding each force trtllm on unconditionally.

JIT and what it costs at startup

FlashInfer ships kernels as JIT-compiled templates or as pre-built cubins downloaded from NVIDIA's artifactory. vLLM's availability check encodes both:

@functools.cache
def has_flashinfer() -> bool:
    """Return `True` if flashinfer-python package is available."""
    # Use find_spec to check if the module exists without importing it
    # This avoids potential CUDA initialization side effects
    if importlib.util.find_spec("flashinfer") is None:
        logger.debug_once("FlashInfer unavailable since package was not found")
        return False
    # When not using flashinfer cubin,
    # Also check if nvcc is available since it's required to JIT compile flashinfer
    if not has_flashinfer_cubin() and shutil.which("nvcc") is None:
        logger.debug_once(
            "FlashInfer unavailable since nvcc was not found "
            "and not using pre-downloaded cubins"
        )
        return False
    return True

Without the flashinfer-cubin package you need nvcc on PATH, and every kernel specialisation is compiled on first use. Worse, supports_trtllm_attention calls has_nvidia_artifactory(), which does an actual HTTP GET with a 5-second timeout (vllm/utils/flashinfer.py:L367-L391) — an air-gapped node silently loses every trtllm path. vLLM pulls the compilation forward into a warmup phase so it is not paid on the first request:

    if worker.vllm_config.kernel_config.enable_jit_warmup:
        logger.info("JIT kernel warmup starting.")
        jit_warmup_start = time.perf_counter()
        try:
            worker.model_runner.jit_warmup_registry.warmup()
        except Exception:
            logger.exception(
                "JIT kernel warmup failed after %.2fs.",
                time.perf_counter() - jit_warmup_start,
            )
            raise
        logger.info(
            "JIT kernel warmup finished in %.2fs.",
            time.perf_counter() - jit_warmup_start,
        )

That log line is the number to look at; it is printed on every start. On top of it, flashinfer_autotune(runner) (:L252, gated by kernel_config.enable_flashinfer_autotune, default on for SM90/SM100 profiles) benchmarks kernel variants and writes a cache directory set by VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR. Cached artifacts can avoid repeated compilation or tuning, but a new process still allocates buffers, initializes contexts, plans metadata, and warms or captures selected shapes.

A captured CUDA graph retains recorded launch arguments and buffer addresses. Metadata buffers touched by replay must have stable storage and updated contents. Replacing a buffer without recapture or supported graph updates can leave stale addresses; the outcome may be wrong values or a memory-access failure. Compilation caches do not replace per-process allocation, planning, and warmup.

vLLM declares each builder's tolerance as a four-level enum:

class AttentionCGSupport(Enum):
    """Constants for the cudagraph support of the attention backend
    Here we do not consider the cascade attention, as currently
    it is never cudagraph supported."""

    ALWAYS = 3
    """Cudagraph always supported; supports mixed-prefill-decode"""
    UNIFORM_BATCH = 2
    """Cudagraph supported for batches the only contain query lengths that are
    the same, this can be used for spec-decode
        i.e. "decodes" are 1 + num_speculative_tokens"""
    UNIFORM_SINGLE_TOKEN_DECODE = 1
    """Cudagraph supported for batches the only contain query_len==1 decodes"""
    NEVER = 0
    """NO cudagraph support"""

The consequences run through everything above. FlashInfer allocates one decode wrapper per captured batch size, each bound to fixed slices of persistent buffers:

            if use_cudagraph:
                paged_kv_indptr = self.paged_kv_indptr.gpu[: batch_size + 1]
                paged_kv_indices = self.paged_kv_indices.gpu
                paged_kv_last_page_len = self.paged_kv_last_page_len.gpu[:batch_size]
            else:
                paged_kv_indptr = None
                paged_kv_indices = None
                paged_kv_last_page_len = None
# ...
            decode_wrapper = BatchDecodeWithPagedKVCacheWrapper(
                self._get_workspace_buffer(),
                get_kv_cache_layout(),
                use_cuda_graph=use_cudagraph,
                paged_kv_indptr_buffer=paged_kv_indptr,
                paged_kv_indices_buffer=paged_kv_indices,
                paged_kv_last_page_len_buffer=paged_kv_last_page_len,

SGLang does the identical thing (flashinfer_backend.py:L1046-L1059), binding self.kv_indptr[i][: num_tokens + 1] and self.cuda_graph_kv_indices[i] into the wrapper. Its base class states the constraint in prose: "some backends expose metadata tensors to kernels across graph breaks, so the captured graph depends on those tensor addresses" (base_attn_backend.py:L124-L129), and splits init_forward_metadata into an out-of-graph half for host work and an in-graph half whose lint contract forbids .item(), .cpu(), and dynamic-shape torch.empty().

FlashAttention shows the same pattern for a different tensor — its AOT scheduler metadata is copied into a persistent buffer and the tail zeroed, because stale entries would let thread blocks write outside the output:

    def _store_scheduler_metadata(
        self, scheduler_metadata: torch.Tensor | None
    ) -> torch.Tensor | None:
        if self.use_full_cuda_graph and scheduler_metadata is not None:
            n = scheduler_metadata.shape[0]
            assert self.scheduler_metadata is not None
            self.scheduler_metadata[:n] = scheduler_metadata
            # NOTE(woosuk): We should zero out the rest of the scheduler
            # metadata to guarantee the correctness. Otherwise, some thread
            # blocks may use the invalid scheduler metadata and overwrite the
            # output buffer.
            self.scheduler_metadata[n:] = 0
            return self.scheduler_metadata[:n]

Graphs themselves — capture, replay, the piecewise/full modes — are §8.1. What matters here is that a backend's AttentionCGSupport level is a hard cap on the engine's graph mode, and that a backend which cannot promise stable addresses cannot be graphed at all.

Llama-3-70B, TP8 on H100, --attention-backend FLASHINFER, batch 32, one token each, all sequences at 4096 context, page size 16, bf16 KV.

1. Startup, once per layer class. Attention.__init__ calls get_attn_backend(head_size=128, dtype=bfloat16, kv_cache_dtype="auto", ...) (selector.py:L105). It builds an AttentionSelectorConfig, finds no backend_per_kind entry, and hits the @cached _cached_get_attn_backend (:L195). CudaPlatform.get_attn_backend_cls takes the explicit-backend branch: FlashInferBackend.validate_configuration checks head size 128 against [64, 128, 256, 512], capability 9.0 against >= (8,0) and <= (12,1), dtype against [float16, bfloat16]. Empty reason list → logger.info("Using %s backend."). get_required_kv_cache_layout() returns None on SM90 (it would return "HND" on SM100), so no layout override.
2. Startup, once. initialize_attn_backend (gpu_model_runner.py:L7150) groups all 80 layers into one AttentionGroup — same backend class, same KV spec, same 8 Q heads per rank — then initialize_metadata_builders constructs one FlashInferMetadataBuilder. Its __init__ (flashinfer.py:L701) allocates the workspace, sizes paged_kv_indices at max_num_reqs * cdiv(max_model_len, 16), resolves can_use_trtllm_attention(num_qo_heads=8, num_kv_heads=1, is_prefill=False), and calls infer_global_hyperparameters to assert all 80 layers agree on window_left, logits_soft_cap, and sm_scale.
3. Step $N$, host. _prepare_inputs builds CommonAttentionMetadata: query_start_loc = [0,1,2,…,32], seq_lens = [4096]*32, block_table_tensor shape [32, 256], slot_mapping shape [32].
4. Step $N$, build (once). _build_attention_metadata (:L2355) calls builder.build(common_prefix_len=0, cm). Inside: split_decodes_and_prefills returns (32, 0, 32, 0); use_trtllm_attention(..., num_tokens=32, ...) returns True because $32 \le 256$ and the KV dtype is auto; decode_with_flashinfer_trtllm_api is therefore True, so needs_paged_kv_indices is False and the CSR conversion is skipped entirely — trtllm-gen reads the block table directly. The metadata gets a FlashInferTrtllmAPIDecode holding block_tables[:32], seq_lens[:32], and max_seq_len. No plan() call on this path at all.
5. Step $N$, share. The loop at :L2627-L2628 writes the one metadata object into all 80 entries of attn_metadata.
6. Step $N$, run (×80). Each layer's unified_attention_with_output custom op calls get_attention_context(layer_name) → the shared object → FlashInferImpl.forward (:L1904), which reads attn_metadata.decode.kernel, sees FlashInferDecodeKernel.TRTLLM_GEN, and launches trtllm_batch_decode_with_kv_cache. Eighty launches, one plan's worth of host work.

Raise the batch to 300 tokens and step 4 flips: use_trtllm_attention returns False, needs_native_paged_decode becomes True, the Triton CSR gather runs, and fast_plan_decode plans a BatchDecodeWithPagedKVCacheWrapper. Same flag, different kernel.

Reproduce the selection log without a GPU-heavy run by asking for something impossible and reading the reason map:

# 1. See every backend name the enum knows, with no GPU:
python -c "from vllm.v1.attention.backends.registry import AttentionBackendEnum as B; print(len(list(B)), [b.name for b in B])"

# 2. On a NON-SM90 device, HPC_ATTN can trigger a capability error.
# On SM90 this is not an impossible choice; inspect actual supported constraints:
vllm serve meta-llama/Llama-3.1-8B-Instruct --attention-backend HPC_ATTN
#    -> ValueError: Selected backend AttentionBackendEnum.HPC_ATTN is not valid
#       for this configuration. Reason: ['compute capability not supported', ...]

# 3. Watch the auto path pick, and see what it considered:
vllm serve meta-llama/Llama-3.1-8B-Instruct 2>&1 | grep -E "attention backend|Using .* backend"
#    -> Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', ...]

# 4. Cross the trtllm decode threshold and diff the logs:
vllm bench latency --model meta-llama/Llama-3.1-8B-Instruct --attention-backend FLASHINFER \
  --batch-size 256 --input-len 1024 --output-len 8 2>&1 | grep -i trtllm
vllm bench latency --model meta-llama/Llama-3.1-8B-Instruct --attention-backend FLASHINFER \
  --batch-size 257 --input-len 1024 --output-len 8 2>&1 | grep -i trtllm

# 5. SGLang: print the resolved pair rather than guessing.
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct 2>&1 \
  | grep -E "Attention backend not specified|hybrid attention backend"

Command 3's message comes from platforms/cuda.py:L490-L498 and lists every backend that passed validation, in priority order — the fastest way to learn what your configuration actually permits.

• vLLM PR #47361 — "Delete PagedAttention", commit d715b3aa1e. The removal that makes this chapter the whole story on CUDA.
• FlashInfer and its paper, "FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving" (Ye et al., 2025) — the plan/run split and the block-sparse formulation of paged KV, from the source.
• flashinfer#1675 — split-tile sizing, cited by SGLang's deterministic-inference path for SGLANG_FLASHINFER_PREFILL_SPLIT_TILE_SIZE.
• sglang#17411 — the Hopper regression that made fa3, not flashinfer, SGLang's MHA default on SM90. The comment citing it is in _get_default_attn_backend.
• flashinfer#3578 and #3620 — the SM90 sliding-window crash and the SM75 breakage that raised vLLM's compute-capability floor to 8.0.
• vllm#35807 — the benchmark thread cited in _get_backend_priorities for the SM100 sparse-MLA ordering.
• CUTLASS CuTeDSL — the Python DSL FA4 is written in, and the reason FA4_AVAILABLE is a file check rather than an extension import. See also §8.3.
• SGLang: attention backends — the user-facing matrix, referenced from server_args.py itself.
• Next: §3.5 on head-sharing schemes; §7.2 owns the 19 MLA backends listed above; §8.1 owns graph capture and replay.