CUDA graphs and kernel-launch overhead

Take the number this book has been carrying since §0.4: Llama-3-8B in bf16 is 15.0 GB of weights, and reading them once at H100 SXM's 3.35 TB/s costs 4.48 ms. That is the bandwidth-bound floor of a batch-1 decode step. Nothing you do to the kernels gets under it.

Now count the launches it takes to get there. Read vLLM's Llama decoder layer (vllm/model_executor/models/llama.py:L310-L327) and its two children (L221-L231 for attention, L115-L119 for the MLP) and enumerate the ops that actually reach the driver:

Thirty-two layers gives 320 kernels. Add the embedding gather, the final norm, the LM-head GEMM and the sampling chain (temperature scale, penalties, top-k, top-p, softmax, multinomial, gather) and you land at roughly 330 launches per decoded token at TP=1. At TP=8 each layer also carries two all-reduces — after o_proj and after down_proj — so the count rises to about 395, while the GPU work falls by 8× because the weights are sharded.

The launch-overhead number, finally measured on H100

§0.3 raised this as an open question and pointed here: the only published per-launch figures were NVIDIA's Tesla V100 / CUDA 10.1 numbers (2.9 / 9.6 / 3.8 / 3.4 µs), and the microsecond scale was sound but the value on modern hardware was unknown. There is now a better source. Vellaisamy et al., Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures (ISPASS 2025, arXiv:2504.11750), Table V, measures cudaLaunchKernel with a null kernel on PyTorch 2.4.1 / CUDA 12.6 / driver 560.35.03:

Deriving the gap

A launch is asynchronous, so launch cost only hurts when the CPU cannot stay ahead of the GPU. Define the host budget per kernel as the GPU time available divided by the number of launches:

where $T_{\text{GPU}}$ is the device-side time of one decode step and $N_{\text{kernels}}$ the launches in it. If the host cost per kernel exceeds $B_{\text{host}}$, the GPU starves and the step is launch-bound. Two cases, both arithmetic:

The TP=8 row compares a submission estimate with a lower bound on weight-read time, not total GPU work. Collectives, attention, kernel efficiency and synchronization remain. Thus it proves neither launch-bound execution nor 40% GPU idle time. A GPU critical path longer than 0.94 ms could hide much of that submission work.

Likewise, an empty-kernel duration is not an independently additive dispatch constant for every useful kernel. Measure eager and graph timelines with the same capture boundary and inputs before assigning an irreducible per-node cost.

A CUDA graph is a recording. You put a stream into capture mode, run the forward pass exactly as normal, and every kernel launch that would have gone to the GPU instead becomes a node in a directed acyclic graph, with its grid, block, shared-memory size and argument bytes — pointers included — copied verbatim. Nothing executes. When you close capture you hold a graph; you then instantiate it into an executable object, which is the driver pre-resolving the node list into something it can submit in one shot. From then on, replay() submits the whole DAG with a single API call.

Three phases, three very different costs. Capture is a real forward pass with recording overhead. Instantiation is a one-time driver walk that builds the executable. Replay is the only one on the hot path, and it is roughly one launch's worth of host work for the entire model.

Ordinary replay in these runners preserves captured arguments. Rebinding a Python tensor does not update its recorded device address. This is the runner's contract, not a universal CUDA restriction: CUDA supports constrained graph/node parameter updates and conditional nodes. Those facilities require explicit integration and do not rerun arbitrary captured Python.

1. Addresses are frozen, so buffers must be persistent and written in place

The capture path copies each kernel's argument bytes, which for a PyTorch op means the data_ptr() of every tensor. A replay reads whatever is at those addresses now. So the model runner must allocate one set of buffers at startup, at the maximum shape, and each step copy the current values into them rather than allocating fresh tensors.

SGLang makes this an explicit registry with a per-field policy for what happens to the padded tail:

class PaddingPolicy(Enum):
    """How to handle ``raw_n < padded_n`` for a slot.

    KEEP_PAD      — Leave the padded region as-is (caller proves the
                    padded tail will not be read).
    FILL_SENTINEL — Reset the padded region to ``slot.pad_value`` before
                    copy (e.g. ``seq_lens`` filled with
                    ``seq_len_fill_value``).
    ZERO          — Reset the padded region to ``0`` (e.g.
                    ``out_cache_loc`` / ``req_pool_indices`` — padded
                    rows must point at slot 0 so dummy attention reads
                    land harmlessly).
    FOREACH_COPY  — Always copy ``raw_n`` from src; padded region is
                    left as whatever the previous replay (or the init
                    zeros) wrote. Caller is responsible for proving
                    safety.
    FILL_ONCE     — Fill the whole buffer to ``pad_value`` once at alloc;
                    never reset per iter (e.g. ``encoder_lens`` init to
                    ``encoder_len_fill_value``, copied head-only with the
                    tail kept).
    """

vLLM takes the opposite structural choice: its CUDAGraphWrapper deliberately owns no buffers — "CUDAGraphWrapper does not store persistent buffers or copy any runtime inputs into that buffers for replay. We assume implementing them is done outside of the wrapper" (vllm/compilation/cuda_graph.py:L161-L164). The persistent buffers live in the model runner instead. What the wrapper does keep is an audit trail:

        if self.is_debugging_mode:
            # check if the input addresses are the same
            new_input_addresses = [
                x.data_ptr() for x in args if isinstance(x, torch.Tensor)
            ]
            assert new_input_addresses == entry.input_addresses, (
                f"Input addresses for cudagraphs are different "
                f"during replay. Expected {entry.input_addresses}, "
                f"got {new_input_addresses}"
            )

That assertion only fires under VLLM_LOGGING_LEVEL=DEBUG. Without it, an address mismatch is not an error — it is silently wrong output. Memorise the flag.

The same constraint reaches into the attention backends, which is why §3.4 owns AttentionCGSupport — the four-level enum (ALWAYS / UNIFORM_BATCH / UNIFORM_SINGLE_TOKEN_DECODE / NEVER) by which each metadata builder declares how much capture it tolerates. This chapter uses it only as an input to mode resolution.

2. These runners use shape buckets for stable replay

Recorded grids and argument layouts must remain compatible. Rather than updating executable nodes on every request, the cited runners capture a ladder and pad to an eligible bucket:

                    cudagraph_capture_sizes = [
                        i for i in [1, 2, 4] if i <= max_cudagraph_capture_size
                    ]
                if max_cudagraph_capture_size >= 8:
                    # Step size 8 for small batch sizes, up to 256(not included)
                    cudagraph_capture_sizes += list(
                        range(8, min(max_cudagraph_capture_size + 1, 256), 8)
                    )
                if max_cudagraph_capture_size >= 256:
                    # Step size 16 for larger batch sizes
                    cudagraph_capture_sizes += list(
                        range(256, max_cudagraph_capture_size + 1, 16)
                    )

with max_cudagraph_capture_size = min(max_num_seqs × decode_query_len × 2, 512) (1024 on data-center Blackwell), per vllm/config/compilation.py:L692-L706. On an H100 with a default max_num_seqs that gives 51 buckets: [1, 2, 4, 8, 16, …, 248, 256, 272, …, 512].

SGLang's ladder is a different shape, from python/sglang/srt/server_args.py:L5179-L5185: [1, 2, 4, 8, 12] + range(16, 257, 8) + range(272, 512, 16) + range(512, max_bs + 1, 32), with max_bs defaulting to 256 at TP<4 on an 80 GB card and 512 at TP≥4 (server_args.py:L4894-L4903). That is 36 buckets at TP=1.

The extra 12 rung matters more than it looks. A batch of 9 pads to 12 in SGLang (25% of rows fake) and to 16 in vLLM (43.8% fake) — the worst relative waste anywhere on the vLLM ladder.

The weight-only GEMM model gives intensity approximately B for bf16. Below an ideal ridge, its bandwidth term stays constant with B, but real latency can still change through tiling, occupancy, activation traffic and launch behavior. Padding 9 to 16 is not guaranteed free. Null sequence lengths and reserved block IDs protect padded attention rows; other padded operators can still execute work:

            # Fill unused block table entries with NULL_BLOCK_ID (null block)
            # for CUDAGraph padding. Block 0 is reserved for padding.
            blk_table_tensor[num_reqs:num_reqs_padded].fill_(NULL_BLOCK_ID)

Padding 300 to 304 adds about 64 GFLOPs under the 2BN convention, whose peak-compute time is about 65 microseconds. Neither that number nor a sub-ridge batch establishes the actual padding penalty. Choose buckets from measured workload-weighted latency and memory cost.

3. No host-side control flow, which is why prefill mostly is not captured

Captured Python branches are not re-evaluated during ordinary replay. Device values may change inside stable buffers; conditional CUDA graph nodes are a separate API capability. Prefill's variable lengths and metadata create more dispatch cases, but do not make it inherently uncapturable. SGLang's opt-in path captures eligible prefill buckets:

    # full for prefill captures one whole-forward graph per num_tokens
    # bucket with a fixed request-slot count; replay pads num_tokens up to
    # the nearest captured bucket. Opt-in: the padding waste is the
    # operator's call.
    Phase.PREFILL: (
        Backend.FULL,
        Backend.BREAKABLE,
        Backend.TC_PIECEWISE,
        Backend.DISABLED,
    ),

Both engines have converged on the same shape — a per-phase mode, resolved at startup against what the attention backend tolerates — and disagree on almost every name.

vLLM's runtime dispatch is a lookup, not a computation. CudagraphDispatcher precomputes an array from every batch size to its padded rung —

        self._bs_to_padded_graph_size: list[int] = [0] * (max_size + 1)
        for end, start in zip(
            capture_sizes + [max_size + 1],
            [0] + capture_sizes,
        ):
            for bs in range(start, end):
                if bs == start:
                    self._bs_to_padded_graph_size[bs] = start
                else:
                    self._bs_to_padded_graph_size[bs] = end

— and every step it builds a BatchDescriptor (padded token count, request count, uniform flag, LoRA state) and looks it up in one of two key sets. Miss both and you run eager:

        if CUDAGraphMode.FULL in allowed_modes:
            # check if key exists for full cudagraph
            batch_desc_to_check = batch_desc
            if batch_desc_to_check in self.cudagraph_keys[CUDAGraphMode.FULL]:
                return CUDAGraphMode.FULL, batch_desc_to_check

        if CUDAGraphMode.PIECEWISE in allowed_modes:
            # also check if the relaxed key exists for more "general"
            # piecewise cudagraph
            batch_desc_to_check = replace(batch_desc, num_reqs=None, uniform=False)
            if batch_desc_to_check in self.cudagraph_keys[CUDAGraphMode.PIECEWISE]:
                return CUDAGraphMode.PIECEWISE, batch_desc_to_check

        assert CUDAGraphMode.NONE in allowed_modes, (
            f"No matching cudagraph found and NONE is not in "
            f"allowed_modes={allowed_modes}"
        )
        return CUDAGraphMode.NONE, BatchDescriptor(num_tokens)

The replace(batch_desc, num_reqs=None, uniform=False) line is the interesting one: a piecewise graph is keyed only on token count, so a uniform decode batch can fall back to a more general graph. A full graph cannot — the dispatcher's own comment says "FULL mode needs exact num_reqs because FA3's scheduler_metadata computation depends on it" (vllm/v1/cudagraph_dispatcher.py:L199-L202). Piecewise buys generality; full buys the last of the launch overhead. §8.2 owns why the pieces exist at all.

Startup: capture is why your server takes a minute to come up

Capture runs once per bucket, and both engines walk the ladder backwards for the same reason:

        # Trigger CUDA graph capture for specific shapes.
        # Capture the large shapes first so that the smaller shapes
        # can reuse the memory pool allocated for the large shapes.

        # Reverse so cuda graphs share memory better.
        capture_range = (
            tqdm.tqdm(list(reversed(self.capture_bs)))
            if get_parallel().tp_rank == 0

Each bucket is warmed first — SGLang hardcodes two passes (runner_backend/full_cuda_graph_backend.py:L105-L107), vLLM does one. Do not read that number off the dataclass: cudagraph_num_of_warmups declares a default of 0 (vllm/config/compilation.py:L643), but startup overwrites it with cudagraph_num_of_warmups = 1 on every path that is not enforce_eager (vllm/config/vllm.py:L1601-L1608), so the live value on a serving process is 1. On top of that vLLM runs a separate kernel_warmup() before capture (gpu_worker.py:L726-L728) — and garbage collection is frozen throughout (vllm/compilation/cuda_graph.py:L286-L292: "running gc again and again across layers will make the cudagraph capture very slow"). vLLM's own estimate of the total is in a source comment:

        cuda_graph_size = start_free_gpu_memory - end_free_gpu_memory
        # This usually takes 5~20 seconds.
        logger.info_once(
            "Graph capturing finished in %.0f secs, took %.2f GiB",
            elapsed_time,
            cuda_graph_size / (1 << 30),
        )

Memory: the graph pool comes out of the KV cache

The pool is real, and vLLM subtracts it from the KV budget before the block manager ever sees it — which is the direct link back to §2.6:

        self.available_kv_cache_memory_bytes = (
            self.requested_memory
            - profile_result.non_kv_cache_memory
            - cudagraph_memory_estimate_applied
        )

This is a measured-sample extrapolation: two large captures estimate a shared cost and a marginal cost. It can differ from actual allocations across the complete bucket set:

                    first_capture = mem_samples[0]
                    # Use at least 1 MiB per graph for driver overhead
                    per_graph = max(
                        mem_samples[1] if len(mem_samples) > 1 else 0, 1 << 20
                    )

                    shared_memory_estimate[mode] = first_capture
                    per_graph_estimate[mode] = per_graph * (len(descs) - 1)

The estimator is largest-capture cost plus (N-1) times a marginal estimate floored at 1 MiB. For 51 buckets that reserves at least 50 MiB in the marginal term; it does not prove 50 MiB of actual pure driver bookkeeping. Compare reserved estimates with observed allocation.

Under the default FULL_AND_PIECEWISE there are two modes, and they are combined asymmetrically — max(shared_memory_estimate.values()) + sum(per_graph_estimate.values()) (gpu_model_runner.py:L6926-L6931) — because the two mode's graphs overlay in one pool at runtime and are never replayed concurrently, so the shared part must not be double-counted while the per-graph parts must. SGLang accounts the same budget but splits it six ways, one per capture family:

GRAPH_MEMORY_USAGE_KEYS = (
    "prefill",
    "decode",
    "target_verify",
    "draft_prefill",
    "draft_decode",
    "draft_extend",
)

Four of those six are speculative decoding. EAGLE does not share the target model's graphs: it has its own runners (python/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py, eagle_draft_extend_cuda_graph_runner.py) because the draft forward has a different shape and a different number of steps — see §6.4. Turning on speculation multiplies your capture time and your graph pool.

The standard one-line explanation — CUDA graphs remove CPU launch overhead — is incomplete, and SGLang's all-reduce config proves it. §5.4 noted the anomaly: the tuned dispatch table carries two different sets of size thresholds, one for graph mode and one for eager mode. If capture only removed host work, the crossover between two collective algorithms — both of which pay the same host cost — would not move.

        3: config(
            3,
            graph=(1.250 * MB, 1.250 * MB, 128.0 * MB),
            eager=(3.000 * MB, 3.000 * MB, 16.00 * MB),
        ),
        4: config(
            4,
            graph=(384.0 * KB, 384.0 * KB, 128.0 * MB),
            eager=(896.0 * KB, 896.0 * KB, 32.00 * MB, Range(0, 32 * MB)),
        ),
        5: config(
            5,
            graph=(192.0 * KB, 192.0 * KB, 32.00 * MB),
            eager=(384.0 * KB, 384.0 * KB, 32.00 * MB, Range(0, 32 * MB)),
        ),

That is the Hopper table (_sm90_configs), world sizes 3, 4 and 5. Three things are systematic across the whole file. First, the one-shot-push threshold is lower under capture in every single row where the two differ — SM90 at TP=3, 4, 5: 1.25 vs 3.0 MB, 384 vs 896 KB, 192 vs 384 KB; and SM100 at TP=2 through 8, all seven rows: 8 vs 16 MB, 4 vs 8, 2.25 vs 3.0, 1.5 vs 2.0, 1.0 vs 1.25, 0.625 vs 1.0, 0.5 vs 0.75 MB (configs/custom_all_reduce_v2.py:L129-L172). Ten rows, one direction — that is not noise. Second, the two-shot ceiling before NCCL takes over is higher under capture: 128 MB versus 16 MB at SM90 TP=3 and versus 32 MB at TP=4. Third, multicast is restricted or disabled under capture at exactly the sizes where it is enabled in eager mode — SM90 TP=4 has no graph mc range at all against an eager Range(0, 32 * MB), and SM100 TP=8 has Range(8 * MB, 128 * MB) against Range(0, 128 * MB).

The mechanism is in the JIT'd kernel, and it is a data-path change, not a launch-count change:

    const auto local_workspace = data.pull_workspaces[data.rank];
    if (algo == "1shot_pull") {
      // first copy to workspace
      if (!use_graph) cuda_memcpy(local_workspace, in_.data_ptr());
      const auto kernel = (pull_mode == Graph) ? kernel_pull<1, Graph>
                          : pull_mode == Eager ? kernel_pull<1, Eager>
                                               : kernel_pull<1, Multicast>;
      // then launch kernel to reduce and write to output
      LaunchKernel(num_blocks, choose_block_size(num_vecs), stream)  //
          .enable_pdl(kUsePDL)(kernel, params);
    } else /* 2shot_pull */ {
# ...
      // first copy to workspace
      if (!use_graph) cuda_memcpy(local_workspace, in_.data_ptr());
# ...
      // finally copy from workspace to output
      if (!use_graph) cuda_memcpy(out.data_ptr(), local_workspace);
    }

A pull-based all-reduce needs every rank's contribution to sit at an address its peers can read. In eager mode the input tensor is wherever the allocator put it this step, so the rank must first cuda_memcpy it into symmetric memory — and two-shot must copy the reduced result back out. Under capture the input's address is fixed by definition, so it can be exported through cudaIpc handles after capture and written into a device-side pointer table that the captured kernel dereferences at replay: "CUDA-graph inputs are exchanged from Python after capture … and written into a device-side pointer table (graph_params); the kernel captured in the graph dereferences its row at replay time" (custom_all_reduce_v2.py:L13-L16). The communicator calls it what it is — "graph zero-copy input registration" (L171-L173).

Every threshold shift in the table falls out of that one column. Push gets nothing from capture: it never touches the workspace, so there is no copy to elide, and the kernel refuses a graph row outright — RuntimeCheck(!use_graph, "Push mode doesn't have graph mode optimization") at custom_all_reduce.cuh:L577-L578. (Push is still chosen under capture when the message is small enough; _pick_algo returns it with _PullMode.EAGER, which is what keeps that RuntimeCheck from firing.) So capture makes the pull family cheaper by $\Theta(S)$ while leaving push unchanged, and the push/pull crossover moves down. Removing an $\Theta(S)$ term from two-shot is also why it keeps beating NCCL four times further out under capture. And multicast still routes through the workspace (its address is multicast_ptr + pull_ws_offset, custom_all_reduce_v2.py:L253-L254), so it cannot use the zero-copy path — which is why zero-copy two-shot wins the small sizes under capture and multicast is pushed up to 8 MB.

The last twist: the choice is made during capture and then frozen into the graph.

    def _can_use_graph(self) -> bool:
        # `_graph_mode_allowed` is only set inside `capture()`, so the eager
        # hot path never reaches the cudart capture query. During capture,
        # warm-up runs execute immediately and must not consume a
        # graph_params row (it would be dereferenced before registration).
        return (
            self._graph_mode_allowed
            and not is_in_tc_piecewise_cuda_graph()
            and torch.cuda.is_current_stream_capturing()
        )

    def _pick_algo(
        self, nbytes: int, can_use_graph: bool
    ) -> Tuple[Optional[AllReduceAlgo], _PullMode]:
        heuristic = self.config.graph if can_use_graph else self.config.eager
        default_mode = _PullMode.GRAPH if can_use_graph else _PullMode.EAGER

Two identical servers, the same TP degree, the same 512 KB message, running different collectives — at SM90 TP=4 the eager side is still under its 896 KB one-shot-push threshold while the captured side is over its 384 KB one and falls through to zero-copy two-shot pull, because one was capturing and one was not. Note also that the piecewise path is excluded (is_in_tc_piecewise_cuda_graph()) and multi-node groups are forced back to eager pull, because IPC registration is node-local (L170-L176).

Thirteen decode requests arrive at execute_model. Follow the shape.

1. _prepare_inputs writes 13 rows into the persistent buffers and zeroes the rest: self.seq_lens[13:].fill_(0) (gpu_model_runner.py:L2252-L2255). The buffers themselves were allocated at startup for 512 and have never moved.
2. dispatch_cudagraph(13) (gpu_model_runner.py:L4099-L4110) calls CudagraphDispatcher.dispatch.
3. _create_padded_batch_descriptor reads _bs_to_padded_graph_size[13] = 16 — the next rung on [1, 2, 4, 8, 16, 24, …] — and builds BatchDescriptor(num_tokens=16, num_reqs=16, uniform=True, has_lora=False) (cudagraph_dispatcher.py:L132-L156). Three fake rows.
4. The descriptor is in cudagraph_keys[FULL], so dispatch returns (FULL, desc).
5. _build_attention_metadata runs with num_reqs=13, num_reqs_padded=16. The three padded rows get seq_len = 0 and block_table = NULL_BLOCK_ID (L2410-L2412). Attention will read nothing for them.
6. set_forward_context(..., cudagraph_runtime_mode=FULL, batch_descriptor=desc) (L4545-L4556) publishes the decision.
7. Inside the model, CUDAGraphWrapper.__call__ reads the forward context, sees its own mode matches, finds the entry for this descriptor, and calls entry.cudagraph.replay() (cuda_graph.py:L360-L361). One submission covers the captured model region; inspect the capture boundary before including sampling/frontend kernels in its node count.
8. The 13-to-16 padding has a constant weight-read term in the ideal model, but actual kernel time and traffic must be measured. Below-ridge arithmetic is not a zero-cost proof.

Now flip one thing: give one request a two-token query while uniform_decode_query_len is 1. The batch is no longer uniform, the FULL key misses, replace(batch_desc, num_reqs=None, uniform=False) finds the PIECEWISE key instead, and the step runs as a chain of per-piece graphs with attention outside them. Same tokens, different runtime mode, because of one query length.

An allocation during capture, or during replay

PyTorch reroutes allocations into the graph pool during capture, but any code that grows a workspace on its first big shape will grow it again at some replay and hand a captured kernel a stale pointer. vLLM sizes every workspace during warmup and then forbids growth:

            if self._locked:
                raise AssertionError(
                    f"Workspace is locked but allocation from '{get_caller_info()}' "
                    f"requires {required_bytes / _MB:.2f} MB, current size is "
                    f"{current_size / _MB:.2f} MB. "
                    "Workspace growth is not allowed after locking."
                )

lock_workspace() is called at the end of capture_model (gpu_model_runner.py:L7035-L7037). If it fires, a kernel wanted a bigger scratch buffer at a shape warmup never exercised; the fix is to make warmup cover that shape.

Capture happening where it should not

    global cudagraph_capturing_enabled
    if not cudagraph_capturing_enabled:
        raise RuntimeError(
            "CUDA graph capturing detected at an inappropriate "
            "time. This operation is currently disabled."
        )

vLLM flips this off after capture_model returns, so a lazy capture in the steady state is loud rather than silent — a graph captured mid-serving would record whatever happened to be in the persistent buffers at that moment.

A backend that cannot be captured downgrades your mode without failing

At startup vLLM takes the minimum AttentionCGSupport over every attention group (gpu_model_runner.py:L7303-L7317) and feeds it to resolve_cudagraph_mode_and_sizes, which quietly rewrites your mode:

            # attempt to resolve the full cudagraph related mode
            if self.splitting_ops_contain_attention():
                msg += "; setting cudagraph_mode=FULL_AND_PIECEWISE"
                cudagraph_mode = CUDAGraphMode.FULL_AND_PIECEWISE
            else:
                msg += "; setting cudagraph_mode=FULL_DECODE_ONLY"
                cudagraph_mode = CUDAGraphMode.FULL_DECODE_ONLY
            logger.warning(msg)

You asked for FULL, you got FULL_DECODE_ONLY, and the only trace is a logger.warning that begins "CUDAGraphMode.FULL is not supported with <backend> backend (support: …)". Grep your startup log for that string before believing a benchmark. The same function raises outright — not warns — if the resolved mode still wants full graphs and the backend is NEVER (vllm/config/compilation.py:L1463-L1472).

Two switches for the same thing, and only one of them works

SGLang carries a legacy boolean alongside the new per-phase config, and its own source documents the trap:

            self.disable_cuda_graph = True
            # cuda_graph_config was already parsed from the legacy boolean, so
            # flipping the boolean alone would not stop graph capture.
            self.cuda_graph_config.decode.backend = Backend.DISABLED
            self.cuda_graph_config.prefill.backend = Backend.DISABLED

Setting disable_cuda_graph after the config resolution pipeline has run does nothing. Use --cuda-graph-backend-decode=disabled.

You cannot step through a replay

Once a graph is captured there is no Python frame to break on, no NVTX range you can add without recapturing, and a wrong result gives you no stack. Both engines ship an escape hatch. vLLM's is --enforce-eager (gpu_worker.py:L730-L732 skips capture_model entirely). SGLang's is better: it keeps the capture/replay path but breaks the graph at every op, so bugs that only appear under graph dispatch still reproduce:

    debug_cuda_graph: A[
        bool,
        "Enable debug/eager mode for CUDA graph using breakable CUDA graph. When enabled, graph breaks are inserted so every operation runs eagerly while still going through the CUDA graph capture / replay path. Useful for debugging CUDA graph capture / replay issues.",
        NS("exec.graph"),
    ] = False

If eager and breakable-graph modes differ, compare dispatch, shapes, input contents, lifetime and synchronization. This narrows the reproducer but does not exclude a kernel defect: different padding or synchronization can expose a kernel bug only in one path.

Capture OOM at startup

The pool is allocated after the KV cache is sized, so an under-estimate surfaces as a capture failure, not a serving failure. SGLang's message is the checklist:

CUDA_GRAPH_CAPTURE_FAILED_MSG = (
    "Possible solutions:\n"
    "1. set --mem-fraction-static to a smaller value (e.g., 0.8 or 0.7)\n"
    "2. set --cuda-graph-max-bs-decode to a smaller value (e.g., 16)\n"
    "3. disable decode CUDA graph by --cuda-graph-backend-decode=disabled. "
    "(Not recommended. Huge performance loss)\n"
    "Open an issue on GitHub https://github.com/sgl-project/sglang/issues/new/choose \n"
)

A graph output is normally overwritten on later replay: retain a clone if another request needs the old value. Warm kernels and allocations before capture, establish side-stream dependencies, keep captured buffers alive, update inputs in place, and do not concurrently reuse a shared graph pool without its permitted ordering. Forked capture streams must rejoin through recorded events. A useful experiment crosses compile on/off, graph on/off, backend and batch bucket while recording the resolved modes, pool bytes, host submission, device critical path and latency distribution. The CPU alias check below illustrates buffer ownership only; it does not execute CUDA capture. See NVIDIA's graph lifecycle and update contracts.

import numpy as np

persistent_input = np.zeros(3)
captured_reference = persistent_input
persistent_output = np.zeros(3)
def replay_reference():
    np.multiply(captured_reference, 2, out=persistent_output)
    return persistent_output

persistent_input[:] = [1., 2., 3.]
borrowed = replay_reference()
saved = borrowed.copy()
persistent_input = np.array([7., 8., 9.])  # rebind, not an update
np.testing.assert_array_equal(replay_reference(), [2., 4., 6.])
captured_reference[:] = persistent_input
np.testing.assert_array_equal(replay_reference(), [14., 16., 18.])
np.testing.assert_array_equal(borrowed, [14., 16., 18.])
np.testing.assert_array_equal(saved, [2., 4., 6.])
print("In-place input update and replay-output ownership contracts pass.")

1. Price the graphs on your own box. Run the same model twice and diff time-per-output-token:

vllm serve meta-llama/Meta-Llama-3-8B-Instruct --port 8000
vllm serve meta-llama/Meta-Llama-3-8B-Instruct --port 8001 --enforce-eager

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30001 \
  --cuda-graph-backend-decode=disabled

Run one server at a time, stopping it before launching the next, or isolate devices. Set explicit TP flags for the compared configuration and record resolved compile/graph/backend modes. Eager flags can change more than graph replay. Report both host and device timing.

2. Read the padding you are actually paying. Turn on cudagraph_metrics in vLLM's observability config: gpu_model_runner.py:L4149-L4156 records a CUDAGraphStat per step and vllm/compilation/cuda_graph.py:L33-L120 aggregates it into a table of unpadded tokens, padded tokens, paddings, mode and count. Measure both common 9-to-16 and 300-to-304 cases rather than optimizing raw waste percentages. Add eligible capture sizes only when saved latency justifies extra startup time and graph/KV memory opportunity cost.

3. Watch the startup cost. Grep for Graph capturing finished in (vLLM) or the Capturing batches (bs=... avail_mem=... GB) progress bar (SGLang, decode_cuda_graph_runner.py:L1094-L1096), then halve --cuda-graph-max-bs-decode and watch both shrink.

4. Account for every microsecond. Lab 10 — 10-profile-a-decode-step is where you replace every derived number on this page with a measured one: capture one iteration under torch.profiler, sum the kernel durations, sum the gaps, and compare against the 4.48 ms floor. Do it with graphs on and off. That is also the place to measure the per-launch overhead this chapter had to cite from a paper.

• Vellaisamy, Labonte, Chakraborty, Turner, Sury, Shen, Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures, ISPASS 2025 (arXiv:2504.11750). Table V is the H100 / CUDA 12.6 null-kernel launch measurement this chapter leans on.
• NVIDIA, CUDA Graph Best Practice for PyTorch (docs.nvidia.com). The current per-launch overhead breakdown by layer; ranges only, no hardware named.
• NVIDIA, Getting Started with CUDA Graphs (developer.nvidia.com/blog/cuda-graphs). The original V100 / CUDA 10.1 post whose 2.9 / 9.6 / 3.8 / 3.4 µs figures propagated everywhere, including into §0.3.
• vLLM PR discussion for the CUDAGraphMode / dispatcher rework (vllm-project/vllm#20059) — where full and piecewise capture stopped being a boolean.
• SGLang issue tracker for the per-phase config (sgl-project/sglang#15927, referenced by name in server_args.py:L4956-L4958 for the prefill max_bs default).
• §8.2 for how the pieces of a piecewise graph get produced; §3.4 for AttentionCGSupport and the metadata-buffer contract; §5.4 for the all-reduce cost model this chapter's finding modifies; §10.4 for what fixed shapes do to reproducibility.