torch.compile, Inductor, piecewise capture

Start a vLLM server and watch the log. Somewhere between weight loading and the first token you get a line that costs you real money:

@contextlib.contextmanager
def monitor_torch_compile(
    vllm_config: VllmConfig,
    message: str = "torch.compile took %.2f s in total",
    is_encoder: bool = False,
) -> Generator[None, None, None]:

On a cold cache that number is minutes, not seconds, and it is paid before the server binds its port. Autoscaling a fleet on this is a real operational problem — see §9.4 for what that does to tail latency during a scale-out.

So the question that opens the chapter is not "what is torch.compile". It is: what does an inference engine buy for those minutes, and why is it worth it? The honest answer has two halves, and they point in different directions depending on batch size. Here is the first half, made concrete.

vLLM's RMSNorm is a CustomOp: it carries a hand-written CUDA kernel and a pure-PyTorch reference. When Inductor is in play, the custom kernel is switched off by default and Dynamo traces the reference:

    def default_on() -> bool:
        """
        Behavior controlled by `CompilationConfig.custom_ops`: On by default if
        'all', off by default if 'none'.
        When PyTorch Inductor is used, 'none' is the default value,
        otherwise 'all'.
        """
        compilation_config = get_cached_compilation_config()
        count_none = compilation_config.custom_ops.count("none")
        count_all = compilation_config.custom_ops.count("all")
        if count_none + count_all != 1:
            raise ValueError(
                "custom_ops must contain exactly one base mode: 'all' or 'none'"
            )

        return not count_none > 0 or count_all > 0

And the reference that Dynamo sees is this:

@register_op(allow_inplace=True)
def fused_add_rms_norm(
    x: Tensor,
    x_residual: Tensor,
    weight: Tensor | None,
    epsilon: float,
    variance_size: int | None = None,
) -> tuple[Tensor, Tensor]:
    """Fused add and weighted root-mean-square layer normalization"""
    orig_dtype = x.dtype
    x = x.to(torch.float32)
    x = x + x_residual.to(torch.float32)
    x_residual = x.to(orig_dtype)

    x_var = x if variance_size is None else x[..., :variance_size]
    variance = x_var.pow(2).mean(dim=-1, keepdim=True)
    x = x * torch.rsqrt(variance + epsilon)
    if weight is not None:
        x = x.to(weight.dtype) * weight
    return x.to(orig_dtype), x_residual

This is a reference tensor decomposition, not an exact kernel trace. Casts may be no-ops, reductions may launch multiple kernels, and caches can retain intermediates. The production eager baseline already has a fused norm, so this decomposition is not its launch count.

torch.compile is three programs wearing one name. Dynamo is a bytecode interpreter: it runs your Python frame symbolically and emits an FX graph of tensor operations plus a set of guards — predicates on the inputs that must hold for the graph to be reused. Anything Dynamo cannot represent as a tensor op (a print, a data-dependent branch, an unsupported builtin) causes a graph break: the traced region ends there and Python resumes. AOTAutograd functionalises the graph, turning mutations into pure operations so a compiler can reason about it. Inductor lowers the functional graph to Triton source, schedules it into as few kernels as the dependence structure allows, and compiles that Triton to PTX.

The serving-relevant consequence: fusion happens at the Inductor scheduling step, and it can only fuse ops that are in the same graph. Everything else in this chapter follows from that one sentence.

The last box is where this chapter hands off to §8.1. Compilation produces the pieces; CUDA graph capture freezes the pieces. They are separate mechanisms with separate configuration, and the reason they are usually discussed together is that the split points serve both.

Take one fused_add_rms_norm call on Llama-3-8B, hidden size $d = 4096$, activations in bf16, at one token. Write $b_{16} = 2d = 8{,}192$ bytes for a bf16 row and $b_{32} = 4d = 16{,}384$ bytes for an fp32 row. Count logical tensor traffic, assuming materialized intermediates.

The fused input/output payload is 40,960 bytes, about 6.2 times smaller than the 253,976-byte logical reference traffic. Across 64 illustrative call sites, the difference is about 13.63 MB. This is not a measured HBM saving or a production compile speedup:

Now the punchline, which is not the one you expect. The book's H100 constant is 3.35 TB/s of HBM bandwidth and a 4.48 ms Llama-3-8B decode-step floor at batch 1 — that floor is 15.0 GB of weight traffic. At batch 1:

The launch count must come from a trace: this reference contains eleven materializing operations under the stated dtype, not an exact ten-kernel decomposition. A fused eager norm may already use one launch. Compilation can still move fusion boundaries or remove framework overhead, while CUDA graphs separately reduce repeated submission work.

Scale the batch and the arithmetic inverts. Elementwise traffic is proportional to token count; weight traffic is not. At 256 tokens in flight:

At 256 tokens the reference's extra logical bytes are about 23% of the base weight bytes. Converting that ratio to HBM time assumes all intermediates reach HBM at the stated bandwidth. It is an explanatory traffic model, not evidence for a 23% production speedup.

Opaque attention and explicit partition boundaries

vLLM registers attention as an opaque custom operator. Dynamo does not trace into custom operators, by design:

@eager_break_during_capture
@maybe_transfer_kv_layer
def unified_attention_with_output(
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    output: torch.Tensor,
    layer_name: LayerNameType,
    output_scale: torch.Tensor | None = None,
    output_block_scale: torch.Tensor | None = None,
    kv_cache_dummy_dep: torch.Tensor | None = None,
) -> None:
    # kv_cache_dummy_dep is not used but accepting it creates a data dependency
    # that ensures torch.compile preserves ordering between KV cache update and
    # attention forward.
    del kv_cache_dummy_dep
    layer_name = _resolve_layer_name(layer_name)
    attn_metadata, self, kv_cache, _ = get_attention_context(layer_name)

    self.impl.forward(
        self,
        query,
        key,
        value,
        kv_cache,
        attn_metadata,
        output=output,
        output_scale=output_scale,
        output_block_scale=output_block_scale,
    )

Three reasons the opacity is load-bearing. get_attention_context(layer_name) pulls attn_metadata and the KV cache out of a global forward context — Python state that changes every step and that Dynamo would have to guard on. self.impl.forward dispatches into an external backend (FlashAttention, FlashInfer, Triton; §3.4) whose kernels Inductor has no lowering for and could not improve on. And attention writes into a paged KV cache whose addresses are chosen at runtime; a compiler that thought it understood that aliasing would reorder loads across it.

Splitting the graph

An opaque custom op can be an FX node, including in fullgraph compilation, with registered schema, mutation/alias contracts and fake/meta behavior. It need not cause a Dynamo graph break. The engine may deliberately split around it for lowering or capture. vLLM lists split ops here:

    # Use PyTorch operator format: "namespace::name"
    _attention_ops: ClassVar[list[str]] = [
        "vllm::unified_attention_with_output",
        "vllm::unified_mla_attention_with_output",
        "vllm::mamba_mixer2",
        "vllm::mamba_mixer",
        "vllm::short_conv",
        "vllm::linear_attention",
        "vllm::qwen_gdn_attention_core",
        "vllm::qwen_gdn_attention_core_fused_norm_packed",
        "vllm::gdn_attention_core_xpu",
        "vllm::olmo_hybrid_gdn_full_forward",
        "vllm::sparse_attn_indexer",
        "vllm::rocm_aiter_sparse_attn_indexer",
        "vllm::deepseek_v4_attention",
        "vllm::hpc_rope_norm_forward",
    ]

set_splitting_ops_for_v1 installs that list when the user has not overridden it, additionally appending vllm::unified_kv_cache_update and vllm::unified_mla_kv_cache_update in the Dynamo-split path (vllm/config/compilation.py:L1134-L1147, L1184-L1185). The split itself is a linear scan assigning every FX node a subgraph id, bumping the id at each splitting op:

        if should_split(node, splitting_ops):
            subgraph_id += 1
            node_to_subgraph_id[node] = subgraph_id
            split_op_graphs.append(subgraph_id)

            # keep consecutive splitting ops together
            # (we know node.next exists because node isn't the last (output) node)
            if should_split(node.next, splitting_ops):
                # this will get incremented by the next node
                subgraph_id -= 1
            else:
                subgraph_id += 1
        else:
            node_to_subgraph_id[node] = subgraph_id

The "keep consecutive splitting ops together" branch matters for the count: in Attention.forward the KV-cache update op is emitted immediately before the attention op, so the pair collapses into one opaque region rather than two. Read the right branch of that method: Attention.forward has two, selected by use_direct_call = not current_platform.opaque_attention_op() (attention.py:L430), and CUDA returns True there (vllm/platforms/cuda.py:L577-L579), so on an NVIDIA box use_direct_call is False and the live path is the torch.ops.vllm.* branch at vllm/model_executor/layers/attention/attention.py:L550-L569 — the direct-call branch just above it never runs. Both emit the same two ops in the same order, so the split arithmetic is unchanged; the citation is not. should_split matches on the qualified operator name (vllm/compilation/partition_rules.py:L14-L38). For Llama-3-8B, L=32, that gives 32 opaque regions and 33 compilable ones — 65 subgraphs from one FX graph.

The handoff to CUDA graphs is one function, called once the interpreter has built a PiecewiseBackend for a compilable submodule:

    if (
        not compilation_config.cudagraph_mode.has_piecewise_cudagraphs()
        or compilation_config.use_inductor_graph_partition
    ):
        return piecewise_backend

    # We're using Dynamo-based piecewise splitting, so we wrap
    # the whole subgraph with a static graph wrapper.
    from .cuda_graph import CUDAGraphOptions

    # resolve the static graph wrapper class (e.g. CUDAGraphWrapper
    # class) as platform dependent.
    static_graph_wrapper_class = resolve_obj_by_qualname(
        current_platform.get_static_graph_wrapper_cls()
    )

    # Always assign PIECEWISE runtime mode to the
    # CUDAGraphWrapper for piecewise_backend, to distinguish
    # it from the FULL cudagraph runtime mode, no matter it
    # is wrapped on a full or piecewise fx graph.
    return static_graph_wrapper_class(
        runnable=piecewise_backend,
        vllm_config=vllm_config,
        runtime_mode=CUDAGraphMode.PIECEWISE,
        cudagraph_options=CUDAGraphOptions(
            debug_log_enable=is_first_graph,
            gc_disable=not is_first_graph,
            weak_ref_output=is_last_graph,
        ),
    )

Everything downstream of that call — capture, the bucket ladder, address pinning, AttentionCGSupport — is §8.1's.

How a model opts in

A model class declares itself compilable, and declares which argument dimensions vary:

@support_torch_compile(
    # TODO[#32068]: Investigate recompilation
    # mark_unbacked_dims={"input_ids": 0},
    dynamic_arg_dims={
        "input_ids": {0: "b"},
        "positions": {0: "b"},
        "intermediate_tensors": {0: "b"},
        "inputs_embeds": {0: "b"},
    },
)

The shared shape id "b" tells Dynamo that all four dimensions are the same symbol. Before the first trace, _mark_dynamic_inputs walks the bound arguments and calls torch._dynamo.mark_dynamic — or mark_unbacked, under DynamicShapesType.UNBACKED — on each of them (vllm/compilation/decorators.py:L414-L446). Without this, Dynamo specialises on the first batch size it sees, and since a serving batch changes every step you would recompile forever.

vLLM then goes further than stock torch.compile and throws the guards away:

        if mode != CompilationMode.STOCK_TORCH_COMPILE:
            # Drop all the guards.
            if self.evaluate_guards:
                assert not envs.VLLM_USE_BYTECODE_HOOK, (
                    "compilation_config.dynamic_shapes_config.evaluate_guards "
                    "requires VLLM_USE_BYTECODE_HOOK=0. "
                )
                assert ds_type != DynamicShapesType.UNBACKED, (
                    "UNBACKED dynamic shapes do not add guards"
                )

                options["guard_filter_fn"] = lambda x: [
                    entry.guard_type == "SHAPE_ENV" for entry in x
                ]
            else:
                if hasattr(torch.compiler, "skip_all_guards_unsafe"):
                    # Torch 2.10+ provides skip_all_guards_unsafe
                    options["guard_filter_fn"] = torch.compiler.skip_all_guards_unsafe
                else:
                    # Equivalent fallback for older PyTorch: skip all guards
                    options["guard_filter_fn"] = lambda x: [False for _ in x]

In this path fullgraph=True makes Dynamo graph breaks errors. Trace-once and guard-dropping assumptions do not imply one backend artifact for every shape: pieces and compile ranges can have separate Inductor artifacts. Shape-dependent Python branches are unsafe unless represented correctly or guarded; explicit custom-op boundaries can keep runtime choices out of tracing. The debug evaluate-guards path is an important exception.

SGLang: the same skeleton, a narrower deployment

SGLang's python/sglang/srt/compilation/ is a lineal descendant of vLLM's — the file header says so — and split_graph is the same scan, minus the consecutive-op merge:

    split_op_graphs = []
    for node in graph.graph.nodes:
        if node.op in ("output", "placeholder"):
            continue
        if node.op == "call_function" and str(node.target) in ops:
            subgraph_id += 1
            node_to_subgraph_id[node] = subgraph_id
            split_op_graphs.append(subgraph_id)
            subgraph_id += 1
        else:
            node_to_subgraph_id[node] = subgraph_id

The divergence is where the split list comes from. vLLM keeps a central allowlist in config; SGLang registers split ops at the definition site:

def register_split_op(op_name: Optional[str] = None):
    def decorator(op_func: Callable):
        name = op_name or op_func.__name__
        SPLIT_OPS.append(f"sglang.{name}")
        return op_func

    return decorator

@register_custom_op(mutates_args=["output"])
@register_split_op()
def unified_attention_with_output(

Decentralised registration means a new attention variant becomes a split point by adding one decorator — no config edit, no forgotten entry — but there is then no single place to read the full split set, and it depends on which modules got imported. vLLM's list is greppable and must be maintained by hand, which is why it has fourteen entries with names like vllm::olmo_hybrid_gdn_full_forward.

The larger difference is deployment surface. In SGLang the piecewise-compile machinery is one CUDA-graph backend, selected per phase, whose compiler defaults to eager:

@dataclass
class PhaseConfig:
    """Per-phase CUDA graph settings."""

    backend: str = Backend.DISABLED
    max_bs: Optional[int] = None
    bs: Optional[List[int]] = None
    # Only meaningful when backend == tc_piecewise; ignored otherwise.
    tc_compiler: str = "eager"

and at 7d89325 the default prefill backend on CUDA is not tc_piecewise at all: default_prefill_backend() returns Backend.BREAKABLE on CUDA, Backend.TC_PIECEWISE elsewhere (python/sglang/srt/model_executor/cuda_graph_config.py:L110-L119). On a stock CUDA SGLang server, Dynamo does not run. Turning it on is --cuda-graph-backend-prefill=tc_piecewise --cuda-graph-tc-compiler=inductor, and even then it covers prefill only — the flag's own comment says "currently only the prefill phase consumes it" (python/sglang/srt/server_args.py:L1926-L1930).

SGLang's second, older torch.compile path is unrelated to the piecewise backend: it compiles the whole model forward as a black box, inside the decode CUDA-graph capture loop:

        if enable_compile:
            _to_torch(model, reverse=False, num_tokens=num_tokens)
            backup_ca_comm = tp_group.ca_comm
            yield torch.compile(
                torch.no_grad()(model.forward),
                mode=os.environ.get(
                    "SGLANG_TORCH_COMPILE_MODE", "max-autotune-no-cudagraphs"
                ),
                dynamic=_is_hip and get_bool_env_var("SGLANG_TORCH_DYNAMIC_SHAPE"),
            )

dynamic= is False on CUDA, so this path specialises per batch size — which is why it is bounded by torch_compile_max_bs, default 32 (python/sglang/srt/server_args.py:L2087-L2097): compile once per captured bucket up to 32, fall back above it.

vLLM's custom passes, and one that breaks the piecewise contract

Because vLLM owns the Inductor backend it can insert FX passes after AOTAutograd's functionalisation and before Inductor's lowering; vllm/compilation/passes/pass_manager.py assembles them from PassConfig flags. Take SequenceParallelismPass, which §5.1 flagged as vLLM's route to a fusion SGLang reaches differently. Read it as an opt-in, not as what your server is doing: -O2's pass_config sets "enable_sp": IS_DENSE and IS_DENSE is hardcoded False at this SHA (vllm/config/vllm.py:L147-L154), so on a stock -O2 server this pass does not run for any model. It matches an all-reduce feeding an RMSNorm:

    def register(self, pm_pass: PatternMatcherPass) -> None:
        def pattern(
            residual: torch.Tensor,
            mm_1: torch.Tensor,
            rms_norm_weights: torch.Tensor,
        ) -> tuple[torch.Tensor, torch.Tensor]:
            all_reduce = self._all_reduce(mm_1)
            rmsnorm = vllm.ir.ops.fused_add_rms_norm(
                all_reduce, residual, rms_norm_weights, self.epsilon
            )
            return rmsnorm[0], rmsnorm[1]

        def replacement(
            residual: torch.Tensor,
            mm_1: torch.Tensor,
            rms_norm_weights: torch.Tensor,
        ) -> tuple[torch.Tensor, torch.Tensor]:
# ...
            reduce_scatter = self._reduce_scatter(mm_1)
            local_len = reduce_scatter.size(0)
# ...
            residual = residual[
                self.tp_rank * local_len : self.tp_rank * local_len + local_len, ...
            ]
            rmsnorm = vllm.ir.ops.fused_add_rms_norm(
                reduce_scatter, residual, rms_norm_weights, self.epsilon
            )
            all_gather = self._all_gather(rmsnorm[0])
            return all_gather, rmsnorm[1]

Pattern: all_reduce → fused_add_rms_norm. Replacement: reduce_scatter → norm on 1/TP of the rows → all_gather. Wire volume is unchanged — an all-reduce is a reduce-scatter plus an all-gather. The class docstring is unusually candid about why the pass exists anyway:

    While this pass itself does not directly yield performance improvements,
    it lays the groundwork for subsequent fusion passes, such as
    GEMM + ReduceScatter and AllGather + GEMM fusions. These fusions can
    significantly reduce communication overhead and improve overall model
    performance.

    This pass is only supported when compiling the whole graph (fullgraph
    mode, i.e. using Inductor graph partition or empty splitting_ops).
    Piecewise compilation is not supported because the residual tensor
    gets split across TP ranks, causing size mismatches at subgraph
    boundaries.

This is the chapter's sharpest tradeoff, stated by the code itself. The pass makes the residual stream shorter between the reduce-scatter and the all-gather, and a tensor that changes shape cannot cross a subgraph boundary whose shapes were fixed at split time. So enabling sequence parallelism means giving up FX-level piecewise splitting — either splitting_ops=[] or use_inductor_graph_partition=True, which defers partitioning to Inductor codegen time, after all passes have run (vllm/config/compilation.py:L669-L687). Piecewise splitting and whole-graph rewriting want opposite things; the Inductor-partition mode exists to have both.

Caching, and what invalidates it

Compilation time varies with graph, shapes and cache state. The engine uses several outer hash factors to organize artifacts; this alone is not proof that every dependency is covered:

        if not self.compilation_config.cache_dir:
            # no provided cache dir, generate one based on the known factors
            # that affects the compilation. if none of the factors change,
            # the cache dir will be the same so that we can reuse the compiled
            # graph.
            factors = [env_hash, config_hash, code_hash, compiler_hash]
            # Use SHA-256 for cache key hashing to be consistent across
            # compute_hash functions. Truncate for a short cache dir name.
            hash_key = hashlib.sha256(str(factors).encode()).hexdigest()[:10]
            cache_dir = os.path.join(
                envs.VLLM_CACHE_ROOT, "torch_compile_cache", hash_key
            )
            self.compilation_config.cache_dir = cache_dir

Four factors, each worth knowing because each is a different way to lose your warm start:

A cache miss can delay readiness, but rolling deployments do not inherently compile serially. Prewarm compatible artifacts and measure startup coverage. SGLang's outer directory uses a compiler-manager hash; compare complete inner compiler/JIT keys and validation before claiming greater staleness risk from the number of outer factors. Use private, access-controlled cache roots and a toolchain/model/shape manifest for reproducible comparisons.

Follow the very first LlamaModel.__call__ on a cold cache, naming functions in order.

Hop I→J. PiecewiseCompileInterpreter subclasses torch.fx.Interpreter and overrides call_module. For each submodule in compile_submod_names it constructs a PiecewiseBackend, wraps it, and replaces the submodule in self.module.__dict__ with the wrapper (vllm/compilation/backends.py:L730-L775). What comes out is structurally the split graph with each compilable child swapped for a compiled-and-captured callable. That is the entire mechanism of "piecewise": no special runtime, just a graph whose children happen to be CUDA graph replays.

Hop L. The cache key inside CompilerManager is (runtime_shape, graph_index, backend_name) — a Range, a piece index, and the compiler name (vllm/compilation/backends.py:L124-L138). Compilation is therefore per-piece and per compile-range: compile_ranges_endpoints partitions [1, max_num_batched_tokens] into intervals, each getting its own Inductor artefact (vllm/config/compilation.py:L580-L593). Multiply 33 pieces by the number of ranges and you see where the minutes go.

SGLang's trace has one structural difference worth naming: it compiles the general dynamic-shape graph during tracing, inside call_module, and defers shape-specialised compilation to the first real call at that shape (python/sglang/srt/compilation/backend.py:L304-L343). CUDAPiecewiseBackend.__call__ then compiles and captures lazily per bucket:

    def __call__(self, *args) -> Any:
        if not self.first_run_finished:
            self.first_run_finished = True
            self.check_for_ending_compilation()
            return self.compiled_graph_for_general_shape(*args)

        if len(self.sym_shape_indices) == 0:
            return self.compiled_graph_for_general_shape(*args)

        runtime_shape = args[self.sym_shape_indices[0]]
        if runtime_shape not in self.concrete_size_entries:
            # we don't need to do anything for this shape
            return self.compiled_graph_for_general_shape(*args)

vLLM went the other way at this SHA: PiecewiseCompileInterpreter's docstring says it "creates a PiecewiseBackend and compiles all ranges up front", and monitor_profiling_run asserts that num_backend_compilations does not change during the profiling run (vllm/compilation/monitor.py:L61-L82). Front-loading everything makes startup slower and steady-state latency more predictable; SGLang's laziness does the opposite.

Compiled kernels do not expose the original Python frames, but FX/generated code, compiler dumps and backend bisection remain useful. Compare eager, tracing-only backend, AOT processing and Inductor to localize the first divergent tensor. Fusion may change floating-point bits, and sometimes tokens, but it does not guarantee token differences. Test numerical tolerances separately from deterministic token equality and sampling randomness.

Ahead-of-time kernel compilation is a different animal

SGLang ships one more thing called compilation that is not torch.compile. python/sglang/compile_deep_gemm.py launches a real server with SGLANG_IN_DEEPGEMM_PRECOMPILE_STAGE and SGLANG_ENABLE_JIT_DEEPGEMM set, sends one four-token request through it, waits, and exits (python/sglang/compile_deep_gemm.py:L31-L37, L58-L90). DeepGEMM JIT-compiles a kernel per problem shape on first use, so on a live server the first request at each new shape pays a compile. Prewarming covers only specializations exercised by the script or its explicit shape sweep. A four-token request alone proves no coverage of all future model/dtype/backend shapes; log later JIT misses and maintain a prewarm manifest.

This CPU example registers a functional operator, supplies fake-tensor shape behavior, checks its schema, and captures it as an opaque node in a full graph. The inspection backend executes that FX graph without claiming Inductor fusion or speedup. A mutating production operator must additionally declare every mutated argument and compatible aliases; hidden cache writes cannot be omitted. Shape-dependent branching and fake stride errors deserve negative tests before removing guards. See PyTorch's custom-operator contracts.

import torch

torch.set_num_threads(1)
torch.manual_seed(2)
@torch.library.custom_op("curriculum_runtime::scaled_sine", mutates_args=())
def scaled_sine(x: torch.Tensor) -> torch.Tensor:
    return 2 * torch.sin(x)

@scaled_sine.register_fake
def _(x):
    return torch.empty_like(x)

x = torch.randn(3, 5)
torch.library.opcheck(scaled_sine, (x,), test_utils=("test_schema", "test_faketensor"))
seen = []
def inspect_backend(graph, inputs):
    seen.extend(str(node.target) for node in graph.graph.nodes if node.op == "call_function")
    return graph.forward

def model(value):
    return scaled_sine(value) + 1

compiled = torch.compile(model, backend=inspect_backend, fullgraph=True)
torch.testing.assert_close(compiled(x), model(x))
assert any("curriculum_runtime.scaled_sine" in target for target in seen)
print("Opaque custom op remains an FX node under fullgraph=True.")

Everything below is CPU-cheap except the server launches, which need a GPU. Numbers you get are yours, not the book's.

# 1. Cold compile. Watch for "Dynamo bytecode transform time" and
#    "torch.compile took N s in total".
export VLLM_CACHE_ROOT="$(mktemp -d)"  # private cold trial; retain it for warm restart
vllm serve meta-llama/Meta-Llama-3-8B-Instruct 2>&1 | grep -E "Dynamo|torch.compile took|cache directory"

# 2. Stop the first server, then restart with the same private cache and configuration.

# 3. Read the split graph vLLM wrote for you. This file is the answer to
#    "how many pieces did it make", printed as Python.
find "$VLLM_CACHE_ROOT" -name computation_graph.py  # inspect call_module nodes, not grep line counts

# 4. Turn compilation off, keep CUDA graphs, and compare.
vllm serve meta-llama/Meta-Llama-3-8B-Instruct -cc.mode=0

# 5. Turn both off.
vllm serve meta-llama/Meta-Llama-3-8B-Instruct --enforce-eager

# 6. Full torch.compile logs for a real diagnosis.
pip install tlparse
TORCH_TRACE=~/trace_dir vllm serve meta-llama/Meta-Llama-3-8B-Instruct
tlparse ~/trace_dir/<rank_0_log_file>

On the SGLang side the comparison worth running is the one the defaults do not give you:

# Default on CUDA: prefill uses the breakable backend, no Dynamo at all.
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct

# Piecewise compile the prefill phase with Inductor.
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct \
  --cuda-graph-backend-prefill tc_piecewise --cuda-graph-tc-compiler inductor

# The other, unrelated path: whole-forward compile inside decode capture,
# bounded at batch 32 by default.
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct \
  --enable-torch-compile --torch-compile-max-bs 32

• vLLM design doc: torch_compile.md — the maintainers' own overview of the integration, at the pinned SHA.
• vLLM design doc: debug_vllm_compile.md — the tlparse workflow, the dynamic-shape failure modes, and the cache escape hatches quoted in §6. Read this before your first compile bug, not after.
• vLLM PR #20059 — why full CUDA graphs are captured outside a piecewise FX graph rather than by flattening it, cited in the set_splitting_ops_for_v1 comment.
• vLLM issue #33267 — unified_kv_cache_update's string parameter blocking Inductor from reusing piecewise graphs, the reason it is appended to splitting_ops.
• vLLM issue #25689 — why IS_QUANTIZED and IS_DENSE are hardcoded False in the optimization-level tables, disabling SP and attention-quant fusion by default.
• vLLM issue #25094 — the blocked-weights case that forces +quant_fp8 back on even under Inductor; a good example of a custom op that Inductor cannot replace.
• tlparse — the log renderer that turns TORCH_TRACE output into browsable compilation metrics, guards, and generated Triton.
• PyTorch: Dynamo core concepts — the reference the vLLM debug doc points you at when you hit a graph break.
• PyTorch: torch.compile caching — the layer underneath vLLM's cache; worth knowing which of the two you are fighting.