Profiling, roofline of a decode step, capacity planning

Turn on vLLM's built-in performance metrics and the engine will start telling you how fast it is going. The log line is one printf in vllm/v1/metrics/loggers.py, fed by an analytic model in vllm/v1/metrics/perf.py whose module docstring says exactly what it is for:

"""
Analytic flops/memory estimation module for transformer components,
to help derive MFU (Model Flops Utilization) stats for a running model.
"""

Analytic. Not counted — multiplied out from the config, per scheduler step, in vllm/v1/core/sched/scheduler.py:L1764-L1766. That is the right design; counting real FLOPs would cost more than the model does. But it makes the number only as good as its arithmetic, and on Multi-head Latent Attention the arithmetic is wrong:

        # KV projection (always compressed, shared across heads)
        # kv_a: h -> (kv_lora_rank + qk_rope_head_dim)  [replicated]
        flops["kv_a_proj"] = 2 * T * D * (c + r) * L
        # kv_b: kv_lora_rank -> num_heads * (qk_nope + v_head_dim)
        flops["kv_b_proj"] = 2 * T * c * q * (self.qk_nope_head_dim + v_d) * L

        # Attention core
        flops["attn_qk"] = 2 * q * TC * qk_head_dim * L
        flops["attn_av"] = 2 * q * TC * v_d * L

        # Output projection: num_heads * v_head_dim -> h
        flops["out_proj"] = 2 * T * q * v_d * D * L

qk_head_dim is qk_nope_head_dim + qk_rope_head_dim = 192 for DeepSeek-V3, and v_d is 128. Those are the widths of the un-absorbed attention core. §7.2 walked one decode token through vLLM's real MLA path and found the kernel attending against a 576-wide latent row with a 512-wide value slice, because forward_impl folds $W_{UK}$ into the query before the kernel ever runs. The model charges 320 units of head width per context token; the hardware does 1,088. That finding was handed to this chapter to verify and price; it holds, with one correction that makes it cleaner. Section 6 has the arithmetic.

That is the failure mode this chapter exists to prevent, in miniature: a number produced by a model nobody checked against the machine. The cure is not to distrust models — the whole book is models — but to pair every derived bound with a measurement and an explicit account of the difference.

Performance work on an inference engine is three steps in a fixed order, and the order is what makes it a method rather than a hunt. Derive the floor for the specific model, batch and context from §0.4's roofline — this is pure arithmetic and takes two minutes. Measure one step under a trace. Attribute the gap: every microsecond between the floor and the measurement belongs to a named cause, and the trace tells you which.

The branch labels in Figure 1 are the whole diagnostic vocabulary. Four shapes, four causes. Section 4 draws them.

Fix an operating point and never move it mid-analysis: Llama-3-8B, bf16, TP=1, one H100 SXM, batch $B = 32$, context $s = 2048$ tokens per sequence. Symbols follow FORMULAS: $L=32$ layers, $d=4096$, $h=32$ query heads, $h_{kv}=8$, $d_h=128$, $b=2$ bytes, $\beta = 3.35$ TB/s and $\pi = 989.4$ TFLOP/s (cited NVIDIA peaks), $I^{*} = \pi/\beta = 295$ FLOP/byte.

Bytes

§0.4 established that a decode step streams $\text{bytes}_{\text{weights}} = 15.01$ GB — 7.50 B parameters at 2 bytes, the full 8.03 B count minus the input embedding table, which is gathered one row at a time rather than streamed. KV bytes come from the cell size $k = 2 L h_{kv} d_h b = 131{,}072$ bytes = 128 KiB per token, times the resident tokens $B \cdot s$.

Note what batching did to the mix. At batch 1 and short context, §0.4 found 98% of the traffic was weights; at $B=32$, $s=2048$ the KV cache is already a third of it — which is the difference between "quantise the weights" and "quantise the KV" being the right next move.

FLOPs

From FORMULAS, cached decode costs $F = 2P + 4 L h d_h s$ per token, with $P = 7.50\times10^{9}$ the streamed parameter count. Per token: $1.500\times10^{10}$ for the GEMVs and $4 \times 32 \times 32 \times 128 \times 2048 = 1.074\times10^{9}$ for attention — attention is 6.7% of decode FLOPs, matching §0.2's point that the cache saves weight GEMMs, not attention arithmetic. Times $B = 32$:

Placing it, and the prediction

$I = F/Q = 5.144\times10^{11} / 2.360\times10^{10} = 21.8$ FLOP/byte, so $\min(\pi, \beta I) = \beta I = 73.0$ TFLOP/s and $t = Q/\beta = 2.360\times10^{10}/3.35\times10^{12} = 7.04$ ms. Even a perfect implementation of this operating point reaches 7.4% of the H100's bf16 peak. If someone demands 50% MFU on decode, the answer is not "optimise harder", it is "that number is unreachable by a factor of seven at this batch and context, and here is why".

What a measurement would have to show

Three quantities are checkable independently, and each plausible gap has a distinct signature.

The launch-overhead row is where this operating point diverges from §8.1's. §8.1 derived 330 kernel launches for a TP=1 Llama-3-8B step against a cited driver-only 2.374 µs per cudaLaunchKernel (Vellaisamy et al., ISPASS 2025, arXiv:2504.11750, Table V) — 0.78 ms of host time. That was 17% of §8.1's 4.48 ms batch-1 step and "marginal"; it is 11% of this one and comfortably hidden. Batching buys launch headroom as well as throughput. The other direction is the dangerous one: at TP=8 device work falls to roughly an eighth while the launch count rises to 395, giving 0.94 ms of launches against 0.56 ms of work — launch-bound before Python is considered. Derive first and you know which world you are in.

A decode-step trace has two lanes that matter: a host lane of cudaLaunchKernel calls and a device lane of kernels. Everything diagnosable is a relationship between them.

The decision procedure is mechanical. Is the GPU lane gappy? If yes, and the host lane is dense, you are launch-bound; nothing about the kernels matters until you fix that, and §8.1 owns the fix. If the GPU lane is dense, is one kernel most of it? If yes, that kernel is your entire problem — take its name to Nsight Compute. If neither, are the kernels near a roof? Only per-kernel counters answer that, and this is exactly the question where using the wrong tool costs an afternoon: shapes 2 and 4 are visually identical in a timeline and differ only in achieved DRAM throughput. Shape 2 means you are done and the only lever left is fewer bytes. Shape 4 means the kernel is leaving the machine idle; for decode the usual cause is the FlashAttention-2 grid, which at query length 1 is $\text{batch} \times h$ blocks — 1,024 at $B=32$, $h=32$, but only 128 at $B=4$, under-filling 132 SMs before the kernel starts.

Pick by question, not by habit. "The step is 40% slower than the floor" is a Systems question. "This attention kernel is 40% of the step" is a Compute question. "Which op is 40% of the step" is a torch.profiler question.

Both engines expose torch.profiler behind an HTTP endpoint, and differ in one instructive way.

vLLM: configure at launch, arm at runtime

As of a556f3f, vLLM's profiling is a config object, not an environment variable — the old VLLM_TORCH_PROFILER_DIR is gone from vllm/envs.py and a ProfilerConfig has taken its place:

    profiler: ProfilerKind | None = None
    """Which profiler to use. Defaults to None. Options are:

    - 'torch': Use PyTorch profiler.
    - 'cuda': Use CUDA profiler.
    - 'proton': Use Triton Proton profiler."""

    torch_profiler_dir: str = ""
    """Directory to save torch profiler traces. Both AsyncLLM's CPU traces and
    worker's traces (CPU & GPU) will be saved under this directory. Note that
    it must be an absolute path."""

Forget to set it and the engine tells you the exact flags, which is the friendliest error string in this part of the codebase:

    def profile(self, is_start: bool = True, profile_prefix: str | None = None):
        # Check if profiling is enabled
        if self.profiler_config is None or self.profiler_config.profiler is None:
            raise RuntimeError(
                "Profiling is not enabled. Please set --profiler-config to enable "
                "profiling. Example: "
                "'--profiler-config.profiler=torch --profiler-config.torch_profiler_dir"
                "=YOUR_DIR_PATH_TO_DUMP_TRACE'"

The endpoints are two bare POSTs, attached to the app only when a profiler is configured:

@router.post("/start_profile")
async def start_profile(raw_request: Request):
    logger.info("Starting profiler...")
    await engine_client(raw_request).start_profile()
    logger.info("Profiler started.")
    return Response(status_code=200)


@router.post("/stop_profile")
async def stop_profile(raw_request: Request):
    logger.info("Stopping profiler...")
    await engine_client(raw_request).stop_profile()
    logger.info("Profiler stopped.")
    return Response(status_code=200)

Both take no body. When to capture is decided at launch instead, through delay_iterations, max_iterations, and a torch.profiler.schedule built from wait_iterations / warmup_iterations / active_iterations (default 5) in vllm/profiler/wrapper.py:L215-L232. The per-step bookkeeping lives in WorkerProfiler.step (vllm/profiler/wrapper.py:L97-L129), which counts iterations, starts after the delay and auto-stops once max_iterations of recorded steps have elapsed. Without it, a capture on a busy server is gigabytes of trace with no way to find the step you wanted.

SGLang: describe the capture in the request

SGLang puts the same knobs in the POST body. python/sglang/profiler.py is a thin client that builds the request and blocks until the trace is flushed:

    # Start profiler. The API replies when all steps are processed
    # and files are generated.
    json_data = {
        "output_dir": str(output_dir),
        "num_steps": str(num_steps),
        "activities": activities,
        "profile_by_stage": profile_by_stage,
        "merge_profiles": merge_profiles,
        "profile_prefix": profile_prefix,
    }
    if start_step is not None:
        json_data["start_step"] = str(start_step)

    response = requests.post(url=url + "/start_profile", json=json_data)
    response.raise_for_status()

activities is a list from ["CPU", "GPU", "MEM", "RPD"] (python/sglang/srt/managers/io_struct.py:L2073-L2074), plus "CUDA_PROFILER" handled separately. The scheduler arms and disarms by forward-pass counter, checked once per batch:

            # Check profiler
            if (
                self.profiler_target_forward_ct
                and self.profiler_target_forward_ct <= self.get_forward_ct()
            ):
                self._stop_profile()
            if (
                self.profiler_start_forward_ct
                and self.profiler_start_forward_ct == self.get_forward_ct()
            ):
                self._start_profile()

The call site is python/sglang/srt/managers/scheduler.py:L3723-L3724, inside the batch-run path, so start_step counts forward passes, not requests. SGLang also has profile_by_stage, which no vLLM flag matches: it starts a fresh profile the first time it sees a prefill batch and another the first time it sees a decode batch, force-flushing the prefill trace in between (same file, L413-L430). On a continuously-batched server that is the difference between one trace containing both phases mixed and two traces you can read.

The annotations are the reason the trace is readable

A raw kernel timeline cannot tell you what the batch was. Both engines solve this by naming the per-step trace span after the batch composition. SGLang wraps every ModelRunner.forward in a span built here:

    mode = forward_batch.forward_mode
    bs = forward_batch.batch_size
    if mode == ForwardMode.EXTEND:
        ext_toks = forward_batch.extend_num_tokens or 0
        base = f"step[EXTEND bs={bs} toks={ext_toks}"
    else:
        base = f"step[{mode.name} bs={bs}"

    if detailed_annotations:
        suffix = build_detailed_annotation_suffix(forward_batch)
        if suffix:
            base = f"{base} {suffix}"

With detailed_annotations on, a decode span also carries four aggregates whose names are worth memorising, because they are the roofline inputs:

    if mode == ForwardMode.DECODE or mode == ForwardMode.TARGET_VERIFY:
        if seq_lens_cpu is None:
            return ""
        nq = _decode_query_width(forward_batch)
        nkvs = [int(x) for x in seq_lens_cpu.tolist()]
        nqs = [nq] * len(nkvs)
        sq, sk, sqsq, sqsk = _agg(nqs, nkvs)
        # ``sq`` is always emitted (self-contained suffix): it equals ``bs``
        # (vanilla decode) or ``bs * num_tokens_per_req`` (spec draft-decode /
        # target-verify).
        return f"g_sq={sq} g_sqsq={sqsq} g_sqsk={sqsk} g_sk={sk}"

$\Sigma N_Q$, $\Sigma N_{KV}$, $\Sigma N_Q^2$, $\Sigma N_Q N_{KV}$. The last one is the token-context product — the exact quantity attention FLOPs are linear in. vLLM emits the same four under different spelling: sq, sk, sqsq, sqsk, split into context and generation groups, assembled in vllm/v1/worker/gpu_worker.py:L1000-L1024 behind --profiler-config.detailed_trace_annotation. Two projects independently landing on the same four aggregates is decent evidence they are the right four. Turn detailed annotations on and every step span carries its own roofline inputs, so you re-derive the floor for the batch that step actually ran rather than the batch you assumed.

Worked trace: one POST to one trace file

The whole vLLM path, in order, so you know where to put a breakpoint when nothing appears in your trace directory:

1. start_profile handles the POST and calls engine_client(raw_request).start_profile() (vllm/entrypoints/serve/profile/api_router.py:L22-L24).
2. AsyncLLM.start_profile fans out: an engine_core.profile_async(True, profile_prefix) RPC to every worker, plus self.profiler.start on a thread for the front-end CPU trace unless ignore_frontend is set (vllm/v1/engine/async_llm.py:L945-L949).
3. In each worker, Worker.profile(is_start=True) raises if no profiler is configured, builds the rank-qualified trace name from get_worker_rank_suffix, constructs a TorchProfilerWrapper on first call, and calls .start() (vllm/v1/worker/gpu_worker.py:L1146-L1199).
4. WorkerProfiler.start sets _active and, if delay_iterations == 0, calls _call_start → TorchProfilerWrapper._start → torch.profiler.profile.start() (vllm/profiler/wrapper.py:L85-L95, L68-L74, L302-L306).
5. Every subsequent step, Worker.annotate_profile(scheduler_output) calls self.profiler.step() for the delay/limit bookkeeping and returns the annotation context manager that wraps the forward — a record_function for the torch profiler, an NVTX range for the CUDA one (vllm/v1/worker/gpu_worker.py:L930-L942).
6. stop_profile unwinds the same path to TorchProfilerWrapper._stop, which stops Kineto, optionally builds the self_cuda_time_total table, and lets tensorboard_trace_handler write the gzipped trace (vllm/profiler/wrapper.py:L309-L332).

Two failure points live in that chain. If step 3 raises, you get the flag-naming RuntimeError quoted above and nothing else happens. If max_iterations fires in step 5, _call_stop runs without your ever calling /stop_profile, and a later stop logs "Profiler was not started, nothing to stop." — which reads like a bug and is not one.

Nsight Systems on a live server

The "cuda" profiler kind in vLLM is a two-line wrapper around torch.cuda.profiler.start()/.stop() whose annotation context manager is torch.cuda.nvtx.range(name) (vllm/profiler/wrapper.py:L472-L489). SGLang's equivalent is the "CUDA_PROFILER" activity, and its log message says the quiet part out loud:

    if "CUDA_PROFILER" in profile_activities:
        try:
            torch.cuda.cudart().cudaProfilerStart()
            rank_print("CUDA Profiler started (nsys will begin capturing)")
        except Exception as e:
            rank_print(f"Failed to start CUDA profiler: {e}")

Neither engine launches nsys; you wrap the server in it and let the endpoint trip the capture range. SGLang's own test spells out the invocation (test/registered/profiling/test_start_profile.py:L233-L241): nsys profile -c cudaProfilerApi --capture-range-end stop -o OUT python3 -m sglang.launch_server .... vLLM ships a post-processing tool for the result under tools/profiler/nsys_profile_tools/gputrc2graph.py, which shells out to nsys stats -r cuda_gpu_kern_trace and computes non-overlapped GPU cycles per kernel — the number you actually want, because overlapping kernels double-count in a naive sum.

Now back to the opening. §7.2 flagged two defects in MLAAttentionMetrics. Reading the code against §7.2's own trace of the absorbed decode path, one holds exactly and one needs correcting — and the correction makes the remaining error cleaner, not smaller.

The attention core is under-counted by exactly 3.4×. Confirmed. attn_qk charges qk_head_dim = $d_{\text{nope}} + d_{\text{rope}}$ = 192 and attn_av charges v_head_dim = 128, for $2(192+128) = 640$ FLOP per head per context token. §7.2's trace of forward_impl shows the decode kernel receiving a 576-wide query (concat of the 512-wide folded $\tilde q$ and the 64-wide $q^{\text{pe}}$) against the 576-wide latent cache row, with V the leading 512-wide slice: $2(576+512) = 2{,}176$ FLOP per head per context token. $2176/640 = 3.4$.

The spurious kv_b_proj charge is, arithmetically, not spurious. §7.2 is right that the absorbed path never calls kv_b_proj. But it runs two bmms instead — the q-fold $2h\,d_{\text{nope}}r_{kv}$ and the v-fold $2h\,r_{kv}d_v$ — whose sum is $2h\,r_{kv}(d_{\text{nope}}+d_v)$, identically the product 2 * T * c * q * (qk_nope + v_d) the model bills. For DeepSeek-V3 both come to 33.55 MFLOP per layer per token. That line is mislabelled but numerically correct, so the entire error is the attention core — one term, one constant factor. That is a better bug report than "two things are wrong".

What that does to a reported MFU depends on context length, because the error is on the only term that scales with $s$. DeepSeek-V3 activates about 37 B parameters per token, so the non-attention work is roughly $2 \times 37\times10^{9} = 74$ GFLOP per decode token and is unaffected. Reported total versus true total:

If you are tuning a DeepSeek deployment against vLLM's reported TFLOP/s, at 16k context you are reading a number 2.3× too low, and the error grows with your context window. The consequence is a wrong decision: you conclude there is tensor-core headroom that does not exist and spend a week chasing it. The byte model is right — get_read_bytes_breakdown reads kv_compressed_dim = $r_{kv}+d_{\text{rope}}$ from cache on the decode path (vllm/v1/metrics/perf.py:L747-L751) — which is why the GB/s half of the log line can be trusted while the TF/s half cannot.

The unit test that covers MLA decode (tests/v1/metrics/test_perf_metrics.py:L1056-L1087) asserts only on get_write_bytes_breakdown and get_read_bytes_breakdown; the FLOP breakdown is asserted only in the prefill test at L1118-L1122, where the un-absorbed 192/128 geometry is correct. The decode FLOP path is untested. That is the shape of the upstream issue: one term, one constant, one missing test.

Everything above prices one step. Capacity planning is the arithmetic that turns a step time into a replica count and a bill. Here is the template, worked. Change the six inputs and it re-runs.

Step 1 — concurrency from the KV budget. §2.1 derives a 52.32 GiB KV pool for this deployment, holding 428,569 tokens at 128 KiB each. A request's mean resident context over its life is $1800 + 220/2 = 1{,}910$ tokens, so $B_{\max} = \lfloor 428{,}569 / 1{,}910 \rfloor = 224$ concurrent sequences.

Step 2 — price a step at that batch. Step bytes at $B=224$, $\bar s = 1910$: $15.01 + 224 \times 1910 \times 131072 = 71.09$ GB, so $t = 71.09/3.35 = 21.2$ ms, and decode throughput is $224 / 21.2\ \text{ms} = 10{,}556$ tok/s. Note that 21.2 ms is the step time, not the ITL a user sees: the reader's ITL is the step time divided by the fraction of wall clock the replica spends in decode rather than in someone else's prefill. Step 5 comes back to that.

Step 3 — price the prefill. $F_{\text{prefill}}(1800) = 2PS + 2Lhd_hS^2 = 2.700\times10^{13} + 8.49\times10^{11} = 2.785\times10^{13}$ FLOP. At the assumed 40% of 989.4 TFLOP/s that is 70.4 ms per request. Prefill is above the ridge, so it is compute-bound and the MFU assumption is doing real work here — it is the softest number on the page. Replacing it is Lab 11, 11-end-to-end-benchmark's job, not Lab 10's: prefill MFU is a saturated-throughput measurement over a realistic workload on 80 GB-class hardware, whereas Lab 10 is deliberately one decode iteration on a 24 GB card and has no prefill mode.

Step 4 — the GPU-time budget. Prefill and decode contend for the same card. Per request the replica spends 70.4 ms of prefill plus $220/10{,}556 = 20.8$ ms of decode share: 91.2 ms. Saturation is $1000/91.2 = 10.96$ req/s. Prefill is 77% of the GPU time at this input/output ratio — a result that surprises people who assume decode dominates because decode dominates latency.

Step 5 — one consistent queueing proxy. Use the same full GPU service share $S=91.2$ ms that defined saturation, not just the 70.4 ms prefill component. As a deliberately simplified M/M/1 waiting-time model, let $W_{q,99}=S\ln(100\rho)/(1-\rho)$ and approximate TTFT by $W_{q,99}+70.4$ ms. A 900 ms target then gives $\rho\approx0.5579$ and $\lambda=\rho/S\approx6.117$ requests/s. This is a fixed-$B=224$ screening calculation, not a measured capacity or a guarantee about a continuous-batching scheduler.

Step 6 — close the occupancy and ITL model. The memory ceiling $B_{\max}=224$ does not establish the actual live decode batch. Define $t(B)=(15.01\times10^9+B\cdot1910\cdot131072)/(3.35\times10^{12})$ seconds and $S(B)=0.0704+220t(B)/B$. In a stationary fluid model with output flow $220\lambda$, decode duty cycle is $f_d=220\lambda t(B)/B$ and effective ITL is $t(B)/f_d=B/(220\lambda)$. These equations explicitly distinguish kernel time, GPU service share, and per-request residence.

At a binding 60 ms ITL target, set $B=13.2\lambda$ and solve $S(B)\ln(100\lambda S(B))/(1-\lambda S(B))+0.0704=0.9$, requiring $0<\lambda S(B)<1$ and $B\le224$. The higher-load root is $\lambda\approx5.0964$, $B\approx67.27$, $S(B)\approx0.10149$ s, and $\rho\approx0.51725$. A nearest-integer batch and the actual scheduler must be rechecked. The model gives 48 replicas for 240 requests/s and about USD 0.743 per million output tokens at the illustrative USD 3/GPU-hour price. Neither this root nor the earlier fixed-batch estimate is hardware certification: batching changes service distributions, TTFT depends on chunking and queue discipline, and live occupancy is length-biased.

FORMULAS names this chapter the owner of the cost expression, so here is the derivation. A replica of $N_{\text{GPU}}$ cards costs $\text{price}_{\text{GPU-hour}} \cdot N_{\text{GPU}}$ per hour and delivers $\text{goodput}_{\text{tok/s}}$ sellable tokens per second, i.e. $3600 \cdot \text{goodput}$ per hour. Dividing and scaling to $10^{6}$ tokens:

For the fluid-closure example, $10^6\times3/(3600\times5.0964\times220) \approx\$0.743$ per million output tokens. The price is illustrative, not a current cloud quote. Cost is linear in price and inversely proportional to measured goodput. Include CPU, RAM, network, idle reserve, and failure capacity in a real TCO.

Define the goodput policy. A p99 SLO violation does not make every token unbillable. Request-level goodput can count only successful requests meeting latency and quality gates, while a contractual service-period SLO may trigger credits at a different granularity. Report both the successful fraction and the aggregate percentile. At $\rho=0.9$, the fixed-batch proxy predicts $91.2\ln(90)/0.1+70.4\approx4174$ ms TTFT, but this does not determine the fraction billed without that policy.

The same screening model therefore moves cost by about 4.9 times between these SLOs with unchanged hardware. The workload changes the model again. For 500 input and 2000 output tokens, the mean-context estimate is 1500 tokens, $B_{\max}=285$, and the assumed step is about 21.2 ms. Prefill is 19.1 ms and the decode service share is 148.8 ms, so use total $S=167.9$ ms in the queue formula. With a 900 ms TTFT target and 19.1 ms prefill offset, $\rho\approx0.3323$, rate $\approx1.979$ requests/s, and illustrative cost $\approx\$0.211$ per million output tokens. This is about 2.9 times lower than the 1800/220 fixed-batch screen, not a certified engine speedup.

Decode is about 89% of the latter workload's GPU service share, versus about 23% for 1800/220. That changes which optimizations deserve experiments: prefill throughput and reuse for long inputs, decode batching and KV traffic for long outputs. It does not make any optimization universally worthless. Recompute actual occupancy and effective ITL before choosing a fleet size. See §10.1 for joint length distributions and §10.3 for measuring success-conditioned goodput and uncertainty.

Capture five decode steps on a live vLLM server and reconcile them against the 7.04 ms floor:

vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --max-num-seqs 32 \
  --profiler-config.profiler=torch \
  --profiler-config.torch_profiler_dir=/abs/path/traces \
  --profiler-config.detailed_trace_annotation=true \
  --profiler-config.warmup_iterations=3 \
  --profiler-config.active_iterations=5 \
  --profiler-config.ignore_frontend=true

# drive load in another shell, then:
curl -X POST http://localhost:8000/start_profile
sleep 2
curl -X POST http://localhost:8000/stop_profile

Open the trace in Perfetto. Find a span named execute_32_context_0(...)_generation_32(...) — zero context requests, 32 generation requests, i.e. a pure decode step at your batch. Read sk from the generation group: that is $\Sigma N_{KV}$, which times 128 KiB is the KV bytes that step actually read. Add 15.01 GB, divide by 3.35 TB/s, and compare to the span's wall duration. Measure the union of device-active intervals and the critical path; a naive sum double-counts overlapping kernels. Uncovered wall time can include launch gaps, synchronization, communication, and CPU work, not only driver launch overhead. Compare those components against §8.1's illustrative 0.78 ms budget.

SGLANG_TORCH_PROFILER_DIR=/abs/path/traces \
  python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct

python3 -m sglang.profiler --url http://localhost:30000 \
  --num-steps 5 --profile-by-stage --output-dir /abs/path/traces

# Nsight Systems instead, using the engine's own capture range:
nsys profile -c cudaProfilerApi --capture-range-end stop -o /abs/path/step \
  python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct
curl -X POST http://localhost:30000/start_profile \
  -H 'Content-Type: application/json' \
  -d '{"num_steps": 5, "activities": ["CUDA_PROFILER"]}'

SGLang's step spans are named step[DECODE bs=32 g_sq=32 g_sqsq=32 g_sqsk=... g_sk=...] when detailed_annotations is set, so the same reconciliation works verbatim. Full procedure in Lab 10.

• Vellaisamy, Kumar, Hoe et al., Characterizing GPU Kernel Launch Overheads, ISPASS 2025, arXiv:2504.11750. Table V is the source of the 2.374 µs driver-only cudaLaunchKernel figure this book uses on H100 / CUDA 12.6. §8.1 owns its application.
• Williams, Waterman, Patterson, Roofline: An Insightful Visual Performance Model, CACM 2009. The original; §0.4 derives the inference-specific form.
• DistServe, Zhong et al., arXiv:2401.09670 — the goodput framing behind the cost formula, and the paper vLLM's --goodput help text points at.
• PyTorch profiler recipe, pytorch.org, and the memory-timeline post for activities=["MEM"], which SGLang exposes directly (python/sglang/profiler.py:L117-L123).
• vLLM's nsys post-processor, tools/profiler/nsys_profile_tools/gputrc2graph.py — computes non-overlapped GPU cycles per kernel from an nsys report, which is the correct denominator when kernels overlap.
• SGLang's profiling test, test/registered/profiling/test_start_profile.py:L144-L252 — the only in-tree documentation of the nsys profile -c cudaProfilerApi wrapper, including the --capture-range-end stop flag that makes a single capture terminate cleanly.
• Formula sheet, FORMULAS — the cost expression derived here, plus every bound this chapter composed.