Formula sheet

KV cache, one token, all layers — the "cell size"

The leading 2 is K and V. It does not depend on h — only on KV heads, which is exactly why GQA and MQA shrink the cache. 128 KiB/token for Llama-3-8B bf16, 320 KiB/token for 70B. Owned by §2.1.

KV cache, full batch

Linear in total resident tokens, not in batch size. Many short requests and few long ones cost the same. Owned by §2.1.

MLA cell size

One latent vector per token per layer instead of per-head K and V. Owned by §2.1; the mechanism is §7.2. Caveat: With a replicated MLA latent cache, per-rank cache size stays constant and aggregate cache storage grows with TP. Proportionally sharded GQA instead reduces per-rank cache size while keeping aggregate storage constant, until KV-head replication changes that scaling.

Group size, and the cell size written through it

The same ideal cell size, holding query-head count, widths, precision and layers fixed: doubling g halves KV storage and doubles the compulsory-traffic attention intensity. Head replication, cache layout and kernel reuse can break this simple runtime scaling. Owned by §3.5.

Cluster KV capacity, GQA versus MLA

P here is the KV pool per GPU in bytes and N the cluster-wide distinct resident tokens. GQA's capacity grows linearly in TP until TP reaches h_kv and then flattens — that saturation is the max(1, h_kv // tp_size) replication. MLA's is constant only at a fixed per-rank pool: sharding weights can enlarge that pool as TP grows, even though the latent cell remains replicated. The crossover is where min(TP, h_kv) passes c / c_MLA: 4.66 for Llama-3-70B against DeepSeek-V3, so MLA's cluster-KV advantage is spent by TP ≈ 5 (derived). Owned by §3.5; the same law with the weight term folded into P produces §5.1's tensor-parallel capacity table.

Block accounting

The slot index is what the kernel is actually handed. Owned by §2.2.

Block reuse under alignment

For m matching tokens. The loss is bounded by one block — about 15 tokens, or 0.68 ms of Llama-3-8B prefill. Small, and the book says so rather than overselling it. Owned by §2.3.

The VRAM budget

u is gpu_memory_utilization — 0.92 by default in vLLM at this SHA, not the 0.90 usually quoted. vLLM measures the activation term by profiling; SGLang's mem_fraction_static is a heuristic reserve, not a measurement, so the two are not interchangeable. Owned by §2.6.

Maximum concurrency

The single most useful back-of-envelope in inference serving.

KV / weight traffic crossover

The resident-token count at which KV traffic per decode step equals weight traffic — so the point where the two modeled traffic contributions are equal, not a threshold below which KV savings disappear. 1.23 × 10⁵ tokens for Llama-3-8B bf16; 4.31 × 10⁵ for 70B. Below it, fp8 KV reduces modeled step bytes by $r/[2(1+r)]$, where $r=Q_{KV}/Q_W$. At $r=0.5$ the saving is $1/6$ (16.7%); just below the crossover it approaches 25%. The under-1% example applies only near $r=0.018$, not to all $r<1$. Metadata, kernel changes and other work determine actual latency. Owned by §2.5.

Hierarchical tiers

The prefill rate below which fetching from a tier beats recomputing, and how long a tier holds a prefix at arrival rate λ. Owned by §2.6.

Arithmetic intensity and the roofline

This is a bound for the chosen compute/traffic model, not a promised execution rate. State whether $Q$ is compulsory traffic or measured HBM traffic. Better utilization, fewer rereads, launch reduction, communication overlap and different algorithms can all matter. Owned by §0.4.

Prefill GEMM intensity, and where it crosses the ridge

This finite crossing requires $d>bI^{*}$. Equality is approached only as $T$ tends to infinity when $d=bI^{*}$; if $d<bI^{*}$ this compulsory-traffic curve never reaches the ridge.

Intensity grows with the number of tokens in the pass — which makes batching an important throughput lever. The ridge threshold depends on bytes per element, shapes and actual traffic; small chunks may remain bandwidth- or launch-sensitive. Owned by §0.4 and §1.1.

Decode intensities

Both are two orders of magnitude below the 295 ridge, and neither depends on sequence length or batch size. GQA raises attention intensity linearly in the group size. Owned by §0.4.

Decode step floor

About 4.48 ms → 223 tok/s at batch 1 for the illustrated short-context Llama-3-8B bf16 model on H100, assuming 15.01 GB of streamed weights and 3.35 TB/s bandwidth. The full 8.03B-parameter resident allocation is 16.06 GB (14.96 GiB); the input embedding is gathered rather than streamed in full each step. Context KV traffic and other overhead increase latency. The throughput figure assumes one full target-model forward per emitted token, not speculative verification. Quantization changes the byte budget; CUDA graphs address host/launch overhead and can matter independently of proximity to this floor. Owned by §0.4.

Decode-step FLOPs, cached vs uncached

At s = 2560 for Llama-3-8B, attention is only 7.0% of cached decode FLOPs (4.1% at 70B). The cache's payoff is deleting redundant weight GEMMs, not saving attention arithmetic — which is why decode attention kernels optimise a small share of FLOPs but a large share of latency. Owned by §0.2.

Prefill FLOPs

Owned by §1.4, where it prices recompute; restated by §1.5 and §2.3.

Quantised-GEMM crossover batch size

The ideal weight-only ridge batch, where pi_eff is the compute ceiling in FLOP/s and beta the bandwidth in bytes/s. It is not the pairwise point where dequantization costs more than the competing format saves. M* ≈ 76 (W4A16), 148 (W8A16 FP8), 295 (W8A8 FP8) on H100 SXM — which is why kernels dispatch on M. Owned by §4.3.

Effective bits per weight, and where a weight format stops being memory-bound

b stored bits per weight, b_s bits per scale, b_z bits per zero-point, g the group size. The second expression is the batch at which a weight format crosses the ridge, and it contains no model term at all: 295 for bf16, 148 for 8-bit weight-only, 74 for ideal int4 without metadata (about 76 with group-128 scales), and 295 again for W8A8 fp8 because I* doubles with the format. That is why the crossovers are worth memorising and the absolute latencies are not — ideal proportional weight/FLOP sharding cancels from this simplified ridge. Actual TP changes local shapes, replicated parameters and communication, so measured boundaries can move. Own-ridges are not pairwise winner boundaries: the ideal W4/W8A8 crossing is near 148. Owned by §4.1; the decision it feeds is §4.4.

Tensor-core row utilisation and launch overhead

A decode step with one row wastes most of an m16n8k16 tile. Introduced in §0.3; launch overhead is owned by §8.1.

Host budget per kernel, and whether you are launch-bound

A launch is asynchronous, so its cost only hurts when the CPU cannot stay ahead. Divide the device time of one step by the launches in it and compare against the host cost per launch — a cited 2.374 µs of driver time per cudaLaunchKernel on H100 with CUDA 12.6 (Vellaisamy et al., ISPASS 2025, Table V), with the PyTorch and Python layers on top of it and unquantified. For Llama-3-8B this gives 13.6 µs of budget at TP=1 against 0.78 ms of driver time for 330 launches (17% of the assumed device floor) and 1.42 µs at TP=8 against 0.94 ms for 395 launches (derived from an ideal sharded-weight floor). A lower bound on device time cannot establish the actual host bottleneck: collective, KV and compute time can make the device interval longer. Compare measured host submission time with measured device busy time and dependencies. Tensor parallelism can shrink some device work while adding collective launches, making this comparison particularly important. Owned by §8.1.

Materialised attention: traffic and intensity

For square noncausal MHA (h_kv = h), this simplifies to T d_h / [b(T + d_h)], independent of head count at fixed head width. For bf16 and d_h = 128, the large-context limit is 64 FLOP/byte, below the illustrated H100 ridge near 295. This conclusion holds for those widths, dtype and traffic assumptions; different geometry or a nonmaterializing algorithm changes the comparison. Owned by §3.1.

Compulsory attention intensity

Ideal compulsory-traffic attention intensity

For square noncausal MHA with bf16 inputs, counting Q/K/V reads and the output write exactly once gives this ideal curve and a crossing near T = 590. It is not actual FlashAttention intensity: finite SRAM causes tile-dependent rereads, causal masking changes useful work, and GQA changes traffic. FlashAttention removes quadratic score materialization; it does not provide unbounded real reuse with fixed on-chip memory. Owned by §3.2.

Online softmax recurrence

Equal to standard softmax in exact arithmetic for valid finite, nonempty support; floating-point reordering can change the last bits. All-masked rows need an explicit policy. Proved by induction in §3.1, and verified numerically there on a worked 8-logit example. This recurrence is the foundation of FlashAttention and of every split-K decode kernel.

Log-sum-exp merge

How two partial attention results over disjoint key ranges combine. Both engines pass a single fp32 LSE scalar per (token, head) between kernels to represent a partial softmax state — this is the interface that makes split-K possible. Owned by §3.1.

Overflow threshold

Why the max subtraction is not optional.

FlashAttention tile constraints

Shared memory and the register file jointly pin the tile shape; the FA2 grid shows why decode (T_q = 1) leaves the machine idle and needs §3.3's split-K. Owned by §3.2.

MLA decode intensity, and what tensor parallelism does to it

All h query heads attend over one shared latent: each scores against r_kv + d_rope dimensions and outputs over r_kv, while the bytes read are one r_kv + d_rope-wide row per position. At 242 against the H100's 295 ridge, MLA decode attention has a modeled intensity equal to 82% of that ridge, higher than the modest-group GQA examples here. This is not a universal separation from GQA: its intensity also increases with query-to-KV head ratio. MLA trades additional attention-core FLOPs for reduced stored state. The catch is the TP behaviour: query heads shard and the latent does not, so per-rank intensity falls as 1/TP — 242, 30, 15 at TP = 1, 8, 16 (derived), while GQA's I = 2g/b is TP-invariant only while both head counts shard proportionally. Once KV heads replicate across ranks, that invariance no longer holds. Owned by §3.5.

Split-K decode grid

S is the number of KV splits and N_SM = 132 on H100 SXM. S* is the smallest count giving at least one modeled CTA per SM, not a guaranteed occupancy optimum. Extra splits may help latency hiding or load balance while adding merge costs. Actual grids and clamps depend on kernel tiling, sequence lengths, registers and shared memory. Llama-3-8B at batch 1 gives G_2D = 8 and S* = 17; Llama-3-70B at TP=8 batch 1 gives G_2D = 1 and S* = 132, against vLLM's fixed 16 — which is exactly the axis on which SGLang's adaptive policy differs. Owned by §3.3; the same batch-dependence, seen as a bug rather than a feature, is §10.4.

Number formats

Effective bits per weight including group scales: 4.25 for MXFP4 (block 32, E8M0 scale), 4.5 for NVFP4 (block 16, E4M3), 4.125 for int4 with 128-element groups. Owned by §0.5.

Static batch utilisation

For exponentially distributed output lengths. At B = 32 that is roughly a quarter of the machine doing useful work; the complementary approximately 75.4% is wasted on finished rows in this asymptotic exponential-length model. Owned by §1.3.

Chunked prefill traffic and chunk sizing

Here N = S/c is an integer number of equal chunks, P the streamed parameter count, b_w bytes per weight and k cache bytes per token. Unequal final chunks need their actual prefix sum. Both leading terms scale as 1/c, but the weight re-streaming term dominates in the example: the KV re-read is only 13% of it at 32k on Llama-3-8B and would not overtake until ~245k tokens. And chunking ramps ITL rather than flattening it — later chunks attend to more keys. Owned by §1.5.

Recompute vs swap crossover

Where f is the prefix-cache survival fraction of a preemption victim. Largely historical for vLLM, which dropped swapping entirely; still live for SGLang's decode-role disaggregation. Owned by §1.4.

Prefix-cache time saved

Owned by §2.4. Note the reported hit length is capped at num_tokens − 1, so a fully cached prompt can never report 100%.

Disaggregation viability

Prompt length cancels only in this linearized model. It assumes projection-dominated prefill with fixed achieved throughput and ignores handshake latency, quadratic attention, cache hits, queueing, overlap and link contention. A deployment decision needs those workload-dependent costs and the achieved aggregate bandwidth; active rather than total MoE parameters determine the projection term. Owned by §1.6.

Metric definitions

Owned by §1.2. The two benchmark harnesses do not implement these identically — vLLM samples ITL per streamed chunk, SGLang per token — so ITL and TPOT are not comparable across them.

Queueing

The first waiting-time formula assumes a stable M/M/1 queue, $0.01<\rho<1$, and mean service time $S$; for $\rho\le0.01$ the waiting p99 is zero. Exact M/M/1 sojourn p99 is $S\ln(100)/(1-\rho)$, not mean service plus waiting p99. Kingman's approximation assumes a stable single-server queue and is most useful in heavy traffic. Batched inference is not M/M/1; measure its phase breakdown before attributing a tail to queueing. Owned by §1.2; Little's Law is reused in §10.5.

Cost per million output tokens

Define goodput according to the actual acceptance contract, including latency, quality, failures and billing policy. A run-level p99 violation does not automatically make every token worthless, and the chosen counting rule must be stated with the cost figure. Owned by §10.5.

Expected accepted length

For a chain draft of length κ with i.i.d. per-token acceptance α. The bonus token from the target model is why the exponent is κ+1. At $\alpha=1$, use the continuous limit $E=\kappa+1$; at $\kappa=0$, $E=1$. EOS and output caps can truncate this yield.

Batch-aware speedup, and where speculation stops paying

For nonnegative draft cost this model's asymptote is at most one; equality is possible with perfect acceptance and a zero-cost draft. This is not a theorem about every compute-bound implementation. The middle-regime break-even expression requires $T^*/(\kappa+1)\le B^*\le T^*$. B* ≈ 150 concurrent sequences for Llama-3-8B drafted by Llama-3.2-1B on H100 at κ=3, α=0.70. Owned by §6.2.

Speedup condition

With c the cost of one draft step relative to one target step — derivable as the streamed-weight-byte ratio while both are memory-bound. The batch-aware form above says where the loss begins. Owned by §6.2.

Acceptance rate as a distance

Acceptance is exactly one minus the total-variation distance between target and draft. The proof of distribution-preservation turns on the residual identity $\sum_x \max(0, p-q) = 1 - \sum_x \min(p, q)$. Owned by §6.2.

Draft cost as a parameter ratio, and the indifference curve

Under an ideal equally efficient bandwidth-bound model with equal bytes per weight, bandwidth cancels and c is approximated by the ratio of parameters actually streamed — every matmul weight plus the output head, but not the gathered input embedding table. Under equal tensor parallelism the shards divide out too. A Llama-3.2-1B draft costs 16.5% of a Llama-3-8B step and 1.8% of a Llama-3-70B one (derived) — a ninefold change from one target swap with the draft untouched. The second expression is the indifference curve: the locus of (α, c) pairs delivering the same speedup s. Where it crosses c = 0 is the acceptance a zero-cost matcher needs to tie a draft model — 0.43 against the 8B target, 0.67 against the 70B one in the illustrated configuration. These thresholds also depend on draft acceptance, proposal length, target verification and bonus-token costs. The near-flat trend in one target configuration is not a general independence from κ. Owned by §6.3.

Tree verification

T verified nodes, D maximum tree depth, S draft forward passes per iteration, and β_i the probability that the target's continuation at depth i appears among the tree's candidates there, given depth i−1 was accepted. Three checks. Setting topk = 1 collapses it to §6.2's chain exactly. T is absent from the denominator only under a constant-cost, weight-only verification approximation. Being below a nominal ridge does not make added nodes free: attention, KV, masks, activations and launch overhead can grow. Adding candidates while preserving an existing greedy path cannot reduce coverage of that path, but gains need not be strict: extra candidates can have zero target probability. A general tree acceptance probability cannot be compared with a chain's scalar alpha without matching histories, proposal/verification rules and budget. Greedy argmax coverage is not a proof of exact stochastic target sampling. A simple compute-limited extension scales the modeled verification term by TB/I*, suggesting a break-even shrinkage of roughly T/(κ+1); use measured verification cost for real comparisons. The ridge condition itself assumes the stated dtype and traffic model. Owned by §6.4.

Parallel drafting

Emitting all κ draft tokens in one forward pass collapses the cost term. At κ = 3, c = 0.165 that is a 28% larger speedup at unchanged α, and it drops the break-even acceptance against ordinary decoding from 0.34 to 0.14 (derived). Those two break-even points do not prove that a parallel drafter with half a chain drafter's acceptance beats that chain drafter. Compare their complete expected-token/cost ratios, using the parallel forward's actual cost. In the compute-bound approximation, substituting κc → c gives S_∞ = E/((κ+1) + c) ≤ 1; equality requires perfect acceptance and zero added draft cost. Owned by §6.6.

Ring all-reduce volume, and one transformer forward pass

S bytes per collective, p ranks, n tokens in the pass. The 2(p−1)/p coefficient describes the balanced, bandwidth-efficient ring model, not a universal per-rank lower bound across unbalanced algorithms. The volume tends to 2S sent per rank, plus the same received volume, rather than pS. Actual algorithm selection depends on topology, message size and runtime policy. The 2L is the two all-reduces per block. For Llama-3-70B at TP=8 one token costs 4.375 MiB per rank per token (derived). Owned by §5.1.

The comm fraction, and where TP stops paying

B_net is one-way interconnect bandwidth; the numerator counts bytes sent per rank, β HBM bandwidth, P the parameter count. The first expression is linear in batch and in p−1, and for Llama-3-70B on H100 + NVLink it is 2.78 × 10⁻⁴ · n · (p−1) using 450 GB/s one-way NVLink bandwidth: negligible at batch 1, about 50% of the weight-streaming floor at batch 256 and TP=8, over 200% at batch 256 and TP=32 (derived). The second is an interconnect arithmetic intensity — FLOPs of compute per byte of TP traffic — and comparing it against the machine's own π/B_net puts break-even at p ≈ 13 on NVLink and p ≈ 2.35 across InfiniBand NDR for these idealized machine balances. Intra-domain TP is a common recommendation, not a mathematical prohibition on multi-node TP. Note also what neither expression contains: at small batch the binding constraint is not bandwidth but the 2L fixed collective latencies, which TP does not amortise. Owned by §5.1.

Pipeline bubbles and the comparison baseline

S stages, M micro-batches, t one stage on one micro-batch, W the weight bytes one token forces off HBM. S cancels out of the latency. A token still traverses every layer and still forces the same W bytes, under the ideal fixed-TP, added-stage weight-bandwidth model. This does not prove a latency penalty when PP avoids offload or replaces costly TP. Ideal saturated throughput grows with added stages; fixed-total-GPU comparisons change TP and need a new budget. And GPipe's remedy inverts at decode: t is set by weight traffic, which does not shrink when the micro-batch does, so splitting a decode batch into M pieces multiplies makespan by (M+S−1)/S relative to the unsplit batch's St; M=2,S=4 gives 5/4. Owned by §5.2.

DP-attention boundary, and the MoE all-to-all

N the world size, T_g the global token count this step, T_ℓ the rank's local token count, E evenly placed experts, k distinct experts selected uniformly per token. ρ is the expected number of distinct remote destination ranks per token — real dispatch libraries deduplicate, so two chosen experts on one rank cost one crossing. As EP widens, deduplication stops helping: ρ goes 4.63, 6.12, 7.04, 7.56 at EP = 8, 16, 32, 64 for uniform top-8 of 256 experts. Node-limited routing and skew change these expectations. The counter-intuitive result is that the all-to-all is cheaper in bytes than the alternative — 12.88 MB sent per layer per rank with FP8+scale dispatch and bf16 combine against 25.7 MB for gather-plus-TP-FFN at EP=8 (derived). What makes it hard is that per-peer sizes are data-dependent, there are two serialised round trips per layer, and bandwidth drops an order of magnitude outside the NVLink domain. Owned by §5.3.

Collective cost models, and the algorithm crossover

α is the fixed per-invocation cost, α_step the incremental cost of one more lock-step synchronisation, B the achievable unidirectional link bandwidth. Ring achieves the minimum wire bytes and pays 2(N−1) rounds for them; two-shot achieves the same byte count in two rounds because a full mesh lets a rank address every peer at once; one-shot spends (N−1)S bytes to get down to one round. Running vLLM's two hard-coded constants backwards through S* yields an implied barrier cost of 1.75 µs at N = 4 and 3.06 µs at N = 8 — the same microsecond band §0.3 gives for a device-wide sync, growing with N as an N-way rendezvous should. Note S* → ∞ as N → 2, which is why two-shot is never used at world size 2. Owned by §5.4.

Collective barriers per second

Llama-3-70B at TP=8 crosses 160 collective barriers per rank per step against a 5.25 ms floor — roughly 30,000 barriers per second per rank at the roofline (derived). Every one is a chance for ranks to disagree. Mismatched collective sequences can hang or fail; matching shapes and collectives with inconsistent request ordering can instead produce silent wrong results. Plan broadcasts and semantic identity checks address different parts of this correctness contract. Owned by §5.5.

How many experts a batch touches

E routed experts, k experts per token, B tokens in the step. The same coupon-collector shape as ρ in §5.3, applied to experts instead of ranks, and its two limits are the story: E_touched(1) = k, and it reaches E fast. Half the layer by batch 22 for DeepSeek-V3 and 95% by batch 94 (derived) — under independent uniform routing. Small or skewed batches need not touch all experts, and skipped experts need not read their weights. Resident capacity and actual HBM traffic are different budgets. Owned by §7.1.

MoE arithmetic intensity and its ridge batch

d and d_ff cancel completely, so this is a property of E, k and the weight dtype and nothing else. At B = 1 it equals the dense GEMV intensity 2/b_w — at batch 1 an MoE layer and a dense layer are the same kind of object. Asymptotically the slope is the dense slope divided by E/k: 32 for DeepSeek-V3, 16 for Qwen3-30B-A3B. Hence batch 9,456 to reach an H100's fp8 ridge against 296 for a dense layer (derived), under the ideal uniform-routing traffic model. Wide EP is a deployment option, not a universal optimum; local GEMM shapes, skew and interconnect costs matter. Owned by §7.1; the same M* · E/k from the kernel side is §4.4's.

Grouped-GEMM padding factor

m_e the rows routed to expert e, M_blk the tile height. When the mean row count M_e = Bk/E is below the tile height, every touched expert costs a full tile — 2–16× at decode-like batches. It is a tensor-core tax, not a bandwidth one: the padded rows are zeros that still occupy MMA slots. vLLM's autotuned configs independently rediscover M_e, pinning the tile at 2 M_e from batch 256 upward. In the masked layout the capacity is E · m_max, where m_max is the chosen, possibly rounded per-expert reservation, not necessarily the observed maximum expert load. A path may reserve a whole-chunk bound; an optional cap can use the measured maximum. This reserved shape is not automatically the executed row count: kernels may skip invalid rows or tiles. Measure executed work and dispatch traffic separately. Owned by §7.1.

MLA: absorbed versus naive, per layer per query token

The first identity is the whole trick in one line: the two folds cost exactly what the up-projection they replace costs, so absorption moves a fixed cost from the context axis to the query axis rather than deleting work. The second is the price: per-head key width goes 192 → 576 and value width 128 → 512, a flat 3.4× more FLOPs per context token for DeepSeek-V3 — which is what buys the 71× reduction in the modeled per-token cache payload. With every one of the S keys up-projected anew, the rectangular Q·S attention-pair model gives a decode crossover near S* ≈ 1.006 and no FLOP advantage for absorption once Q > 171. These are not universal runtime thresholds. A causal block with prefix P and Q new tokens has P·Q + Q·(Q+1)/2 valid pairs; use that count instead of the rectangle. For cold prefill the projection costs cancel in the comparison, but the total-cost ratio is not 3.4 because projection work remains. Kernel efficiency, reuse and workspace determine measured latency. Owned by §7.2. Note: vLLM's analytic FLOP model charges the naive widths on the decode path and therefore under-counts MLA decode attention by that same 3.4× — see §10.5.

Recurrent state, and where it beats a KV cache

d_conv = d_inner + 2 g d_state, W the depthwise kernel width, H_m SSM heads, b' the SSM dtype width (fp32 by default). Constant in context, but not small: 97.4 MiB per Nemotron-H-8B sequence, of which the conv state is 1.4%. The crossover s* against a KV cache of cell size k is 779 tokens against Llama-3-8B, or 395 with a bf16 SSM dtype (derived), comparing only the recurrent-state component. Nemotron-H is a hybrid: its attention layers additionally store 16 KiB of KV per token. With the fp32 state, the total-hybrid crossover is about 891 tokens. At 128k tokens the comparison is 16 GiB versus 2 GiB of attention KV plus 97.4 MiB of state, about 2.095 GiB total, or 7.64 times smaller. This is a cache/state budget, not a full serving-memory budget. Owned by §7.3.

LoRA: adapter size and the decode multiplier

r the adapter rank, k the number of distinct active adapters in the batch. For Llama-3-8B with all seven linear targets, P(r) = r · 2,621,440 parameters — exactly r · 5 MiB in bf16 — so ρ = r · 3.264 × 10⁻⁴. The modeled distinction is compute work per token versus distinct adapter weights read per iteration. In an ideal compute-bound prefill, the FLOP increment is related to ρ; in an ideal bandwidth-bound decode with equal precision and no cache reuse, adapter traffic scales with k·ρ. The heuristic k·r ≲ 300 is an approximate byte-budget condition, not a measured 10% latency guarantee. Grouping, kernels, launch costs, base precision and adapter caches change runtime. Owned by §7.5.

Image tokens, and the two multimodal ratios

H', W' the resized dimensions, p the ViT patch size and m the spatial merge factor — 14 and 2 for Qwen2-VL, so one merged vision token is exactly 28² = 784 pixels. Under the illustrated resize rounding and without a tighter processor cap, a 1024×1024 screenshot yields 1,369 merged tokens; an un-downscaled 4K one yields 10,549 (derived). Actual counts require the checkpoint's processor settings, resize bounds, merge rule and any special tokens. The estimate I_enc ≈ N_p is a weight-traffic approximation, not a proof that the complete encoder is compute-bound: activation traffic, attention, kernel shapes and hardware utilization also matter. For illustration, 5,476 divided by the approximately 295 FLOP/byte ridge is 18.6, but only a measured kernel roofline can establish the bottleneck. The last formula assumes image embeddings enter the decoder token sequence and equal storage precision. It gives a 16× payload ratio for illustrative Llama-3-8B shapes, not a Qwen2-VL measurement or a universal cross-attention result. Owned by §7.4.

Frontend budget, and what a process split buys

c is the host cost of turning one generated token into bytes on a socket — detokenise, build the delta, serialise, write — and B the batch. One forward pass yields one token per sequence and every sequence has its own socket, so the frontend does B units of c per step. The additive form assumes serialized work on one interpreter's critical path. GPU execution and native code can overlap or release the GIL, and threads/processes or free-threaded builds change the model. Measure the actual critical path rather than infer it from process count. 140 µs of budget at batch 32, 35 µs at batch 128 against the 4.48 ms floor (derived). At equal CPU and GPU stage times the ideal zero-overhead pipeline model gives a 2× gain; real overlap and IPC costs change it. Frontend saturation depends jointly on batch size, step period, emission policy, CPU resources and queueing, not on batch size alone. Owned by §9.3 and §11.2.

Cache dilution: what the load balancer does to your hit rate

N replicas, M distinct cached prefixes one replica's pool holds, W the workload's distinct-prefix working set, λ the arrival rate, L_p the shared prefix length. Under uniform independent requests and equal-size prefixes, random-replica routing gives hit probability M/W; this is not a bound on fleet storage. The union of caches can hold up to NM distinct keys even with round-robin. Balanced affinity narrows each replica's working set, giving the second idealized hit probability. For the chapter's assumed 104 slots per replica and 1,000 prefixes across 8 H100s, these are 10.4% and 83.2%. At 60 requests/s and an assumed 86 ms of eliminated prefill per hit, the difference is about 3.8 GPU-seconds of recompute per wall-second, not measured reclaimable capacity. Skew, variable prefix lengths, queue imbalance and remaining suffix attention change the result. Owned by §9.4.

Grammar bitmask storage

16,032 bytes per request per step at |V| = 128,256, and |Q| × 16,032 to materialise an uncompressed whole state→mask table — 1.6 GB at 100k states. This motivates lazy computation, compressed representations and reuse of identical masks; it does not mean every backend avoids precomputation. Under speculative decoding the illustrated per-step buffer grows to max_num_seqs × (1 + κ) rows. Owned by §6.5.

The prefix-cache ceiling

T_p the GPU time a request's prefill costs and T_d its decode, both amortised to a per-request share of a batched step; φ the prefill fraction; η ∈ [0,1] the fraction of that prefill cost eliminated by a hit, not generally the reused-token fraction. A cached prefix still participates in suffix attention. This is an Amdahl bound under fixed batching, decode cost, resource allocation and negligible lookup overhead. Cache-induced changes to scheduling or memory capacity are outside this simple model.

Its use is to settle an argument before you have it. Worked for Llama-3-8B on an H100: chat (512 in / 1024 out) gives φ = 0.065 and a ceiling of 1.06×; RAG (2048/256) gives φ = 0.440 and 1.66×; an agent loop (8192/128, batch capped at 48 by KV fit) gives φ = 0.702 and 2.71×. These are illustrative model outputs for the chapter's assumed cost-saved fractions, not universal workload ceilings. A measured larger gain can reflect different costs, batch capacity or scheduling; evaluate those assumptions before interpreting the comparison. All derived arithmetic on published shapes; nothing measured. Owned by §13.2.

Rebase debt

r upstream commits per week touching the files you must change, T the length of your project in weeks. R is a rough historical churn indicator, not a conflict count or mathematical upper bound; one commit can cause many conflicts and many commits can cause none. Counted from git log over the 29 weeks ending at each pinned SHA, vllm/v1/core/sched/scheduler.py took 115 commits while vllm/v1/core/sched/interface.py took 7 — both reachable through the same scheduler_cls seam, so extending the abstract base rather than the concrete class is a 16× difference in the cited historical commit counts, not a guaranteed reduction in maintenance effort. Repository metadata, reproducible with the command named in the chapter; not a measurement. Owned by §13.4.

The two-constant workload model

S_in, S_out the input and output lengths, s̄ the mean resident sequence length. Calibrated against this book's constants, Llama-3-8B on an H100 gives c_p = 40.6 µs, c_q = 6.63 × 10⁻¹⁰ s, c_d = 4.96 × 10⁻⁸ s and therefore c_p/c_d ≈ 820. This reduced full-pool model is bilinear in output length and an independently held resident length, not a universal formula for total decode time. If the resident length is approximated by input length plus half the output length, substituting it introduces a quadratic output term. Doubling input length increases the context-dependent attention term, not necessarily total decode-step cost. Under the chapter's illustrative workload assumptions, the split swings from 82% prefill on summarisation to 2% on agentic traffic, with a symmetric-workload balance near x* = 551. Refit constants and include weight traffic, scheduling and overhead before using this approximation for capacity decisions. Owned by §10.1.

Arrival variability is a flag

vLLM draws inter-arrival gaps from a Gamma distribution whose shape parameter is --burstiness; mean 1/λ, variance 1/(λ²κ), so the flag is exactly the reciprocal of Kingman's arrival-variability term. SGLang samples exponential only and is pinned at C_a² = 1. Two engineers benchmarking "at 10 req/s" can therefore differ by 2× in mean queueing delay and 0% in throughput. Owned by §10.1.

Warmup is a queueing problem

An M/M/1 spectral-relaxation planning scale and its expected arrivals, assuming stationary Poisson arrivals, exponential service and rho below one. Three such scales is an illustrative heuristic, not a guarantee of convergence for a batched inference engine. Check rolling occupancy, throughput and latency after warmup or cache flushing. At ρ = 0.9 on the worked Llama-3-8B configuration with 70.6 ms mean service, three such scales correspond to approximately 1,025 arrivals and 80.4 seconds (derived) — and it gets worse the closer you push to the part of the curve worth characterising, which is precisely why short benchmarks systematically understate tail latency. Both harnesses default to 0 or 1 warmup requests. Owned by §10.3.

An overloaded run measures your patience

In an initially empty, constant-rate fluid model with no rejection, abandonment or admission cap, offered load above service capacity has no steady state, so the backlog after T seconds is (λ−μ)T and the reported percentile is a linear function of run duration. 12 s, 59 s and 119 s of p99 wait for 60-, 300- and 600-second runs of the same modeled server at the same offered load (derived). This is the arrival-cohort waiting-time distribution, not necessarily the completed-only distribution observed when collection stops at T. Report run duration, drain policy, pending and failed requests, and admission limits with overload results. Owned by §10.3.

What a difference has to be before you may claim it

An approximate normal-theory minimum detectable difference for independent equal-sized arms, common standard deviation and a two-sided test at α = 0.05 with 80% power. In the final expression MDE is a relative difference and CV is dimensionless. With only one run per arm, within-arm variability cannot be estimated from those runs alone. At an assumed 5% CV, the plug-in approximation gives about 11.4% for three runs per arm; such a small sample calls for uncertainty-aware or small-sample methods, not a hard detectability cutoff. More independent runs and lower variance both improve power. Paired designs instead use the variance of paired differences. A p99 based on 1,000 independent observations has only about ten observations in its upper 1%; report tail uncertainty, dependence and run-to-run variation rather than treating a minimum count as certification. Owned by §10.3.