The decode loop and why the KV cache exists

Autoregressive generation has an ugly property: to produce token $t+1$ you need a forward pass over tokens $1 \dots t$, and to produce token $t+2$ you need a forward pass over tokens $1 \dots t{+}1$. The naive implementation re-runs the whole model over the whole sequence, once per output token. Every previous token's key and value vectors get recomputed from scratch at every single step, from identical inputs, to identical values.

Here is what that costs on one real model. Llama-3-8B, bf16, batch 1, a 2048-token prompt, 512 post-prefill decode forwards (513 sampled outputs including prefill's first token). Both columns are arithmetic from published shapes (L=32, d=4096, h=32, h_kv=8, d_h=128, d_ff=14336, vocab 128256, $P = 8.03\times10^{9}$), not measurements.

The last number is the twist, and it is what the rest of the book is about. The cache does not make the problem small. It converts an arithmetic problem into a memory problem: a cached decode step performs roughly one floating-point operation per byte it moves, which on any modern accelerator means the GPU spends its time waiting on HBM. Chapter §0.4 makes that precise; this chapter derives why it is inevitable.

A decoder-only transformer with causal masking has one structural gift: the key and value vectors for token $i$ depend only on tokens $1 \dots i$. They cannot change when token $i{+}1$ arrives, because token $i$ is not allowed to look forward. So every K and V you compute is permanently correct and can be written down once. The query vector is different — at each step you have exactly one new query, you use it once, and you throw it away. That asymmetry is the whole design. Keys and values are state; queries are transient.

Both columns compute the same last row of logits. The left one computes 2,559 other rows first and throws them away.

Symbols, matching the formula sheet: $L$ layers, hidden size $d$, $h$ query heads, $h_{kv}$ key/value heads, head dimension $d_h$, FFN intermediate size $d_{ff}$, total parameter count $P$, bytes per cached element $b$ (2 for bf16). Let $p$ be the prompt length and $n$ the number of subsequent decode forwards. Prefill already samples the first output; $n$ more forwards produce $n+1$ output tokens in total. At decode step $t \in \{1 \dots n\}$, $s_t=p+t$ is the processed context, including the previously sampled token fed into this forward, not the output about to be sampled. Generating exactly 512 outputs would instead require 511 such forwards.

Cost of one cached decode step

The input to the layer stack is a single row $x \in \mathbb{R}^{1 \times d}$. Per layer:

• QKV projection. $W_q \in \mathbb{R}^{d \times h d_h}$, $W_k, W_v \in \mathbb{R}^{d \times h_{kv} d_h}$. A matrix-vector product of an $m \times k$ matrix costs $2mk$ FLOPs (one multiply, one add per element), so this is $2d(h d_h) + 4 d (h_{kv} d_h)$.
• Attention. One query per head, $s_t$ keys. $QK^\top$ is $2 h d_h s_t$; the weighted sum over $V$ is another $2 h d_h s_t$. Softmax is $O(h s_t)$ and drops out. Total $4 h d_h s_t$.
• Output projection. $2 (h d_h) d$.
• MLP. SwiGLU has three matrices of shape $d \times d_{ff}$: $2 \cdot 3 d d_{ff} = 6 d d_{ff}$.

Every term except attention is a fixed cost per token that depends only on weight shapes. We use the common approximate $2P$ projection model below. It is not an exact parameter identity: input embeddings are gathered, normalization has different costs, and an optimized uncached implementation can project only the last hidden state into vocabulary logits. The roofline chapter distinguishes approximately 7.50B streamed matrix parameters from 8.03B resident parameters. The table here intentionally uses the coarse 8.03B accounting consistently, so its ratios are estimates, not exact implementation FLOP counts:

Sanity check on Llama-3-8B: one layer holds $d(h d_h) + 2 d (h_{kv} d_h) + (h d_h)d = 41{,}943{,}040$ attention parameters and $3 d d_{ff} = 176{,}160{,}768$ MLP parameters, $218{,}103{,}808$ in total, so $2 \times$ that is 436.2 MFLOP per layer — which is what you get by adding 50.3 (QKV) + 33.6 (o_proj) + 352.3 (MLP) MFLOP directly.

Cost of one uncached decode step

Without a cache the step must reconstruct every K and V, which means a full forward pass over $s_t$ tokens. Every per-token weight cost is now paid $s_t$ times. Attention becomes $s_t$ queries against $s_t$ keys; a causal kernel skips the masked half, so it is $\tfrac{1}{2} \cdot 4 h d_h s_t^2$ per layer.

The two headline asymptotics fall straight out. Attention work per step goes from $\Theta(s^2)$ to $\Theta(s)$; weight-GEMM work per step goes from $\Theta(s)$ to $\Theta(1)$. Summed over $n$ steps with $p = 0$, generating $n$ tokens costs $O(n^3)$ attention and $O(n^2)$ projection FLOPs without a cache, versus $O(n^2)$ and $O(n)$ with one.

The Llama-3-8B example, all the way through

$p = 2048$, $n = 512$, so $s_t$ runs 2049 … 2560.

Read the last row before anything else. Attention is a minority of decode FLOPs at 2.5k context on an 8B model — 7% — and the cache's main job is not saving attention arithmetic. Its main job is deleting the 2,559 redundant copies of the weight GEMMs. That is why the ratio is 2,224× and not 2×.

Now count bytes, because bytes are what you actually pay

Per cached decode step, batch 1, a GPU must move: every weight, once ($P \cdot b = 16.06$ GB); every cached K and V element, once ($2 L h_{kv} d_h b s_t$ bytes = 335 MB at $s = 2560$); plus the 128 KiB of new KV it writes. Arithmetic intensity at $s = 2304$ (the midpoint):

The uncached step moves almost exactly the same weight bytes — you still stream the model once — but performs 2,460× the arithmetic, giving $I \approx 2.7\times10^{3}$ FLOP/byte. The cache does not make decode faster by making it cheaper in bytes. It makes it faster by deleting arithmetic, and in doing so it moves the workload from the compute-bound side of the roofline to the memory-bound side. Every later chapter — quantization, MLA, paged memory, CUDA graphs, speculative decoding — is an attack on the memory-bound regime the cache creates.

Over the full 512-step decode phase the machine moves 7,658 GiB of weights and 144 GiB of KV. At the 3.35 TB/s HBM3 figure NVIDIA publishes for H100 SXM (datasheet), that is a floor of 2.5 s, or 4.9 ms per token — a derived roofline bound against a vendor peak, not a benchmark. Note the shape of it: 98% of the traffic is weights, at batch 1. Batching amortises the weight read across sequences and does nothing for the KV read, which is why the KV term dominates capacity planning and the weight term dominates single-stream latency.

Three things are true of a decoder-only transformer, and only the first is about caching.

The actual precondition is invariant cached states. Extending a bidirectional segment can change earlier hidden states and invalidate its old K/V; a causal decoder avoids that dependency. However, a fixed encoder output, cross-attention K/V, or an immutable bidirectional prefix can still be cached. The following vLLM policy gives encoder-only layers no autoregressive cache group; it is not a theorem that noncausal computation cannot be cached. Reuse additionally requires matching weights/adapters, positions, masks, and any multimodal inputs.

    def get_kv_cache_spec(self, vllm_config: VllmConfig) -> KVCacheSpec | None:
        # Block size may get updated after model loading, refresh it
        block_size = vllm_config.cache_config.block_size
        # Encoder-only attention is prefill-only and keeps no autoregressive KV
        # cache. In hybrid models (e.g. Qwen3.5 / ColQwen3.5: GatedDeltaNet
        # linear_attention interleaved with full_attention) the runner iterates
        # every attention module to build the KV-cache spec, so an ENCODER_ONLY
        # full_attention layer reaches here; it contributes no KV cache group.
        if self.attn_type in (AttentionType.ENCODER_ONLY, AttentionType.ENCODER):
            return None

The same assumption sits in the attention metadata as a plain default. As of a556f3f, CommonAttentionMetadata carries causal: bool | torch.Tensor = True at vllm/v1/attention/backend.py:L485 — causal is the default for the decoder use case, not a universal condition on every reusable tensor.

Two fields above it encode the entire asymmetry of this chapter:

    query_start_loc: torch.Tensor
    query_start_loc_cpu: torch.Tensor
    """(batch_size + 1,), the start location of each request in query Tensor"""

    seq_lens: torch.Tensor
    """(batch_size,), the number of computed tokens for each request"""

    num_reqs: int
    """Number of requests"""
    # TODO(lucas): rename to num_tokens since it may be padded and this is misleading
    num_actual_tokens: int
    """Total number of tokens in batch"""
    max_query_len: int
    """Longest query in batch"""
    max_seq_len: int
    """Longest context length (may be an upper bound)"""

Query length and sequence length are separate tensors. In fresh, unchunked prefill with no reused prefix they are equal; chunked or warm-prefix prefill can have a shorter query than its total context. In decode, query length is 1 and sequence length is $s_t$. Every attention backend in vLLM is written against that pair, because that pair is the KV cache.

The cache is not free. Its size is fixed by four model constants and one runtime variable:

The leading 2 is K and V. The dependence is on $h_{kv}$, not $h$ — which is the entire point of grouped-query attention, and worth 4× on Llama-3-8B (8 KV heads instead of 32). Note also what is missing: batch size $B$ appears only inside the sum. The cache is linear in total resident tokens, not in the number of sequences.

For Llama-3-8B in bf16: $2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072$ bytes = 128 KiB per token, or 4 KiB per token per layer. The 2560-token sequence from the worked example holds 320 MiB. Thirty-two such sequences hold 10 GiB; sixty-four sequences at 8k context hold 64 GiB, which does not fit on an 80 GB H100 alongside 16 GB of weights. Turning that observation into an admission-control policy is chapter §2.1, and storing it without wasting half of it to fragmentation is §2.2.

vLLM writes this formula down almost literally, as a property on the per-layer cache spec:

    @property
    def state_content_size_bytes(self) -> int:
        """Bytes per (head slot, stored state) cell of the page."""
        if self.state_content_bytes is not None:
            return self.state_content_bytes
        return (self.head_size + self.head_size_v) * get_dtype_size(self.dtype)

    @property
    def unpadded_page_size_bytes(self) -> int:
        return self.num_heads * self.storage_block_size * self.state_content_size_bytes

(head_size + head_size_v) * dtype_size is the $2 d_h b$ of the formula (split into K and V head dims, which differ only under MLA — §7.2). Multiply by num_heads, which resolves to num_kv_heads, and by storage_block_size tokens, and you have bytes per layer per block. The $L$ in the formula is the number of layers each contributing such a spec.

Both engines make the same structural choice — keep K and V in a large pre-allocated pool, pass exactly one token per sequence during decode — and both name the two resulting regimes explicitly. SGLang names them in an enum:

class ForwardMode(IntEnum):
    # Extend a sequence. The KV cache of the beginning part of the sequence is already computed (e.g., system prompt).
    # It is also called "prefill" in common terminology.
    EXTEND = auto()
    # Decode one token.
    DECODE = auto()
    # Contains both EXTEND and DECODE when doing chunked prefill.
    MIXED = auto()
    # No sequence to forward. For data parallel attention, some workers will be IDLE if no sequence are allocated.
    IDLE = auto()

The comment on EXTEND is doing quiet work: SGLang's prefill mode is not “run the prompt”, it is “extend a sequence whose prefix may already be cached”. That framing is why RadixAttention (§2.4) fits so naturally into SGLang — the mode was designed around a partially populated cache. vLLM reaches the same place from the other direction, with a single unified path in which prefill is just a request whose num_scheduled_tokens happens to be large.

The one-token-per-decode property shows up as arithmetic in the scheduler, not as a branch. For a running request:

            num_new_tokens = (
                request.num_tokens_with_spec
                + request.num_output_placeholders
                - request.num_computed_tokens
            )
            if 0 < self.scheduler_config.long_prefill_token_threshold < num_new_tokens:
                num_new_tokens = self.scheduler_config.long_prefill_token_threshold
            num_new_tokens = min(
                num_new_tokens, token_budget, input_budget - draft_slots
            )

A request mid-generation has computed everything but the token just sampled, so num_tokens_with_spec - num_computed_tokens evaluates to 1 (more with speculative decoding, which is exactly why the field is named _with_spec). Decode is not a special case; it is the general case with the difference equal to one.

SGLang states it directly instead, because ForwardMode is available:

    def _init_ngram_embedding_info(self, batch: ScheduleBatch, device: torch.device):
        if self.forward_mode.is_decode():
            column_starts, req_lens = self.seq_lens - 1, 1
        else:
            column_starts, req_lens = self.extend_prefix_lens, self.extend_seq_lens

Same statement at the position level: in decode the position of the new token is simply seq_lens, with no per-token offset array needed at all.

        # Init position information
        if ret.forward_mode.is_decode() or ret.forward_mode.is_target_verify():
            if ret.positions is None:
                ret.positions = clamp_position(batch.seq_lens)
        else:

The storage layouts differ in a way worth noticing. SGLang's MHA pool keeps K and V as two separate Python lists of per-layer tensors:

    def _kv_buffer_shapes(self):
        """(k_shape, v_shape)"""
        if self.use_hnd:
            return (
                (self.num_pages, self.head_num, self.page_size, self.head_dim),
                (self.num_pages, self.head_num, self.page_size, self.v_head_dim),
            )
        rows = self.size + self.page_size
        return (
            (rows, self.head_num, self.head_dim),
            (rows, self.head_num, self.v_head_dim),
        )

                    k_shape, v_shape = self._kv_buffer_shapes()
                    self.k_buffer = [
                        torch.zeros(k_shape, dtype=self.store_dtype, device=self.device)
                        for _ in range(self.layer_num)
                    ]
                    self.v_buffer = [
                        torch.zeros(v_shape, dtype=self.store_dtype, device=self.device)
                        for _ in range(self.layer_num)
                    ]

There is the size formula as an allocation: $2$ (two lists) $\times$ layer_num ($L$) $\times$ rows (token slots) $\times$ head_num ($h_{kv}$) $\times$ head_dim ($d_h$) $\times$ itemsize ($b$). vLLM instead lets each attention backend declare its own packing via AttentionBackend.get_kv_cache_shape (vllm/v1/attention/backend.py:L89-L96) and reduces everything to a page_size_bytes the allocator can reason about. The tradeoff is real: SGLang's concrete layout makes kernels and PD transfer simple to write against; vLLM's abstraction lets FlashAttention, FlashInfer, Triton and MLA backends each choose a layout their kernel prefers, at the cost of a level of indirection everywhere the cache is touched. Both converge on the same non-negotiable: startup-sized backing pools whose slots are reused. The excerpt itself allocates separate per-layer K and V tensors, not one global contiguous allocation.

SGLang's write path takes K and V and nothing else — the API itself refuses to cache a query:

    def set_kv_buffer(
        self,
        layer: RadixAttention,
        loc_info,
        cache_k: torch.Tensor,
        cache_v: torch.Tensor,
        k_scale: Optional[float] = None,
        v_scale: Optional[float] = None,
        layer_id_override: Optional[int] = None,
        dcp_kv_mask: Optional[torch.Tensor] = None,
    ):

Follow the 512th post-prefill decode forward of the running example through a556f3f.

1. The loop. EngineCore.step() is the outer loop of the whole system — schedule, execute, sample, update:

    def step(self) -> tuple[dict[int, EngineCoreOutputs], bool]:
        """Schedule, execute, and make output.
        # ...
        """
        # Check for any requests remaining in the scheduler - unfinished,
        # or finished and not yet removed from the batch.
        if not self.scheduler.has_requests():
            return {}, False
        scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())
        future = self.model_executor.execute_model(scheduler_output, non_block=True)
        grammar_output = self.scheduler.get_grammar_bitmask(scheduler_output)
        # ...
        engine_core_outputs = self.scheduler.update_from_output(
            scheduler_output, model_output
        )

2. Scheduling. Scheduler.schedule() computes num_new_tokens = 1 for our request by the subtraction quoted above (vllm/v1/core/sched/scheduler.py:L566-L575) and records it in scheduler_output.num_scheduled_tokens.

3. Input preparation. The model runner turns scheduler_output into flat GPU tensors, flattening the whole batch into one ragged token stream. Two GPU runners ship at this SHA and they do it differently, so it matters which one you are reading:

In V1, GPUModelRunner.execute_model() (vllm/v1/worker/gpu_model_runner.py:L4287-L4291) calls _prepare_inputs(); in V2, GPUModelRunner.execute_model() (vllm/v1/worker/gpu/model_runner.py:L1416-L1424) calls the public prepare_inputs(). The V1 body is the piece worth reading twice, because it is where prefill and decode stop being different:

        # Get request indices.
        # E.g., [2, 5, 3] -> [0, 0, 1, 1, 1, 1, 1, 2, 2, 2]
        req_indices = np.repeat(self.arange_np[:num_reqs], num_scheduled_tokens)

        # cu_num_tokens: [2, 5, 3] -> [2, 7, 10]
        # self.query_pos.np[:10]: [0, 1, 0, 1, 2, 3, 4, 0, 1, 2]
        cu_num_tokens = self._get_cumsum_and_arange(
            num_scheduled_tokens, self.query_pos.np
        )

        # Get positions.
        positions_np = (
            self.input_batch.num_computed_tokens_cpu[req_indices]
            + self.query_pos.np[: cu_num_tokens[-1]]
        )

The worked comment shows a batch of three requests scheduled for 2, 5 and 3 tokens. A pure-decode batch is the case where num_scheduled_tokens is all ones: then req_indices is [0, 1, 2, ...], query_pos is all zeros, and positions_np collapses to num_computed_tokens — one position per sequence, exactly the SGLang clamp_position(seq_lens) above.

4. Slot assignment. The device-side companion — still V1; V2 fuses the same two lines into prepare_pos_seq_lens — computes positions and sequence lengths and hands them to the block table to resolve where in the pool each new K/V goes:

        self.positions[:total_num_scheduled_tokens] = (
            self.num_computed_tokens[req_indices_gpu].to(torch.int64)
            + self.query_pos.gpu[:total_num_scheduled_tokens]
        )
        self.seq_lens[:num_reqs] = (
            self.num_computed_tokens[:num_reqs] + num_scheduled_tokens_gpu
        )
        self.seq_lens[num_reqs:].fill_(0)

        self.input_batch.block_table.compute_slot_mapping(
            num_reqs,
            self.query_start_loc.gpu[: num_reqs + 1],
            self.positions[:total_num_scheduled_tokens],
        )

Here positions has one entry per scheduled token and seq_lens one entry per request. In our decode step they have the same length and differ by one: positions = [2559, ...], seq_lens = [2560, ...]. How compute_slot_mapping turns a position into a physical address is PagedAttention's job — §2.2.

5. The layer. Inside the model, the shape split is explicit:

    def forward(
        self,
        positions: torch.Tensor,
        hidden_states: torch.Tensor,
    ) -> torch.Tensor:
        qkv, _ = self.qkv_proj(hidden_states)
        q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
        q, k = self.rotary_emb(positions, q, k)
        attn_output = self.attn(q, k, v)
        output, _ = self.o_proj(attn_output)
        return output

With self.q_size = self.num_heads * self.head_dim and self.kv_size = self.num_kv_heads * self.head_dim (vllm/model_executor/models/llama.py:L157-L158), that split is 4096 / 1024 / 1024 for Llama-3-8B at TP=1 — the 4:1 GQA ratio, visible in one line. Note that rotary_emb is applied to q and k only: what lands in the cache is post-RoPE K, which is why positions must be correct before the write and why a bug in positions corrupts the cache silently rather than crashing. self.attn(q, k, v) is the Attention module, which writes K and V into self.kv_cache at the slot mapping and runs the backend kernel against the full seq_lens context.

6. Sample, append, repeat. sample_tokens reduces the single row of logits to one token id, update_from_output appends it to the request, and num_computed_tokens advances by one — making the next num_new_tokens subtraction yield 1 again. The loop is closed.

Position drift corrupts the cache without an error. Because RoPE is applied before the cache write, an off-by-one in positions produces K vectors rotated to the wrong angle. Attention still runs, shapes still match, and the model degrades into plausible nonsense a few hundred tokens in. There is no exception to catch. The debugging move is to dump positions and seq_lens for a single-request batch and assert positions[-1] == seq_lens[0] - 1 at every step.

Treating the cache as a compute optimisation. Engineers who arrive from training reach for bigger batches to fix slow decode, then are surprised that per-token latency barely improves. At batch 1 in our example, 98% of the bytes moved are weights; batching amortises exactly those and leaves the KV read growing linearly with batch. Batching fixes throughput; it does not fix TTFT or single-stream TPOT. Those metrics get defined properly in §1.2.

Assuming the cache is proportional to $h$. Sizing Llama-3-8B's cache from 32 query heads gives 512 KiB per token instead of 128 KiB — a 4× over-estimate, and the wrong number to plan a fleet on. The formula depends on $h_{kv}$. On an MQA model ($h_{kv}=1$) the gap is 32×; on MLA it is a different formula entirely.

Expecting the cache to shrink. For full attention without eviction, the logical context grows during generation. Sliding windows, speculative rollback, and preemption can release active entries; prefix caching can retain reusable entries after EOS. A long-running full-attention request can therefore hold a large footprint throughout its lifetime, which is why preemption and recomputation exist, and why the scheduler (§1.4) is fundamentally a memory allocator wearing a batching costume.

Measure the ratio yourself on CPU, no GPU required. HuggingFace transformers exposes the flag directly. This example downloads a public checkpoint on first use; the warmup is excluded and both paths generate the same number of tokens. Timing is illustrative, not a GPU speedup prediction:

python3 - <<'PY'
import time, torch
torch.set_num_threads(1)
torch.manual_seed(7)
from transformers import AutoModelForCausalLM, AutoTokenizer
m = "HuggingFaceTB/SmolLM2-135M"          # small enough to run on a laptop CPU
tok = AutoTokenizer.from_pretrained(m)
mod = AutoModelForCausalLM.from_pretrained(m, dtype=torch.float32).eval()
ids = tok("The capital of France is", return_tensors="pt").input_ids
outputs = []
with torch.inference_mode():
    for use_cache in (True, False):
        mod.generate(ids, min_new_tokens=4, max_new_tokens=4,
                     do_sample=False, use_cache=use_cache)
        t0 = time.perf_counter()
        out = mod.generate(ids, min_new_tokens=128, max_new_tokens=128,
                           do_sample=False, use_cache=use_cache)
        elapsed = time.perf_counter() - t0
        assert out.shape[1] - ids.shape[1] == 128
        outputs.append(out)
        print(f"use_cache={use_cache}: {elapsed:.2f}s, 128 output tokens")
assert torch.equal(outputs[0], outputs[1])
PY

Then confirm the shapes claim: run one forward pass with use_cache=True, take the layer-0 key tensor out of the returned past_key_values, and check its logical shape is [batch, h_kv, seq, d_h] for this model/backend. The dimension to stare at is the second one — it is the KV head count, not the query head count. On a GQA model the two differ, and every capacity mistake in this chapter's pitfalls list comes from confusing them. (The exact accessor moved when transformers replaced the legacy tuple with a Cache object; inspect the object rather than assuming an index.)

On the engine side, the fastest way to see the size formula bite is to start vLLM and read its own startup accounting, which prints the KV pool it allocated and the concurrency that implies:

vllm serve meta-llama/Meta-Llama-3-8B-Instruct --max-model-len 4096 2>&1 | grep -i "GPU KV cache size\|maximum concurrency"

Divide the reported KV cache size in tokens by 4096 and you should recover the maximum concurrency line; multiply by 131,072 bytes and you should recover the pool size in bytes. If those three numbers are consistent, you have verified the formula against a running engine.

• Shazeer, “Fast Transformer Decoding: One Write-Head is All You Need” (2019) — the paper that identified the KV cache as the decode bottleneck and proposed MQA. Section 2 is the cleanest published statement of the memory-bandwidth argument this chapter derives.
• Ainslie et al., “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints” (2023) — why Llama-3 has $h_{kv} = 8$ and not 32.
• Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (SOSP 2023) — the vLLM paper. Read Section 3 now for the memory-waste measurements; save Section 4 for §2.2.
• Zheng et al., “SGLang: Efficient Execution of Structured Language Model Programs” — the RadixAttention paper, whose central premise is that the “extend” framing quoted above is the right one.
• “vLLM V1: A Major Upgrade to vLLM's Core Architecture” (27 Jan 2025), linked from docs/usage/v1_guide.md:L28-L29 at this SHA, plus RFC #18571, the V0 deprecation thread it points at. Between them they document why the scheduler stopped having separate prefill and decode batches and became the single num_scheduled_tokens dictionary quoted above.
• Pope et al., “Efficiently Scaling Transformer Inference” (2022) — the analytic cost model this chapter's bytes-and-FLOPs accounting follows, extended to multi-chip partitioning.