The KV cache size formula

You have an 80 GB H100 and a Qwen3-32B checkpoint. The weights are 61 GiB in bf16, which fits with room to spare, so you start the server. It comes up. Then the second concurrent user arrives and throughput collapses; by the fourth, requests are being preempted and recomputed. Push the context window up and vLLM refuses to start at all:

        raise ValueError(
            f"To serve at least one request with the model's max seq len "
            f"({max_model_len}), ({format_gib(needed_memory)} GiB KV "
            f"cache is needed, which is larger than the available KV cache "
            f"memory ({format_gib(available_memory)} GiB). {estimated_msg}"
            f"Try increasing `gpu_memory_utilization` (which also controls "
            f"CPU memory on the CPU backend) or decreasing `max_model_len` "
            f"when initializing the engine. "
            f"See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ "
            f"for more details."
        )

Nothing here is a bug. The card had 6.3 GiB left for KV after weights, activations and headroom, and one Qwen3-32B token costs 256 KiB, so the pool holds ~25,600 tokens — three 8k conversations. The whole failure is arithmetic you could have done before provisioning. §0.2 established why the cache exists and quoted 128 KiB per token for Llama-3-8B; this chapter derives that number from layer shapes, generalises it to four architectures, and turns it into a fleet plan.

An 80 GB H100 is a fixed box with four tenants, only one of them elastic. Weights are a rent paid once, identical whether you serve one request or a thousand. Activations and CUDA graph buffers are a roughly fixed working set sized by the prefill chunk and the largest captured batch. The engine holds back a slice as headroom against fragmentation. Whatever survives is the KV pool, and the KV pool is your concurrency — there is no second lever.

The middle bar is the whole chapter in one picture: Qwen3-32B has 4× the parameters of Llama-3-8B but does not get 1/4 of the concurrency — it gets 1/17, because weights eat the budget before the division happens.

Walk one layer and count what must survive to the next decode step. The layer takes a token's hidden state $x \in \mathbb{R}^{d}$ and projects it three ways:

with $L$ layers, $h$ query heads, $h_{kv}$ key/value heads, head dimension $d_h$, and $b$ bytes per stored element. Attention computes $\text{softmax}(qK^\top/\sqrt{d_h})V$ against all previous positions. Three facts fall out:

Multiplying through gives the formula this book uses everywhere, for a batch of $B$ requests with lengths $s_1 \ldots s_B$:

Everything before the sum is a per-token constant fixed by the checkpoint and the cache dtype. Call it $c$, the cell size — SGLang's own name for it. For Llama-3-8B in bf16 ($L=32$, $h_{kv}=8$, $d_h=128$, $b=2$):

Check it a second way: one K vector is $8 \times 128 = 1024$ elements at 2 bytes = 2 KiB, K plus V is 4 KiB per layer, 32 layers is 128 KiB. Run that agreement check whenever you meet a new architecture.

Where MLA breaks the formula

DeepSeek-V3 stores one compressed latent vector per token per layer, of width $\text{kv\_lora\_rank} + \text{qk\_rope\_head\_dim}$, and reconstructs per-head keys and values inside the kernel. vLLM encodes that by setting the KV-head count to one and the V-side head size to zero:

@dataclass(frozen=True, kw_only=True)
class MLAAttentionSpec(FullAttentionSpec):
    # TODO(Lucas/Chen): less hacky way to do this
    cache_dtype_str: str | None = None
    # DeepseekV4 only fields. Non-DeepseekV4 MLA models leave these at defaults.
    alignment: int | None = None  # Default to None for no padding.
    compress_ratio: int = 1  # Default to 1 for no compression.
    model_version: str | None = None
    # Marks draft groups that flatten a non-causal query block into decode rows.
    non_causal_multi_token_decode: bool = False
    # MLA stores a single latent vector per state; there is no separate V.
    head_size_v: int = 0

The layer supplies num_kv_heads=1 and a head size that is the sum of the two latent widths (vllm/model_executor/layers/attention/mla_attention.py:L438 and :L1193-L1197), so the formula collapses to

with no factor of 2 and no head count. For DeepSeek-V3 ($L = 61$, $r_{kv} = 512$, $d_{\text{rope}} = 64$, $b = 2$): $61 \times 576 \times 2 = 70{,}272$ bytes, or 68.6 KiB per token — smaller than Llama-3-8B, for a 671B-parameter model. Why it works, and what it costs in kernel complexity, is §7.2; here it is just a different $c$.

Every shape below was read from the model's published config.json on the Hugging Face Hub (fetched August 2026), not from memory — Llama-3 from the ungated NousResearch/Meta-Llama-3-{8B,70B} mirrors, Qwen from Qwen/Qwen3-32B and Qwen/Qwen2.5-72B-Instruct, DeepSeek-V3 from deepseek-ai/DeepSeek-V3. Parameter counts are recomputed from those shapes rather than quoted, and land on the published headline figures (8.03B, 70.6B, 32.8B) — itself a check that the shapes were read correctly.

*DeepSeek-V3's config declares num_key_value_heads: 128, qk_nope_head_dim: 128, qk_rope_head_dim: 64, v_head_dim: 128. Those are the logical attention shapes, not what is stored; the 1 and 576 in the table are what the engines allocate. Plugging the logical numbers into the GQA formula — $61 \times 128 \times (192 + 128) \times 2$ — gives 4.77 MiB per token, a 71× over-estimate, and it is the most common sizing mistake on MLA models.

Two other things. Llama-3-70B and Qwen2.5-72B-Instruct cost identical bytes because $L$, $h_{kv}$ and $d_h$ match exactly — parameter count tells you nothing about cache cost. And Llama-3-8B lands on exactly 1 GiB at 8,192 tokens, a useful mental unit: at 8k, one gibibyte is one request.

The subtraction has four terms, and three of them are not exact, so every assumption is stated.

1. $C_{\text{total}} = 79.65$ GiB — an H100 SXM 80 GB reports 81,559 MiB to the driver. Conventional reported figure, not measured here; if your card differs, everything below scales.
2. $u = 0.92$, vLLM's gpu_memory_utilization default at this SHA (vllm/config/cache.py:L80), leaving 6.37 GiB untouched.
3. Weights = parameter count × bytes per weight, both derived from config.
4. Activations + CUDA graphs + non-torch = 6.0 GiB. The soft term. vLLM measures it with a profiling forward pass (§7); SGLang estimates it a priori and reserves 13.1 GiB at H100 defaults. 6.0 sits between the two; §2.6 owns it.

Because Llama-3-8B costs exactly 1 GiB per 8k request, the sensitivity reads straight off the page: every gibibyte of error in that allowance is one request.

Four things in that table are worth more than the table.

Llama-3-70B in bf16 does not fit on one H100 under this budget. With fp8 weights, the illustrative 1.57 GiB KV pool cannot admit an 8192-token request, but can hold shorter contexts. Zero in the last column means zero full 8k requests, not an impossibility of every single-GPU quantized serving configuration.

Tensor parallelism divides the KV cache too. Under TP=8 each rank owns one KV head, so the per-GPU cell size is $320/8 = 40$ KiB — the division is explicit in get_num_kv_heads, quoted in §7. Every GPU holds one eighth of every token, so 1.33M resident tokens is a cluster-wide figure and 162 concurrent 8k requests is what the node serves. §5.1 owns the mechanism.

MLA's cache is not divided by TP. With $h_{kv} = 1$ there is nothing to shard; every rank keeps the full latent. Hence 68.6 KiB per token per GPU and 52 concurrent 8k requests across a 16-GPU deployment — the same concurrency as Llama-3-8B on one card. Adding GPUs does not divide the latent cell size, but can free KV capacity by reducing per-rank weight memory. (Decode context parallelism is the escape hatch; both engines carry a dcp_size that shards the latent.)

Quantising the weights is a KV lever. Qwen3-32B goes from 3 concurrent requests to 18 by halving the weights alone. Where weights dominate the card, weight quantisation buys concurrency more cheaply than KV quantisation does — the reverse of the usual intuition. §2.5 covers the other half.

The batch enters the formula only through $\sum_i s_i$; nothing multiplies $B$ on its own. The consequence is the most useful fact in this chapter, and it is routinely got wrong.

All three saturate the same pool. A server that comfortably runs 1,600 concurrent 256-token requests falls over at 60 concurrent 8k requests, and no batch-size tuning changes that: the constraint is a token budget, not a request budget. This is why both engines' admission control is written against token counts, why --max-num-seqs is a weak knob beside the pool size, why the scheduler is best understood as a memory allocator (§1.4), and why prefix caching is a capacity feature rather than only a latency one — shared prefixes are shared tokens (§2.3, §2.4).

Both engines take the same three steps at startup — measure what is free, subtract what is spent, divide by a per-token constant — but factor them differently, and the difference shows up in what you can predict before launching.

vLLM: a per-layer page, then integer division

vLLM never writes the size formula as one expression. It writes the size of one page of one layer and lets the allocator multiply:

    @property
    def state_content_size_bytes(self) -> int:
        """Bytes per (head slot, stored state) cell of the page."""
        if self.state_content_bytes is not None:
            return self.state_content_bytes
        return (self.head_size + self.head_size_v) * get_dtype_size(self.dtype)

    @property
    def unpadded_page_size_bytes(self) -> int:
        return self.num_heads * self.storage_block_size * self.state_content_size_bytes

    @property
    def page_size_bytes(self) -> int:
        if self.page_size_padded is not None:
            assert self.page_size_padded >= self.unpadded_page_size_bytes
            return self.page_size_padded
        return self.unpadded_page_size_bytes

Map it onto the derivation. head_size + head_size_v is the factor of 2 — written as a sum rather than a product so MLA can set head_size_v = 0 and asymmetric-head models ($d_v \neq d_k$) work without a special case. num_heads is $h_{kv}$; storage_block_size is the block length in tokens, default 16 (vllm/config/cache.py:L59); $L$ is absent because this is one layer. For Llama-3-8B: $8 \times 16 \times 256 \times 2 = 65{,}536$ bytes per page.

The per-request ceiling multiplies by the block count for the longest allowed sequence:

    def max_memory_usage_bytes(self, vllm_config: VllmConfig) -> int:
        max_model_len = vllm_config.model_config.max_model_len
        dcp_world_size = vllm_config.parallel_config.decode_context_parallel_size
        if dcp_world_size > 1:
            max_model_len = cdiv(max_model_len, dcp_world_size)
        return cdiv(max_model_len, self.block_size) * self.page_size_bytes

And the pool-to-blocks conversion is one line — note the second division, which is where $L$ finally enters:

    num_blocks = int(available_memory // page_size // num_layers)
    num_blocks = max(num_blocks, 0)
    return may_override_num_blocks(vllm_config, num_blocks)

SGLang: one cell size, one division

SGLang collapses the same product into a single scalar it calls the cell size, computed once for the whole model:

        else:
            n = model_config.get_num_kv_heads(tp_size, dcp_size)
            cell_size = (
                n
                * (model_config.head_dim + model_config.v_head_dim)
                * effective_num_layers
                * kv_size
            )

That is $\;h_{kv} \cdot (d_k + d_v) \cdot L \cdot b\;$ — the chapter's formula verbatim, with the same sum-instead-of-2 trick. The KV-head count is already the per-rank count, which is the citation behind the TP claim in §4:

    def get_num_kv_heads(self, tensor_parallel_size: int, dcp_size: int = 1) -> int:
        """Number of KV heads per GPU.

        DCP ranks replicate KV, so heads shard across ``tp // dcp`` groups.
        Drafts never join the group and ignore ``dcp_size``. With fewer heads
        than groups, each GPU keeps one.
        """
        total_num_kv_heads = self.get_total_num_kv_heads()
        if self.is_draft_model:
            dcp_size = 1
        kv_tensor_parallel_size = tensor_parallel_size // dcp_size
        return max(1, total_num_kv_heads // kv_tensor_parallel_size)

The MLA branch a few lines earlier (pool_configurator.py:L257-L265) takes the latent dimension instead — calculate_mla_kv_cache_dim(...) * effective_num_layers * kv_size — and crucially does not divide by tp_size. That asymmetry is the code-level statement that MLA caches are replicated per rank. The latent width is the sum from §3:

    kv_lora_rank = model_config.kv_lora_rank
    qk_rope_head_dim = model_config.qk_rope_head_dim
    kv_cache_dim = kv_lora_rank + qk_rope_head_dim  # default mla kv cache dim

    # For non-DSA models, MLA kv cache dim is simply kv_lora_rank + qk_rope_head_dim
    if not is_dsa_model:
        return kv_cache_dim

The division is then trivial:

    def calculate_pool_sizes(
        self, available_bytes: int, page_size: int
    ) -> MemoryPoolConfig:
        max_total_num_tokens = (
            available_bytes // self._cell_size
            if self._cell_size
            else self._zero_kv_max_tokens
        )
        max_total_num_tokens = max_total_num_tokens // page_size * page_size
        return MemoryPoolConfig(max_total_num_tokens=max_total_num_tokens)

The two shapes are visible in the allocated tensors too — MHA takes two buffers per layer indexed by KV head, MLA one with a head axis pinned to 1:

        rows = self.size + self.page_size
        return (
            (rows, self.head_num, self.head_dim),
            (rows, self.head_num, self.v_head_dim),
        )
# ...
                self.kv_buffer = [
                    torch.zeros(
                        (self.size + self.page_size, 1, self.kv_cache_dim),
                        dtype=self.store_dtype,
                        device=self.device,
                    )
                    for _ in range(self.layer_num)
                ]

Where they differ, and why it matters

                # Constant meta data (e.g., from attention backend) + activation slack.
                reserved_mem = 512
                reserved_mem += activation_tokens * 1.5
                # Some adjustments for large parallel size
                reserved_mem += self.tp_size * self.pp_size / 8 * 1024
                reserved_mem += self.reserve_for_graph_mb()
                if gpu_mem is not None and gpu_mem > 60 * 1024:
                    reserved_mem = max(reserved_mem, 10 * 1024)
                # Reserve headroom for DeepEP all-to-all buffers on top of the floor.
                reserved_mem += self.reserve_for_deepep_a2a_mb()

            self.mem_fraction_static = (
                round((gpu_mem - reserved_mem) / gpu_mem, 3)
                if gpu_mem is not None
                else 0.88
            )

Work it for an H100 at defaults: chunked_prefill_size is 8192 in the 60–90 GB band (server_args.py:L4894-L4903), so activation_tokens = 8192, and reserve_for_graph_mb contributes decode max_bs * 2 = 512 MB at max_bs = 256 (server_args.py:L5124). Total $= 512 + 12{,}288 + 128 + 512 = 13{,}440$ MB, above the 10,240 MB floor, so $\text{mem\_fraction\_static} = \text{round}((81559 - 13440)/81559, 3) = 0.835$ — computable on paper before touching a GPU, which is the point.

What happens between vllm serve and the line that tells you your capacity.

1. Worker.init_device takes a MemorySnapshot and calls request_memory (vllm/v1/worker/utils.py:L444-L451), computing ceil(total_memory * gpu_memory_utilization) = $0.92 \times 79.65 = 73.28$ GiB and raising if that much is not already free.
2. Weights load: 14.96 GiB, recorded as model_memory_usage.
3. Worker.determine_available_memory (vllm/v1/worker/gpu_worker.py:L475) enters memory_profiling and calls model_runner.profile_run() — a real forward pass on a dummy batch of max_num_batched_tokens tokens, which is what makes the activation term measured rather than guessed — then profile_cudagraph_memory().
4. The subtraction:
   vllm/v1/worker/gpu_worker.py:L559-L562 vLLM

           self.available_kv_cache_memory_bytes = (
               self.requested_memory
               - profile_result.non_kv_cache_memory
               - cudagraph_memory_estimate_applied

   With our 6.0 GiB allowance for everything non-weight and non-KV, this is $73.28 - 14.96 - 6.0 = 52.32$ GiB.
5. Each Attention layer contributes a FullAttentionSpec (vllm/model_executor/layers/attention/attention.py:L648-L654) with num_kv_heads=8, head_size=128, head_size_v=128, dtype=bfloat16. All 32 are identical, so get_kv_cache_config_from_groups makes one group of group_size = 32.
6. get_num_blocks divides. Carry the pool at full precision — it is $73.27566 - 14.96 - 6.0 = 52.31566$ GiB, displayed as 52.32 above — because the floor divisions amplify a two-decimal rounding into a two-block error: $52.31566 \times 2^{30} \div 65{,}536 \div 32 = \mathbf{26{,}785}$ blocks, or $428{,}560$ tokens.
7. get_max_concurrency_for_kv_cache_config (vllm/v1/core/kv_cache_utils.py:L967-L989) divides that by the blocks one max-length request needs — $\lceil 8192/16 \rceil = 512$ — giving $52.31$. Then:
   vllm/v1/core/kv_cache_utils.py:L1925-L1931 vLLM

       logger.info_once(
           "GPU KV cache size: %s tokens, "
           "Maximum concurrency for %s tokens per request: %.2fx",
           f"{num_tokens:,}",
           f"{max_model_len:,}",
           max_concurrency,
       )

SGLang lands within 9 tokens of the same answer by a different route: _profile_available_bytes (python/sglang/srt/mem_cache/kv_cache_configurator.py:L1764-L1811) computes rest_memory = available_gpu_memory - slack_gb - mm_reservation_gb; calculate_pool_sizes divides by cell_size = 131,072 to get 428,569 tokens; MHATokenToKVPool._finalize_allocation_log (memory_pool.py:L1674-L1701) prints the K and V sizes in GB, and the scheduler prints max_total_num_tokens=... (python/sglang/srt/managers/scheduler.py:L1094-L1099). The 9-token gap is vLLM's 16-token block granularity against SGLang's default page size of 1.

Three ways to get $c$ wrong. Using $h$ instead of $h_{kv}$ over-estimates Llama-3-8B by 4× and an MQA model by $h$× — the tell is that your predicted pool is a clean multiple of the logged one. Deriving $d_h$ as $d/h$ under-counts Qwen3-32B by 37%. And applying the GQA formula to MLA is the 71× error from §4, which makes DeepSeek-V3 look unservable when it has the cheapest cache in the table.

Forgetting the block-granularity rounding. vLLM allocates whole blocks, so a 17-token request occupies 32 slots at block_size=16 — a real tax on short-request workloads, and the subject of §2.2.

Assuming max_model_len does not matter. It does not change bytes per token, but every capacity check is written against it (max_memory_usage_bytes above) and it is the denominator in the concurrency number: cut it from 128k to 8k and the reported concurrency rises 16× on the same pool. When the ValueError from §1 fires, vLLM binary-searches for a length that fits (estimate_max_model_len, kv_cache_utils.py:L821-L840) and puts that suggestion in the message.

Chasing OOM by lowering gpu_memory_utilization. It works, but read the causation: lowering it shrinks the KV pool, usually the thing you wanted. If the OOM is in the activation term — a large prefill chunk, a big captured graph — the fix is --max-num-batched-tokens or the graph batch list, not the utilisation. §2.6.

Compute $c$ from the config, subtract, divide — then check against what the engine says.

python3 - <<'PY'
import json, urllib.request
url = "https://huggingface.co/NousResearch/Meta-Llama-3-8B/raw/main/config.json"
c = json.load(urllib.request.urlopen(url))
L   = c["num_hidden_layers"]
hkv = c["num_key_value_heads"]
dh  = c.get("head_dim") or c["hidden_size"] // c["num_attention_heads"]
cell = 2 * L * hkv * dh * 2
print(f"L={L} h_kv={hkv} d_h={dh}  ->  {cell} B/token = {cell/1024:.1f} KiB")
pool_gib = 52.32
print(f"pool {pool_gib} GiB -> {int(pool_gib*2**30//cell):,} tokens")
PY

vllm serve meta-llama/Meta-Llama-3-8B-Instruct --max-model-len 8192 2>&1 \
  | grep -E "Available KV cache memory|GPU KV cache size|Maximum concurrency"

python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct 2>&1 \
  | grep -E "KV Cache is allocated|max_total_num_tokens|Memory pool end"

Three consistency checks. (a) vLLM's token count divided by max_model_len must equal its reported concurrency. (b) That token count times 131,072 must equal the "Available KV cache memory" line to within one block. (c) SGLang's max_total_num_tokens times 131,072 must equal its K size plus V size in the allocation log. Then flip one knob at a time and predict first: --kv-cache-dtype fp8 doubles the token count, --tensor-parallel-size 2 halves per-rank KV bytes while also freeing weight memory, so capacity can grow by more than two, --max-model-len 4096 leaves it alone and doubles the concurrency figure.

• The specs, in code. vllm/v1/kv_cache_interface.py in full — 1,007 lines in which every layout vLLM supports (full attention, sliding window, chunked local, sink, MLA, Mamba, cross-attention) states its own page_size_bytes and max_memory_usage_bytes. The fastest way to learn what "the KV cache" means across modern architectures.
• Config sources used here. config.json from NousResearch/Meta-Llama-3-8B and -70B (ungated mirrors of the Meta repos), Qwen/Qwen3-32B, Qwen/Qwen2.5-72B-Instruct, deepseek-ai/DeepSeek-V3 on the Hugging Face Hub, fetched August 2026.
• GQA. Ainslie et al., arXiv 2305.13245 — the paper that made $h_{kv}$ a free parameter and shrank every number here by 4× to 8×. Mechanism in §3.5.
• MLA. DeepSeek-V2 (arXiv 2405.04434) introduces the latent cache; the DeepSeek-V3 technical report (arXiv 2412.19437) carries the 671B/37B figures and the shapes used above. The absorbed-weight trick is §7.2.
• PagedAttention. Kwon et al., SOSP 2023 — how the pool this chapter sizes is actually carved up. §2.2. Symbols and the concurrency expression live in FORMULAS.