KV cache sizing calculator

Three things, in order:

1. Bytes per token, derived from the model config alone.
2. Token capacity for a given card and a given memory-utilisation fraction.
3. The residual — engine-reported capacity minus your prediction — and a named cause for it.

The third is the real exercise. A prediction that lands within a few percent teaches you nothing you did not already believe. A 13% gap that you can attribute to a padded page size or a reserved activation buffer teaches you how the allocator actually works.

The formula is FORMULAS' KV cell size, and the derivation is §2.1:

Part A needs only a config.json:

$ python3 run.py --config ./config.json --vram 80
$ python3 run.py --model meta-llama/Meta-Llama-3-8B-Instruct --vram 24 --dtype fp8
$ python3 run.py --help          # every knob, including --tp and --weights-gib

--util defaults to 0.92, matching vLLM's --gpu-memory-utilization at the pinned SHA (vllm/config/cache.py:L80). SGLang's analogue is --mem-fraction-static and does not hold the same value, so when you reconcile against SGLang, pass the number that engine actually used rather than assuming the default carries across — see §2.6.

Predict before you read the output. For Llama-3-8B in bf16 the four numbers you need are L=32, h_kv=8, d_h=128, b=2. Do that arithmetic in your head first; the script should agree exactly, because it is the same arithmetic.

Part B starts an engine and reads the line it prints. vLLM's is emitted once, from the KV cache capacity path:

    max_model_len = vllm_config.model_config.max_model_len
    logger.info_once(
        "GPU KV cache size: %s tokens, "
        "Maximum concurrency for %s tokens per request: %.2fx",
        f"{num_tokens:,}",
        f"{max_model_len:,}",
        max_concurrency,
    )

SGLang prints its equivalent from the scheduler, on rank 0 only — which matters, because under TP the other ranks stay silent and a grep without tp_rank == 0 in mind looks like a missing log:

        if self.ps.tp_rank == 0:
            logger.info(
                f"max_total_num_tokens={self.max_total_num_tokens}, "
                f"chunked_prefill_size={get_schedule().chunked_prefill_size}, "
                f"max_prefill_tokens={self.max_prefill_tokens}, "
                f"max_running_requests={self.max_running_requests}, "
                f"context_len={self.model_config.context_len}, "

$ vllm serve meta-llama/Meta-Llama-3-8B-Instruct 2>&1 | grep "GPU KV cache size"
$ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct 2>&1 \
    | grep max_total_num_tokens

$ python3 run.py --config ./config.json --vram 80 --reported <the number you just read>

Your prediction will be high. The engine has costs your arithmetic does not model, and they run in this rough order of size:

The padding one is worth reading in the source, because it is the only one that is invisible from outside — the allocator will happily hand out a page larger than the data in it:

    @property
    def unpadded_page_size_bytes(self) -> int:
        return self.num_heads * self.storage_block_size * self.state_content_size_bytes

    @property
    def page_size_bytes(self) -> int:
        if self.page_size_padded is not None:
            assert self.page_size_padded >= self.unpadded_page_size_bytes
            return self.page_size_padded
        return self.unpadded_page_size_bytes