21 min read7 sections

Lab 04 · for chapter 02-03

Prefix cache hit rate

Run a shared-prefix workload against both engines and read the hit-rate metric; break the cache and watch it fall.

Download lab script · CPU reference checks · Environment preflight.

A prefix cache either fires or it does not, and the difference is the whole prefill bill. This lab makes both engines tell you their hit rate on a workload you designed, checks that number against arithmetic you did first, and then breaks the cache three different ways to prove you know what it is keyed on.

Hardware

Part A — predicting the hit rate from block size and prefix length — runs anywhere: laptop, no GPU, no network, stdlib only. Part B needs one GPU that can hold an 8B model with room for a few thousand cached tokens; a 24 GB card is comfortable. There is no substitute measurement for Part B, because the point of it is to catch your arithmetic being wrong about this engine at this block size. If you have no GPU, do Part A, read Part B, and treat the metric semantics below as the thing to learn.

Not executed here

run.py was written against the metric names, request schemas, and log lines cited below, and its argument handling and arithmetic were exercised, but it has not been run against a live engine — no GPU was available while writing. Every number the README shows is either cited to source or labelled as arithmetic. Nothing here was measured. Report anything that disagrees.

§1

What you measure

Four numbers, in order:

The hit rate you predict, from the workload shape and the match granularity, before starting anything.
The hit rate each engine reports on a warm shared-prefix workload.
The collapse, when you change one token near the front of the prefix.
The second collapse, when the prefix still matches but the blocks holding it have been evicted.

Steps 3 and 4 look identical from the outside — the hit rate goes to roughly zero — and have completely different fixes. Telling them apart from the metrics alone is the skill this lab is for.

The arithmetic

vLLM's counters have precise semantics, and they are not the ones most people assume. The KV cache manager records, per request, the whole prompt length as queries and the number of cached tokens found as hits:

vllm/v1/core/kv_cache_manager.py:L221-L231 vLLM

    def record_prefix_cache_stats(self, request: Request, num_hits: int) -> None:
        # Don't count a request that skipped the cache lookup.
        if not self.log_stats or not self.prefix_cache_lookup_enabled(request):
            return
        assert self.prefix_cache_stats is not None
        self.prefix_cache_stats.record(
            num_tokens=request.num_tokens,
            num_hits=num_hits,
            preempted=request.num_preemptions > 0,
        )

Two consequences fall straight out of that. First, the denominator is the full prompt, so a long distinct suffix drags the reported rate down even when the shared prefix hit perfectly. Second, a request that hits everything still cannot report 100%, because the engine deliberately holds one token back so there is something to compute logits from:

vllm/v1/core/kv_cache_manager.py:L256-L262 vLLM

        # NOTE: When all tokens hit the cache, we must recompute the last token
        # to obtain logits. Thus, set max_cache_hit_length to prompt_length - 1.
        # This can trigger recomputation of an entire block, rather than just
        # the single last token, because allocate_slots() requires
        # num_computed_tokens to be block-size aligned. Removing this limitation
        # could slightly improve performance in the future.
        max_cache_hit_length = request.num_tokens - 1

Third and largest: hits are found in whole blocks only. The block hasher walks the prompt in fixed-size steps and stops when a step would run off the end — a partial trailing block is never hashed, so it can never be matched:

vllm/v1/core/kv_cache_utils.py:L745-L752 vLLM

        new_block_hashes: list[BlockHash] = []
        while True:
            end_token_idx = start_token_idx + hash_block_size
            if end_token_idx > num_tokens:
                # We only hash full blocks
                break

            # MM and LoRA requests need extra keys for block-hash computation.

So for a workload of $N$ requests sharing a $P$-token prefix, each with a distinct $S$-token suffix, at match granularity $B$ tokens, with the first request arriving cold:

\text{hit rate} \;=\; \frac{(N-1)\,\cdot\, B\left\lfloor P/B \right\rfloor}{N\,(P+S)}

where $B$ is hash_block_size — for a single-KV-group model this is just --block-size, whose default is 16:

vllm/config/cache.py:L59 vLLM

    DEFAULT_BLOCK_SIZE: ClassVar[int] = 16

Work one case now, before you touch a GPU. Take $N=64$, $P=1000$, $S=200$, $B=16$. Then $\lfloor 1000/16\rfloor = 62$ full blocks, so 992 of the 1000 prefix tokens are matchable; eight are stranded in a partial block forever. Hit rate is $63 \times 992 / (64 \times 1200) = 62{,}496/76{,}800 = 81.4\%$. Not 100%, and not 83.3% either (the warm asymptote before rounding). The cold first request costs $100(1000)/(64\cdot1200)\approx1.302$ percentage points; rounding 1000 to 992 on the other 63 requests costs $100(63)(8)/(64\cdot1200)\approx0.656$ points. Their sum is the approximately 1.96-point gap between $P/(P+S)$ and the real answer combines cold-start and granularity penalties. Check both before attributing a discrepancy to the cache algorithm. Derived; arithmetic.

Granularity

The two engines do not agree on $B$. vLLM matches at block granularity, default 16. SGLang's radix tree matches at page granularity, and its default page size is 1 — token granularity — so the floor term vanishes and the same workload reports a strictly higher rate. That is a difference in accounting units, not in cache quality. See §2.4.

python/sglang/srt/arg_groups/overrides.py:L2378-L2380, L2394-L2397 SGLang

def _page_size_default(view: Any) -> dict:
    if view.page_size is not None:
        return {}
    # ... ROCm vectorized-5d branch elided ...
    if not is_musa():
        return {"page_size": 1}
    return {"page_size": 64}

§2

Where each engine keeps the number

Three surfaces, and they do not mean the same thing. Pick deliberately.

vLLM: two monotonic counters

vllm/v1/metrics/loggers.py:L584-L602 vLLM

        counter_prefix_cache_queries = self._counter_cls(
            name="vllm:prefix_cache_queries",
            documentation=(
                "Prefix cache queries, in terms of number of queried tokens."
            ),
            labelnames=labelnames,
        )
        self.counter_prefix_cache_queries = create_metric_per_engine(
            counter_prefix_cache_queries, per_engine_labelvalues
        )

        counter_prefix_cache_hits = self._counter_cls(
            name="vllm:prefix_cache_hits",
            documentation=("Prefix cache hits, in terms of number of cached tokens."),
            labelnames=labelnames,
        )
        self.counter_prefix_cache_hits = create_metric_per_engine(
            counter_prefix_cache_hits, per_engine_labelvalues
        )

These are the ones to use. They are cumulative, so a hit rate for an interval is a delta of both counters divided — which is exactly what you want when the interesting event is a change. Scrape them by prefix match, summing across label sets and tolerating the _total suffix the exposition format adds; vLLM's own integration test does precisely that:

tests/v1/kv_connector/nixl_integration/test_mamba_prefix_cache.py:L234-L247 vLLM

def get_metric(host: str, port: str, metric_name: str) -> float:
    """Scrape a single Prometheus metric from /metrics."""
    url = f"http://{host}:{port}/metrics"
    resp = requests.get(url, timeout=10)
    resp.raise_for_status()
    total = 0.0
    for line in resp.text.splitlines():
        if line.startswith("#"):
            continue
        if line.startswith(metric_name):
            match = re.search(r"[\d.eE+\-]+$", line)
            if match:
                total += float(match.group())
    return total

Prefix matching bites

The Prometheus client also publishes a <name>_created sample alongside every counter, and its value is a Unix timestamp. It shares the prefix you are matching on, so a naive startswith sum quietly returns about $1.7 \times 10^{9}$ instead of a token count. Drop any sample whose metric name ends in _created; run.py does.

vLLM: the log line, which is a sliding window

The number vLLM prints to stdout is not the counters' ratio. It comes from a sliding-window aggregate over recent requests:

vllm/v1/metrics/loggers.py:L287-L297 vLLM

        log_parts.extend(
            [
                "GPU KV cache usage: %.1f%%",
                "Prefix cache hit rate: %.1f%%",
            ]
        )
        log_args.extend(
            [
                self.last_scheduler_stats.kv_cache_usage * 100,
                self.prefix_caching_metrics.hit_rate * 100,
            ]

vllm/v1/metrics/stats.py:L42, L106-L111 vLLM

    def __init__(self, max_recent_requests: int = 1000) -> None:
# ... window bookkeeping elided ...
    @property
    def hit_rate(self) -> float:
        """Calculate the hit rate for the past N requests."""
        if self.aggregated_query_total == 0:
            return 0.0
        return self.aggregated_query_hit / self.aggregated_query_total

A 1000-request window means the collapse you are about to engineer will look gradual in the log and instant in the counters. Both are correct. Use the log for eyeballing and the counters for the measurement, and never quote the log number as the result of an experiment.

SGLang: a cumulative token counter that documents its own formula

python/sglang/srt/observability/metrics_collector.py:L892-L902 SGLang

        self.prefill_effective_tokens_total = Counter(
            name="sglang:prefill_effective_tokens_total",
            documentation=(
                "Effective prefill tokens with retracted-request re-counts "
                "excluded, updated on each log interval. mode: device_hit, "
                "host_hit, storage_hit, input. Windowed prefix cache hit "
                "rate = rate(sum of *_hit) / rate(sum of all modes); "
                "per-tier rate uses a single *_hit mode in the numerator."
            ),
            labelnames=list(labels.keys()) + ["mode"],
        )

This is SGLang's counterpart to vLLM's pair, and it is better instrumented: the mode label splits hits by tier, so you can see a device hit and a host (offloaded) hit separately. The plain sglang:cache_hit_rate gauge also exists, but it is a Gauge with multiprocess_mode="mostrecent" — it reports the last logged batch, not an interval, and one scrape of it can land on any batch at all. Do not build an experiment on it.

python/sglang/srt/observability/metrics_collector.py:L292-L297 SGLang

        self.cache_hit_rate = Gauge(
            name="sglang:cache_hit_rate",
            documentation="The prefix cache hit rate.",
            labelnames=labels.keys(),
            multiprocess_mode="mostrecent",
        )

Both SGLang metric surfaces need --enable-metrics; without it there is no /metrics route at all:

python/sglang/srt/entrypoints/http_server.py:L284-L286 and python/sglang/srt/utils/common.py:L2542-L2548 SGLang

    # Add prometheus middleware
    if server_args.enable_metrics:
        add_prometheus_middleware(app)
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    metrics_route = Mount("/metrics", make_asgi_app(registry=registry))

    # Workaround for 307 Redirect for /metrics
    metrics_route.path_regex = re.compile("^/metrics(?P<path>.*)$")
    app.routes.append(metrics_route)

SGLang: the scheduler log, which gives you the raw token counts

Even with no metrics enabled, every prefill batch prints its own numerator and denominator, and you can sum them yourself:

python/sglang/srt/managers/scheduler_components/metrics_reporter.py:L604-L612 SGLang

        msg = (
            f"Prefill batch{iter_msg}, "
            f"#new-seq: {prefill_stats.num_new_seqs}, "
            f"#new-token: {prefill_stats.log_input_tokens}, "
            f"#cached-token: {prefill_stats.log_hit_tokens}, "
            f"{token_usage_msg}"
            f"#running-req: {prefill_stats.num_running_reqs.total}, "
            f"#queue-req: {len(self.scheduler.waiting_queue)}, "
            f"#pending-token: {prefill_stats.num_pending_tokens}, "

Hit rate for a batch is #cached-token / (#new-token + #cached-token). Note the denominator: #new-token is the tokens that still needed computing, not the whole prompt, so the sum of the two is the prompt length and the ratio lands in the same place as vLLM's counters — from the opposite direction.

The one-token reservation is not a vLLM quirk: SGLang caps its match the same way, for the same reason, so the arithmetic above applies to both engines unchanged.

python/sglang/srt/managers/schedule_batch.py:L1420-L1425 SGLang

    def _compute_max_prefix_len(self, input_len: int) -> int:
        # NOTE: the matched length is at most 1 less than the input length to enable logprob computation
        max_prefix_len = input_len - 1
        if self.return_logprob and self.logprob_start_len >= 0:
            max_prefix_len = min(max_prefix_len, self.logprob_start_len)
        return max(max_prefix_len, 0)

§3

Running it

Part A, no GPU

shell — labs/04-prefix-cache-hit-rate shell

$ python3 run.py predict --prefix-len 1000 --suffix-len 200 --requests 64 --block-size 16
$ python3 run.py predict --prefix-len 1000 --suffix-len 200 --requests 64 --block-size 1
$ python3 run.py --help          # every knob, both subcommands

Do the second one by hand first. Setting --block-size 1 models SGLang's default page size, and the answer should be exactly $63 \times 1000/(64 \times 1200) = 82.0\%$ — 0.7 points above the vLLM figure, entirely from the eight tokens vLLM strands in a partial block.

Part B, one GPU

Start each engine with prefix caching explicitly on, metrics on, and the match granularity pinned so your prediction is about a known quantity. vLLM has prefix caching on by default at this SHA — pass the flag anyway, so the command records the intent:

vllm/config/cache.py:L107-L108 vLLM

    enable_prefix_caching: bool = True
    """Whether to enable prefix caching."""

shell — two servers, one at a time shell

$ VLLM_SERVER_DEV_MODE=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
    --enable-prefix-caching --block-size 16 --port 8000

$ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct \
    --enable-metrics --page-size 1 --port 30000

VLLM_SERVER_DEV_MODE=1 is what gives you a reset button. The cache-reset route is registered only under that environment variable, and the server warns loudly when it is on:

vllm/entrypoints/launchers/api_server/routers.py:L33-L37 vLLM

    if envs.VLLM_SERVER_DEV_MODE:
        from vllm.entrypoints.serve import register_vllm_dev_api_routers

        register_vllm_dev_api_routers(app)

vllm/entrypoints/serve/dev/cache/api_router.py:L20-L24, L32-L34 vLLM

@router.post("/reset_prefix_cache")
async def reset_prefix_cache(
    raw_request: Request,
    reset_running_requests: bool = Query(default=False),
    reset_external: bool = Query(default=False),
# ... docstring head elided ...
    Returns `{"success": bool}`. The reset fails (`success=false`) while
    blocks are still held, e.g. by running requests or in-flight async KV
    offload transfers; callers may retry.

SGLang's equivalent needs no environment variable, and refuses just as politely when the cache is in use:

python/sglang/srt/entrypoints/http_server.py:L966-L976 SGLang

@app.api_route("/flush_cache", methods=["GET", "POST"])
@auth_level(AuthLevel.ADMIN_OPTIONAL)
async def flush_cache(timeout: float = Query(0.0, ge=0.0)):
    """Flush the radix cache."""
    ret = await _global_state.tokenizer_manager.flush_cache(timeout_s=timeout)
    if ret.success:
        content = (
            "Cache flushed.\nPlease check backend logs for more details. "
            "(When there are running or waiting requests, the operation will not be performed.)\n"
        )
    else:

Then drive the workload. run.py sends raw token IDs rather than text, so "change one token at position 3" means exactly that and not "change some characters and hope the tokenizer cooperates". vLLM's completions endpoint takes a list of ints for prompt; SGLang's /generate takes input_ids:

vllm/entrypoints/openai/completion/protocol.py:L50-L56 and python/sglang/srt/managers/io_struct.py:L170-L177 vLLM

    prompt: (
        list[Annotated[int, Field(ge=0)]]
        | list[list[Annotated[int, Field(ge=0)]]]
        | str
        | list[str]
        | None
    ) = None
    text: Optional[Union[List[str], str]] = None
    # The token ids for text.
    # Use C-loop validator to replace Pydantic per-element type check for efficiency.
    input_ids: Annotated[
        Optional[Union[List[List[int]], List[int]]],
        PlainValidator(validate_optional_list_i64_1d_2d),
    ] = None

shell — the four phases shell

$ python3 run.py measure --engine vllm   --base-url http://127.0.0.1:8000 \
    --prefix-len 1000 --suffix-len 200 --requests 64 --block-size 16

$ python3 run.py measure --engine sglang --base-url http://127.0.0.1:30000 \
    --prefix-len 1000 --suffix-len 200 --requests 64 --block-size 1

# same, but flip the token at index 3 of the prefix on the break phase
$ python3 run.py measure --engine vllm --base-url http://127.0.0.1:8000 \
    --break-mode token --break-at 3

# same tokens, different salt (vLLM only)
$ python3 run.py measure --engine vllm --base-url http://127.0.0.1:8000 --break-mode salt

# same prefix, but push enough unique traffic through to evict it
$ python3 run.py measure --engine vllm --base-url http://127.0.0.1:8000 \
    --break-mode capacity --flood-requests 400 --flood-len 4096

§4

Three ways to break it, and why each is different

1. Change a token near the front

Block hashes are chained: each block's hash takes the previous block's hash as an input. So a changed token in block 0 changes block 0's hash, which changes block 1's, and so on to the end of the prompt. One token at index 3 invalidates every block:

vllm/v1/core/kv_cache_utils.py:L618-L622, L640-L645 vLLM

def hash_block_tokens(
    hash_function: Callable[[Any], bytes],
    parent_block_hash: BlockHash | None,
    curr_block_token_ids: Sequence[int],
    extra_keys: tuple[Any, ...] | None = None,
# ... docstring elided ...
        parent_block_hash = NONE_HASH

    curr_block_token_ids_tuple = tuple(curr_block_token_ids)
    return BlockHash(
        hash_function((parent_block_hash, curr_block_token_ids_tuple, extra_keys))
    )

The interesting version of this experiment is not index 3. It is sweeping --break-at across 0, 16, 400, 992, 999 and watching the hit rate fall in a staircase, one step per block boundary, not a smooth slope. A change at index 400 leaves blocks 0–24 intact (400/16 = 25 blocks below it) and kills the rest; each broken request still serves $25 \times 16 = 400$ tokens from cache, so it reports $400/1200 = 33.3\%$ rather than 0, and the workload aggregate lands at 32.8% once the cold request is folded in. Derived; arithmetic. That staircase is the signature of block-granular matching, and on SGLang at --page-size 1 it becomes a straight line instead.

2. Change the salt, not the tokens

vLLM lets a request carry a cache_salt, mixed into the hash of the first block only — and therefore, through the chain, into everything:

vllm/entrypoints/openai/chat_completion/protocol.py:L467-L478 vLLM

    cache_salt: str | None = Field(
        default=None,
        min_length=1,
        description=(
            "If specified, the prefix cache will be salted with the provided "
            "string to prevent an attacker to guess prompts in multi-user "
            "environments. The salt should be random, protected from "
            "access by 3rd parties, and long enough to be "
            "unpredictable (e.g., 43 characters base64-encoded, corresponding "
            "to 256 bit)."
        ),
    )

This is the cleanest possible break: byte-identical prompts, zero sharing. It is worth running because it is also the most common accidental cache killer in production — a per-tenant salt, correctly applied for isolation, costs you the entire cross-tenant hit rate, and the metrics look exactly like a code bug.

3. Keep the prefix, lose the blocks

Cached blocks live in the free-block queue and are recycled in LRU order. Allocating a new block evicts whatever cache entry the recycled block was holding, silently:

vllm/v1/core/block_pool.py:L658-L668 vLLM

        if num_blocks > self.get_num_free_blocks():
            raise ValueError(f"Cannot get {num_blocks} free blocks from the pool")

        ret: list[KVCacheBlock] = self.free_block_queue.popleft_n(num_blocks)

        # In order to only iterate the list once, we duplicated code a bit
        if self.enable_caching:
            for block in ret:
                self._maybe_evict_cached_block(block)
                assert block.ref_cnt == 0
                block.ref_cnt += 1

To make eviction happen on demand rather than by luck, shrink the pool. vLLM has an override that exists precisely for this kind of testing, and SGLang has a token-count equivalent:

vllm/config/cache.py:L101-L103 vLLM

    num_gpu_blocks_override: int | None = None
    """Number of GPU blocks to use. This overrides the profiled `num_gpu_blocks`
    if specified. Does nothing if `None`. Used for testing preemption."""

python/sglang/srt/server_args.py:L797-L808 SGLang

    max_total_tokens: A[
        Optional[int],
        Arg(
            help=(
                "The maximum number of tokens in the memory pool. If not "
                "specified, it will be automatically calculated based on the "
                "memory usage fraction. This option is typically used for "
                "development and debugging purposes."
                + f"\n\n{human_readable_int.__doc__}"
            ),
            type_parser=human_readable_int,
        ),

shell — a pool small enough to thrash shell

$ VLLM_SERVER_DEV_MODE=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
    --block-size 16 --num-gpu-blocks-override 2048     # 32,768 tokens of cache

$ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct \
    --enable-metrics --page-size 1 --max-total-tokens 32768

2048 blocks at 16 tokens is 32,768 tokens: room for about 27 copies of a 1200-token request. Send 400 distinct 4096-token prompts through that and the shared prefix is gone. The tell is that the hit rate recovers on the next pass with no code change — the prefix is still correct, the blocks were merely reused — whereas a fresh token prefix or salt on every request prevents warming. Repeating the same modified prefix or salt can warm that new key after the first request.

§5

What to expect

Predicted shape of each phase. Hit-rate cells are derived arithmetic for N=64, P=1000, S=200; the engine's own numbers go in your notes, not here.

Phase	vLLM, B=16	SGLang, page 1	Signature
Cold start	0%	0%	First request of the group; nothing to match.
Warm, shared prefix	81.4%	82.0%	The 0.7-point gap is block granularity, not engine quality.
Break at token 0	0%	0%	Permanent. Re-running does not recover it.
Break at token 400	32.8%	32.8%	Equal only because 400 is a multiple of 16. Sweep the index and vLLM becomes a staircase, SGLang a straight line.
Salt change	0%	—	vLLM only. Identical tokens, zero sharing.
Evicted	~0%	~0%	Recovers on the next pass without changing anything.

Things that will make your prediction wrong, roughly in order of how often they bite:

You sent text, not token IDs. A "1000-token prefix" that tokenizes to 997 or 1004 tokens moves the floor term and, worse, moves it differently per request. Send token IDs.
Requests overlapped. If the 64 requests are in flight concurrently, the first several all miss, because nothing has finished writing the prefix into the cache yet. The formula assumes one cold request. Run the first request alone, or accept a lower number and know why.
The block size is not what you think. Multi-KV-group models (sliding-window, Mamba hybrids) resolve hash_block_size to a GCD of group block sizes, or to the scheduler block size outright. Use a plain dense model for this lab.
You read the log line instead of the counters. The 1000-request window smears every transition you are trying to observe.
Prompt logprobs were on. A request asking for prompt logprobs skips the cache lookup entirely and is not counted at all — see prefix_cache_lookup_enabled in the first quote above.

To build the workload with the engines' own generators instead of run.py, both ship a shared-prefix dataset. vLLM's:

vllm/benchmarks/datasets/datasets.py:L1858-L1878 vLLM

    prefix_repetition_group.add_argument(
        "--prefix-repetition-prefix-len",
        type=int,
        default=256,
        help="Number of prefix tokens per request, used only for prefix "
        "repetition dataset.",
    )
    prefix_repetition_group.add_argument(
        "--prefix-repetition-suffix-len",
        type=int,
        default=256,
        help="Number of suffix tokens per request, used only for prefix "
        "repetition dataset. Total input length is prefix_len + suffix_len.",
    )
    prefix_repetition_group.add_argument(
        "--prefix-repetition-num-prefixes",
        type=int,
        default=10,
        help="Number of prefixes to generate, used only for prefix repetition "
        "dataset. Prompts per prefix is num_requests // num_prefixes.",
    )

and SGLang's:

python/sglang/benchmark/serving.py:L2610-L2628 SGLang

    group = parser.add_argument_group("generated-shared-prefix dataset arguments")
    group.add_argument(
        "--gsp-num-groups",
        type=int,
        default=64,
        help="Number of system prompt groups for generated-shared-prefix dataset",
    )
    group.add_argument(
        "--gsp-prompts-per-group",
        type=int,
        default=16,
        help="Number of prompts per system prompt group for generated-shared-prefix dataset",
    )
    group.add_argument(
        "--gsp-system-prompt-len",
        type=int,
        default=2048,
        help="Target length in tokens for system prompts in generated-shared-prefix dataset",
    )

shell — the engines' own shared-prefix workloads shell

$ vllm bench serve --dataset-name prefix_repetition --num-prompts 640 \
    --prefix-repetition-prefix-len 1000 --prefix-repetition-suffix-len 200 \
    --prefix-repetition-num-prefixes 10

$ python3 -m sglang.benchmark.serving --dataset-name generated-shared-prefix \
    --gsp-num-groups 10 --gsp-prompts-per-group 64 \
    --gsp-system-prompt-len 1000 --gsp-question-len 200

These are better than run.py for throughput numbers and worse for this lab, because they generate text and you lose exact control of token boundaries. Use them once you have finished arguing with the arithmetic.

§6

Exercises

Compute, by hand, the hit rate for $N=100$ requests sharing a 4096-token prefix with 32-token suffixes at $B=16$, then again at $B=1$. Which of the two differences — block granularity or the reserved last token — dominates here, and why is that the opposite of the $P=1000, S=200$ case?
Predict the hit rate when the shared prefix is 12 tokens long and the suffix is 500, at $B=16$. Then run it. Explain the result to someone who believes prefix caching "always helps".
Sweep --break-at over 0, 16, 32, 400, 992, 999 on vLLM and plot hit rate against break position. Then do the same on SGLang at --page-size 1, and again at --page-size 16. Which curve belongs to which engine, if you are shown only the plots?
Run the salt break, then POST /reset_prefix_cache, then re-run the warm phase. Now run the capacity break, reset, and re-run. One of the two recovers without the reset. Predict which before you run it, and say what that tells you about where the failure lives.
Read vllm/v1/metrics/stats.py and answer: after a POST /reset_prefix_cache, does the Prometheus counter vllm:prefix_cache_hits go back to zero? Does the logged hit rate? Find the code that makes them differ.

Answers

At $B=16$: $\lfloor 4096/16 \rfloor \times 16 = 4096$, all of it matchable, so $99 \times 4096 / (100 \times 4128) = 98.2\%$. At $B=1$: $99 \times 4096/(100 \times 4128)$ — identical, because 4096 is already a multiple of 16 and nothing is stranded. Here the two granularities agree exactly and the only shortfall is the 32-token suffix plus the cold first request. In the $P=1000$ case the prefix was not block-aligned, so granularity cost 8 tokens per request. The lesson is that the granularity penalty is $P \bmod B$ tokens, bounded by $B-1$ — it is invisible on long aligned prefixes and brutal on short ones. (Derived; arithmetic.)
$\lfloor 12/16 \rfloor = 0$. Zero full blocks, zero hits, forever, on every request. The hit rate is 0% no matter how many requests share that prefix. Prefix caching does nothing at all below one block, and a system prompt shorter than --block-size is pure overhead: you pay the hashing and lookup cost for a guaranteed miss.
vLLM is the staircase: hit rate is flat between block boundaries because breaking at index 17 and index 31 both invalidate from block 1 onward. SGLang at page size 1 is a straight line — every token position is its own match boundary. SGLang at --page-size 16 reproduces vLLM's staircase, which is the proof that the difference was units all along and not algorithm.
A capacity break can recover when retained prefixes are warmed again. A per-request rotating salt prevents reuse across requests; a fixed new salt merely creates a new namespace that can warm. Likewise, a repeated changed prefix can be cached. Record the exact token and salt sequence before diagnosing sustained near-zero hit rate.
The counter does not reset — it is a Prometheus Counter, monotonic by contract, and vLLM only ever calls .inc() on it. The logged rate does reset, because PrefixCacheStats carries a reset flag that the observing window checks and, when set, clears its aggregates before folding in the new sample. That is why a reset shows up as a discontinuity in the log and as a plateau (a period of zero slope) in the counters — and why you must diff counters over an interval rather than divide their lifetime totals.

§7

Key takeaways

vLLM's hit rate has a full prompt in the denominator and a block-floored, one-token-short prefix in the numerator. Predict with $\,(N-1)B\lfloor P/B\rfloor / (N(P+S))\,$, not with $P/(P+S)$.
A shared prefix shorter than one block is worth exactly zero. The penalty for misalignment is $P \bmod B$ tokens on every single request, forever.
The two engines' default match granularities differ by 16× — vLLM blocks of 16, SGLang pages of 1 — so a hit-rate comparison between them is meaningless until you pin both.
Use the cumulative counters for measurement and the log line for eyeballing. vLLM's logged rate is a 1000-request sliding window and will smear any transition you engineer.
Key breakage and capacity breakage produce the same metric and need opposite fixes. The discriminator is whether a second identical pass recovers.
Hashes are chained, so damage is prefix-monotone: one changed token at position $i$ costs you everything from block $\lfloor i/B \rfloor$ to the end, and nothing before it.