Open problems

You are running prefill–decode disaggregation (§1.6) across 32 prefill replicas and 32 decode replicas. A new vLLM build is ready. You do what you do with every other stateless service: canary it onto four prefill nodes and watch. A decode worker pulling incompatible KV can fail the compatibility check. The exact failure scope depends on when it is detected; the later fallback path described below can recompute locally rather than kill the worker:

                # Check compatibility hash BEFORE decoding agent metadata
                assert self.compat_hash is not None
                if (
                    self.enforce_compat_hash
                    and handshake_payload.compatibility_hash != self.compat_hash
                ):
                    raise RuntimeError(
                        f"NIXL compatibility hash mismatch. "
                        f"Local: {self.compat_hash}, "
                        f"Remote: {handshake_payload.compatibility_hash}. "
                        f"Prefill and decode instances have incompatible "
                        f"configurations. This may be due to: different vLLM versions,"
                        f" models, dtypes, KV cache layouts, attention backends, etc. "
                        f"Both instances must use identical configurations."
                        f"Disable this check using "
                        f'--kv-transfer-config \'{{"kv_connector_extra_config": '
                        f'{{"enforce_handshake_compat": false}}}}\''
                    )

Nothing is wrong with the cache. Blocks intact, NIC fine, model byte-identical. The two halves of the fleet disagree about a version string, and the connector treats that as a correctness hazard — which, given it is about to interpret remote memory as a KV tensor, it is. Here is the hash:

    factors = {
        # Version compatibility
        "vllm_version": vllm_version,
        "nixl_connector_version": NIXL_CONNECTOR_VERSION,
        # Model architecture - affects KV cache shape
        "model": model_config.model,
        "dtype": str(model_config.dtype),
        "num_kv_heads": model_config.get_total_num_kv_heads(),
        "head_size": model_config.get_head_size(),
        "num_hidden_layers": model_config.get_total_num_hidden_layers(),
        # Attention backend and KV cache dtype affect memory layout
        "attn_backend_name": attn_backend_name,
        "cache_dtype": str(cache_config.cache_dtype),
        "cross_layers_blocks": cross_layers_blocks,
        "is_hma_enabled": is_hma_enabled,
        "speculative_config": _get_speculative_compatibility_factors(vllm_config),
        # push (WRITE) and pull (READ) connectors are protocol-incompatible
        "transfer_mode": transfer_mode,
    }

vllm_version participates in the shown compatibility hash, so arbitrary cross-version KV transfers may be rejected. That does not make rolling deployment impossible: keep old-version prefill/decode pools paired, canary a new compatible pair, route new requests by version, drain the old pair, then retire it. Do not disable checks to infer tensor compatibility. The cited runtime validation of TP size, block size and layout is another gate, not proof that all heterogeneous pairs interoperate.

That is the shape of every problem in this chapter. Not a missing feature — a boundary that the abstraction never modelled, discovered at the moment you cross it.

Both engines are extremely good at one thing: scheduling a fixed set of identical GPUs holding one model and one KV pool. Every abstraction is written against that world — the block pool assumes it owns all the blocks, the token budget assumes it knows the whole batch, get_device_capability() assumes every device answers the same. Each problem below is a boundary where that assumption is crossed, and the honest summary is that the engines have plumbing across every boundary and policy across none.

Three of the five boundaries reduce to one number. From FORMULAS:

with $L$ layers, $h_{kv}$ KV heads, $d_h$ head dimension, $b$ bytes per element, and the leading 2 for K and V. Llama-3-70B ($L=80$, $h_{kv}=8$, $d_h=128$) in bf16 gives $2 \times 80 \times 8 \times 128 \times 2 = 327{,}680$ bytes — 320 KiB per token, arithmetic, not a measurement — so an 8,192-token prompt carries 2.5 GiB across the P/D boundary. DeepSeek-V3's MLA cache (§7.2) stores 576 values per token per layer over 61 layers: $576 \times 2 \times 61 = 70{,}272$ bytes, 68.6 KiB per token, 4.7× less, 549 MiB for the same prompt.

Compare that against the work it replaces. FORMULAS gives $F_{\text{prefill}}(S) \approx 2PS + 2\,L\,h\,d_h\,S^{2}$; for Llama-3-70B at $S = 8192$ ($P = 7\times10^{10}$, $h = 64$) that is $1.15\times10^{15} + 8.8\times10^{13} \approx 1.24\times10^{15}$ FLOPs. At an assumed 40% of an H100's 989 TFLOP/s dense bf16, eight GPUs give 3.16 PFLOP/s — roughly 390 ms of prefill against 54 ms of transfer over one 400 Gb/s NIC.

Two things fall out. MLA is duplicated across TP ranks rather than sharded, so its transfer does not divide by rank count — the last row has no 8-NIC column. And the number that governs disaggregation is bytes per link, not bytes per node, which depends entirely on which local rank reads from which remote rank:

    if transfer_topology.is_mla or tp_size >= remote_tp_size:
        # D (local TP) > P (remote TP): multiple local ranks read different chunks from
        # *one* remote rank, corresponding to different kv heads.
        # For MLA, we only need one remote since cache is duplicated. When P TP=k*TP k,
        # this will spread mla ranks to read from remote k*tp_rank.
        attn_ranks = [tp_rank * remote_tp_size // tp_size]

With homogeneous TP the mapping is one-to-one and the 2.5 GiB spreads over eight links. With $D_{TP} > P_{TP}$ several decode ranks converge on one prefill rank and that link becomes the bottleneck. This is why "disaggregation costs 14% of prefill" and "1.7% of prefill" are both true of the same model — and why a published ratio requires matching topology and assumptions before transfer to your cluster.

The plumbing is mature and plural: 16 connectors registered in vllm/distributed/kv_transfer/kv_connector/factory.py:L152-L241 — NIXL push and pull, Mooncake, MoRI-IO, LMCache, HF3FS, plus a composing MultiConnector — and five transports in python/sglang/srt/disaggregation/utils.py:L589-L594. Transports are not the problem. Three other things are.

Connection setup is O(replicas), serialised, and timeout-bounded

A decode worker does not hold a connection pool to the fleet; it handshakes lazily, per remote engine, against the remote ranks its TP mapping says it will read from:

        # When target instance TP > local TP, we need to perform multiple
        # handshakes. Do it in a single background job for simplicity.
        # Regardless, only handshake with the remote TP rank(s) that current
        # local rank will read from. Note that With homogeneous TP,
        # this happens to be the same single rank_i.
        assert self.transfer_topo is not None
        p_remote_ranks = self.transfer_topo.handshake_target_ranks(remote_tp_size)

All of that runs on one thread:

        # Background thread for initializing new NIXL handshakes.
        self._handshake_initiation_executor = ThreadPoolExecutor(
            # NIXL is not guaranteed to be thread-safe, limit 1 worker.
            max_workers=1,
            thread_name_prefix="vllm-nixl-handshake-initiator",
        )

Each handshake is a ZMQ request with a five-second receive timeout (vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:L657-L658), issued serially from a single-worker executor. With 32 prefill engines that is 32 sequential round trips before a decode rank is warm against the fleet, and one unreachable engine costs five seconds of that thread. Correct for a handful of peers; no story for a thousand. Engineering work remains — the handshake is stateless metadata exchange, so it parallelises trivially once the NIXL thread-safety constraint in that comment is worked around.

Failure semantics diverge, and neither is retry

vLLM makes it a policy, defaulting to recompute:

            kv_load_failure_policy = kv_transfer_config.kv_load_failure_policy
            self.recompute_kv_load_failures = kv_load_failure_policy == "recompute"

The scheduler then walks the affected requests, truncates num_computed_tokens back to the last valid block, and reschedules — with a hole in the middle of it:

            marked_invalid_block = False
            req_id = request.request_id
            # TODO (davidb): add support for hybrid memory allocator
            (req_block_ids,) = self.kv_cache_manager.get_block_ids(req_id)
            # We iterate only over blocks that may contain externally computed

So vLLM's fallback for a broken transfer is to prefill locally on the decode node — which works, and is exactly what disaggregation existed to avoid. SGLang has no such fallback; a failed handshake or transfer aborts the request:

            elif poll == KVPoll.Failed:
                error_message = f"Decode handshake failed for request rank={self.tp_rank} {decode_req.req.rid=} {decode_req.req.bootstrap_room=}"
                is_propagated = False
                try:
                    decode_req.kv_receiver.failure_exception()
                except Exception as e:
                    error_message += f" with exception {e}"
                    is_propagated = getattr(e, "is_from_another_rank", False)
                # Mute error message for propagated exceptions to avoid duplicate logging
                if is_propagated:
                    logger.debug(error_message)
                else:
                    logger.error(error_message)
                prepare_abort(
                    decode_req.req,
                    error_message,
                    status_code=HTTPStatus.INTERNAL_SERVER_ERROR,

and the prefill side does the same for an in-flight transfer that dies:

        req.time_stats.trace_ctx.abort(abort_info={"reason": error_message})
        release_kv_cache(req, self.tree_cache)  # unlock the tree
        if not isinstance(req.finished_reason, FINISH_ABORT):
            prepare_abort(
                req, error_message, status_code=HTTPStatus.INTERNAL_SERVER_ERROR
            )

The client gets HTTP 500. Neither engine retries against a different prefill replica, and both could: the decode node holds the prompt token ids, so re-issuing the prefill elsewhere is a routing decision, not state recovery. Engineering work remains — a router feature (§9.4) plus an engine hook that reports "transfer failed, prompt still valid" instead of a 500.

Nobody schedules across the split

The decode side is the real admission controller. It must reserve KV for a request before the prefill side may send anything, and that budget is computed entirely locally:

            available_size = self.token_to_kv_pool_allocator.available_size()
            # Include evictable decode-radix cache entries in the budget -- they
            # can be freed on demand before allocation.
            if self.scheduler.server_args.disaggregation_decode_enable_radix_cache:
                available_size += self._radix_full_evictable()
        allocatable_tokens = available_size - max(
            reserved_tokens, need_space_for_single_req
        )

The prefill scheduler cannot see this number. It enqueues completed prefills and discovers back-pressure only when transfers stop draining. Between the two sits a router choosing both replicas independently.

Research and engineering remain. Joint P/D scheduling couples two resource pools, variable transfer time and paired TTFT/TPOT targets. This engine snapshot does not establish a universal optimal policy, but absence of a local implementation is not absence of useful research. DistServe co-optimizes phase resources and parallelism under latency targets; Splitwise studies phase-specific placement including heterogeneous clusters. Their assumptions and evaluated workloads define the scope, not a solution to every dynamic fleet objective.

Once a prefix cache outlives one process it is a distributed store, and SGLang's HiCache tier (§2.6) has ten backends behind one interface. Look at the whole write side of that interface:

    def exists(self, key: str) -> bool:
        """
        Check if the key exists in the storage.
        Returns True if the key exists, False otherwise.
        """
        pass

    # TODO: Use a finer-grained return type (e.g., List[bool])
    def batch_exists(
        self, keys: List[str], extra_info: Optional[HiCacheStorageExtraInfo] = None
    ) -> int:
        """
        Check if the keys exist in the storage.
        return the number of consecutive existing keys from the start.
        Can be overridden by subclasses for more efficient implementation.
        """
        for i in range(len(keys)):
            if not self.exists(keys[i]):
                return i
        return len(keys)

    def clear(self) -> None:
        pass

The minimal store API omits explicit per-key deletion, version negotiation and expiry. Content addressing makes reuse safe only when keys capture every semantic determinant and stored bytes are complete and valid. A miss can trigger recomputation; an incorrectly namespaced, torn or corrupted hit can return wrong output. Consequently legal misses alone do not prove that the system is purely AP under CAP, and the source excerpt does not establish that all required metadata is hashed.

    def contains(self, key: str) -> bool:
        with self.lock:
            if key not in self.cache:
                return False
            if self.ttl_seconds == -1.0:
                return True
            if time.monotonic() - self.cache[key] > self.ttl_seconds:
                del self.cache[key]
                return False
            return True

If the page was evicted after the local cache said contains, the fetch fails and the tokens get recomputed. Staleness is a hit-rate bug, not a correctness bug — which is why a zoo of ten backends behind one interface is even possible.

Where it stops working: three things

The keyspace is partitioned by configuration, not just by content. The file backend suffixes every key:

        self.config_suffix = f"_{model_name}"
        if not is_mla_model:
            self.config_suffix += f"_{tp_rank}_{tp_size}"
        if enable_pp:
            self.config_suffix += f"_{pp_size}_{pp_rank}"
        # Under NSA context parallel each CP rank holds a disjoint slice of every
        # page, so give each rank its own file key to avoid a cross-rank write race.
        if attn_cp_size > 1:
            self.config_suffix += f"_cp{attn_cp_rank}_{attn_cp_size}"

A TP8 replica and a TP4 replica of the same model share nothing, because their pages have different shapes. MLA models skip the TP suffix — their cache is replicated per rank, so any rank's page is any rank's page, a second and quieter argument for MLA in a shared-cache fleet. Adding replicas of a different shape does not grow the cache; it forks it.

Cross-process reproducibility is opt-in and easy to lose. vLLM's block hash chains each block to its parent, so an entire prefix collapses to one value — and the chain's root has to agree across processes:

def init_none_hash(hash_fn: Callable[[Any], bytes]):
    global NONE_HASH, _NONE_HASH_SEED

    _NONE_HASH_SEED = resolve_none_hash_seed(hash_fn)
    if hash_fn in _NON_CRYPTO_HASH_FUNCTIONS and os.getenv("PYTHONHASHSEED") is None:
        logger.warning(
            "Using a random per-process NONE_HASH seed because %s is not "
            "collision resistant. Block hashes are therefore not reproducible "
            "across processes; set PYTHONHASHSEED to a shared value to reuse "
            "the prefix cache across instances, or use sha256.",
            hash_fn.__name__,
        )
    NONE_HASH = BlockHash(hash_fn(_NONE_HASH_SEED))

The default is safe (sha256), and the docstring is unusually candid about what the fast option costs:

    prefix_caching_hash_algo: PrefixCachingHashAlgo = "sha256"
    """Set the hash algorithm for prefix caching:

    - "sha256" uses Pickle for object serialization before hashing. This is the current
      default, as SHA256 is the most secure choice to avoid potential hash collisions.
    - "sha256_cbor" provides a reproducible, cross-language compatible hash. It
      serializes objects using canonical CBOR and hashes them with SHA-256.
    - "xxhash" uses Pickle serialization with xxHash (128-bit) for faster,
      non-cryptographic hashing. Requires the optional ``xxhash`` package.
      IMPORTANT: Use of a hashing algorithm that is not considered  cryptographically
      secure theoretically increases the risk of hash collisions, which can cause
      undefined behavior or even leak private information in multi-tenant environments.

Choose xxhash for the throughput and you have traded a shared cross-instance keyspace and a collision bound for some scheduler CPU. Fine in a single-replica server; in a fleet with a shared L3 it is the whole feature.

Eviction has no global view. The file backend's evictor is a per-process LRU over a directory:

``HiCacheFile`` is a thin raw-bytes store: it suffixes keys, reads/writes
``.bin`` pages, and answers existence queries. Everything that bounds how much
disk those pages consume -- the LRU recency index, per-file size accounting,
free-space probing, scanning pre-existing files on startup, and unlinking
victims -- lives here so the backend stays a plain key/value store.

A backend constructs one evictor and drives it through a small lifecycle::

    touch(key, path)                 # read hit / already-on-disk: bump recency
    reserve(key, n_bytes) -> bool    # admit a new write, evicting if needed
        commit(key)                  #   write landed on disk
        abort(key)                   #   write failed; release the reservation
    clear()                          # backend wiped all files

When eviction is not configured the evictor is inert: ``reserve`` always admits
and the other calls are no-ops, so the backend behaves as unbounded storage.

Point two nodes at one shared mount and you have two independent LRUs, each with its own byte accounting, each unlinking the other's hot pages. And read the last sentence: the default is no bound at all.

This remains a workload-dependent policy research problem. A globally optimal policy would rank pages by expected recompute cost saved per byte held, and recompute cost is not uniform: under $F_{\text{prefill}}(S)$ a cached 1M-token prefix is worth four orders of magnitude more than a 1k-token one, so byte-uniform LRU is provably the wrong shape. Worse, the readers are chosen by a cache-aware router whose placement decisions keep changing the reuse distribution the evictor would have to estimate. Optimising placement and eviction against one objective, online, with no oracle for future reuse, is an open problem — not an unwritten patch.

Two different problems wear the same name. Across vendors, the question is how much code a fast path costs. Within one deployment, it is whether the engine can even represent a machine whose GPUs differ. The answer to the second is no.

The portability tax, counted

vLLM's registry declares 37 attention backends, plus 5 more for Mamba-family and linear-attention layers and a CUSTOM placeholder (vllm/v1/attention/backends/registry.py:L44-L131 and vllm/v1/attention/backends/registry.py:L186-L193); SGLang registers 22 (python/sglang/srt/layers/attention/attention_registry.py:L41-L493). These are not 37 algorithms — they are roughly four, crossed with vendor, generation, sparsity and MLA-or-not: FLASH_ATTN, ROCM_AITER_FA, CPU_ATTN, AMX_MLA, XPU_MLA_SPARSE, FLASHMLA_SPARSE_DSV4, ROCM_FLASHMLA_SPARSE_DSV4. Every new accelerator multiplies rather than adds.

Per-platform coverage is then a list of subtractions. On CPU, sparse attention simply does not exist:

        if attn_selector_config.use_sparse:
            raise NotImplementedError("Sparse Attention is not supported on CPU.")

and features that assume a GPU execution model are switched off on the way past:

        if parallel_config.worker_cls == "auto":
            parallel_config.worker_cls = "vllm.v1.worker.cpu_worker.CPUWorker"
        # Disable DBO
        if parallel_config.enable_dbo:
            logger.warning_once("Dual-Batch Overlap is not supported on CPU, disabled.")
            parallel_config.enable_dbo = False

A portable fast path is hard because "the kernel" is not the unit. The unit is a co-designed tuple — memory layout, page size, dtype, tile shape, and what the scheduler may assume. The CPU platform rewrites cache_config.block_size to 16 to fit its MLA kernel (vllm/platforms/cpu.py:L166-L175), and that block size is then part of what the NIXL connector validates between peers. Portability wants the layout to be a free variable at the top of the stack; it is a constant baked in near the bottom.

Mixed silicon in one deployment is not modelled

Capability is a process-level property with device 0 as the default in both engines:

    @classmethod
    def get_device_capability(
        cls,
        device_id: int = 0,
    ) -> DeviceCapability | None:
        """Stateless version of [torch.cuda.get_device_capability][].

        Args:
            device_id: Device index in the visible device namespace, matching
                the argument accepted by torch.cuda.
        """
        return None

def get_device_capability(device_id: int = 0) -> Tuple[int, int]:
    major, minor = None, None
    if (hasattr(torch, "cuda") and torch.cuda.is_available()) or is_musa():
        major, minor = torch.cuda.get_device_capability(device_id)

A no-argument device-capability query is a verification lead, not proof of undefined behavior. In a process with one correctly restricted visible GPU, local device zero can identify the intended rank device. In a multi-visible-device process, inspect binding and the helper's actual argument semantics. Validate every participating rank's hardware against the chosen kernels and collective contracts before declaring support or a bug.

Engineering work remains wherever device binding, backend eligibility or tensor-transfer contracts are incomplete. Separate mixed GPU generations within one collective group from independent heterogeneous replicas behind a router. Splitwise already proposes phase-split homogeneous and heterogeneous clusters; it is incorrect to say nobody has proposed such a fleet. Serving one logical API does not require every pair to transfer raw KV: compatible subpools, explicit conversion or recomputation are alternatives.

Long context is three costs with three different exponents, and only one of them has an answer in the trees. Extrapolate the Llama-3-70B shapes to $S = 10^6$ as hypothetical arithmetic, not a supported checkpoint launch:

The KV term is linear and brutal: 320 KiB/token × $10^6$ = 305 GiB — 3.8× the 79.65 GiB an “80 GB” H100 SXM actually has, before a byte of the model is loaded. On the §2.1 budget (0.92 utilisation, less weights, less a 6.0 GiB allowance) 70B of bf16 weights leaves 34.7 GiB of KV per GPU at TP4 — 139 GiB across the node, short by half — and 51.0 GiB at TP8, or 408 GiB across the node, so one request would hold three quarters of an eight-card node. The prefill term is quadratic and worse: $2\,L\,h\,d_h\,S^2 = 1.31\times10^{18}$ FLOPs against $2PS = 1.4\times10^{17}$ for the projections. At the same 3.16 PFLOP/s assumption as §3, that is about 460 seconds of TTFT, 90% of it attention. A real launch may reject this context length or exceed model, positional, memory or kernel limits. The arithmetic neither validates a million-token checkpoint nor establishes its quality.

The one real answer in the trees is architectural sparsity. SGLang picks the dense/sparse crossover straight out of the checkpoint:

                    # When threshold is not manually set, set it to the index topk of model
                    from sglang.srt.configs.model_config import get_dsa_index_topk

                    envs.SGLANG_DSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD.set(
                        get_dsa_index_topk(hf_config)
                    )
                    logger.warning(
                        f"Set dense attention kv len threshold to model index_topk={envs.SGLANG_DSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD.get()} for DeepSeek with DSA."
                    )

def get_dsa_index_topk(config: PretrainedConfig) -> int:
    assert is_deepseek_dsa(config)
    return config.index_topk

Past index_topk cached tokens attention reads only the selected top-k, turning $S^2$ into $S \cdot \text{topk}$ — at $\text{topk} = 2048$, $S = 10^6$, a 488× cut in the attention term before the indexer's own cost. But look at the gate: is_deepseek_dsa(hf_config). This is not a capability you enable; it is a property of models trained sparse. DeepSeek 3.2, GLM-5, LongCat and MiniMax-M3 have entries in vLLM's backend enum; the cited Llama-3-70B checkpoint does not advertise that sparse-attention architecture. Today's answer to ultra-long context is "use a different model".

The prefix-cache economics also invert. vLLM's hash chain is inherently serial — each block's hash is an input to the next:

            # Compute the hash of the current block
            block_tokens = request.all_token_ids[start_token_idx:end_token_idx]
            block_hash = hash_block_tokens(
                caching_hash_fn, prev_block_hash_value, block_tokens, extra_keys
            )

            new_block_hashes.append(block_hash)
            start_token_idx += hash_block_size
            prev_block_hash_value = block_hash

At block size 16 a 1M-token request is 62,500 sequential SHA-256 calls on the scheduler's thread before the first lookup — while the payoff has never been larger, since a full prefix hit saves those 460 seconds. Long context makes prefix caching both far more valuable and measurably more expensive in the same request, and no scheduler in either tree accounts for the second half.

What would have to be true. Engineering work remains for the hashing (chunk it, move it off the hot path, or hash coarser and refine) and for KV compression policy. No one knows how for the general case: a training-free sparsifier holding accuracy at 1M tokens on an arbitrary dense model. Its absence is why every sparse backend in both trees is named after a model family.

At batch 1 every premise of a server engine reverses. Continuous batching has nothing to batch, and the decode GEMV's arithmetic intensity collapses to $2/b$ (FORMULAS) — 1.0 FLOP/byte at $b=2$, hopelessly memory-bound, and no scheduling policy can change it. Scheduler, prefix cache, router, KV connector: all exist to amortise costs across concurrent requests that do not exist.

Neither tree targets this. The only edge-shaped code in either is a memory-probe correction for unified-memory devices:

    @classmethod
    def is_integrated_gpu(cls, device_id: int = 0) -> bool:
        """
        Returns whether the GPU is an integrated (UMA) device that shares
        system memory with the CPU.

        On UMA systems (e.g. NVIDIA GH200, DGX Spark, Jetson Orin),
        cudaMemGetInfo may underreport free memory because it does not
        account for reclaimable OS memory (page cache, buffers).
        """
        return False

and its consumer in the profiler, which swaps in a host-memory reading when the device is UMA (vllm/utils/mem_utils.py:L148-L156). That is genuinely useful — it makes the VRAM budget of §2.6 correct on a Jetson Orin instead of catastrophically wrong — and it is the entire extent of edge awareness in either project.

What would have to be true. Almost none of this is research. A batch-1 engine is a different program with a different bill of materials, and that program exists — it is called llama.cpp, which §13.1 includes for exactly this reason. The one genuinely open question is the quantization floor. Sub-4-bit weight schemes are already in the tree (vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_wNa16.py:L39 maps 2 bits to uint2b2), so kernels are not the gap. What is missing is any comparable published statement of what 2-bit costs on which model at which size — the subject of the next section.

Determinism versus performance is covered in §10.4; what is open is what happens across the boundaries above. Batch-invariant kernels make one replica reproducible. They say nothing about a request whose KV came from a different replica running a different attention backend — and the compat hash of §1 lists attn_backend_name and cache_dtype among the factors that must match, the maintainers agreeing in code that cross-backend KV is not interchangeable. Fleet-level determinism is strictly harder than replica-level, and nothing in either tree attempts it.

Nobody agrees how to measure goodput. vLLM computes it, per request, against operator-supplied SLOs:

        for req_metric in zip(*valid_metrics):
            is_good_req = all([s >= r for s, r in zip(slo_values, req_metric)])
            if is_good_req:
                good_completed += 1

Two problems. It is opt-in — with no --goodput flag you get throughput at an unstated latency — and the TPOT it tests is the request's mean inter-token time (vllm/benchmarks/serve.py:L607-L613), so a request that stalls two seconds once and is fast otherwise passes. SGLang's harness has no goodput at all: grep -r goodput python/ returns nothing at 7d89325. Two of the most-benchmarked serving systems in the world do not share a definition of "served acceptably", and every head-to-head number you have read inherits that. §10.3 is the discipline; this is why no tool enforces it.

Every optimisation's accuracy cost is under-reported, and you can see the shape of it in the trees. This is the accuracy gate for NIXL-based disaggregation:

TASK = "gsm8k"
FILTER = "exact_match,strict-match"
# TODO(#43186): Widened from 0.03 to absorb chunk_scan/SSU numeric jitter
# on granite-4.0-h-tiny under NIXL PD; tighten when the kernel divergence
# is fixed.
RTOL = 0.05

One task. One filter. A one-sided threshold, widened from three points to five to absorb kernel divergence, with a TODO to tighten it when someone fixes the divergence. That is not negligence — it is a CI budget meeting a hard problem — but it means "P/D disaggregation is lossless" rests on gsm8k exact-match staying within five absolute points on a handful of small models. Elsewhere the question is simply handed to the operator:

            elif self.mamba_ssm_dtype != "float32":
                logger.warning(
                    "--enable-linear-replayssm-spec with --mamba-ssm-dtype=%s: the "
                    "closed-loop fold re-quantizes the committed state each "
                    "commit/flush (fp32 keeps it bit-exact to the fp32 recurrent "
                    "baseline), so it may drift over long sequences. Validate "
                    "accuracy for your model.",
                    self.mamba_ssm_dtype,
                )

"Validate accuracy for your model" is the honest answer and an admission that the engine cannot. Engineering work remains, unglamorous: a harness reporting (throughput, SLO attainment, accuracy delta) as one triple on a workload with a stated prefix-sharing rate and length distribution. Nobody has built it because it is expensive to run and unflattering to everyone.

Follow the request from §1 through vLLM's code, function by function, at a556f3f.

1. The router picks prefill replica P7 (a canary, newer build) and decode replica D3. It has no way to know they are incompatible.
2. P7 prefills normally. NixlPullScheduler.request_finished (vllm/distributed/kv_transfer/kv_connector/v1/nixl/pull_scheduler.py:L237-L281) returns delay_free_blocks=True and a kv_transfer_params dict carrying remote_engine_id, remote_host and remote_port, and records a wall-clock expiry for the request built from the connector's KV lease duration. The blocks are now pinned on a timer.
3. On D3, NixlPullScheduler.get_num_new_matched_tokens sees do_remote_prefill, the request enters WAITING_FOR_REMOTE_KVS, and _nixl_handshake is submitted to the single-slot executor of block AA.
4. The handshake sends GET_META_MSG, gets a NixlHandshakePayload back, and reaches the comparison in block A. Hashes differ. RuntimeError, on a background thread.
5. The request id lands in _failed_recv_reqs (vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:L490-L493), the queue whose comment reads "requests that skipped transfer (handshake or transfer failures)". Its blocks are reported through get_block_ids_with_load_errors.
6. The scheduler calls _handle_invalid_blocks, which calls _update_requests_with_invalid_blocks (block F). Because recompute_kv_load_failures is true by default (block E), num_computed_tokens is truncated to the last valid block and the request is rescheduled — as a local prefill on the decode node.
7. The user gets a correct answer, late. D3 just did the prefill a dedicated prefill fleet exists to do, and on P7 nothing signalled the failure: the pinned blocks are released by the lease timer expiring, not by anyone telling it.

Every step is defensible in isolation. The composite — a silent conversion of a disaggregated deployment into a colocated one, visible only as a TTFT regression — is what "plumbing but no policy" means in practice.

The compat-hash escape hatch is a footgun with a friendly error message. The RuntimeError above ends by telling you to set enforce_handshake_compat: false. Do that between genuinely incompatible builds and you have swapped a startup failure for reinterpreted memory. You want it scoped to the one factor you know is benign; the code offers no such granularity.

A shared L3 that silently forked. The symptom is a cache hit rate that halves when you scale out, with no errors anywhere. The cause is block K: you added replicas at a different TP size, so their keys carry a different config_suffix, so they populate a disjoint namespace in the same directory. Diagnose it by listing key suffixes in the store, not by reading hit-rate metrics.

Two nodes, one mount, mutual eviction. Two SGLang servers with --hicache-storage-backend file pointed at the same shared filesystem each run the LRU of block N over the whole directory. Each treats the other's files as its own eviction candidates. The signature is disk usage oscillating around the cap while both nodes report poor hit rates.

Three experiments, each measuring a boundary rather than a mechanism. The first needs no GPU.

V=~/Documents/other_git_repos/vllm
S=~/Documents/other_git_repos/sglang

# 1. How many attention backends is a fast path worth?
grep -cE '^    [A-Z][A-Z0-9_]* = ' $V/vllm/v1/attention/backends/registry.py
grep -c '@register_attention_backend' $S/python/sglang/srt/layers/attention/attention_registry.py

# 2. How often does the engine ask about a specific device?
grep -rn 'has_device_capability(' $V/vllm | wc -l
grep -rn 'has_device_capability(.*device_id' $V/vllm | wc -l

# 3. Does the benchmark harness know what a good request is?
grep -rn goodput $S/python | wc -l

Then reproduce the keyspace fork on one machine: start two SGLang servers with --hicache-storage-backend file, the same storage directory and different --tp-size, send both the same prompt, and list the files. Identical token content, two objects with different suffixes — block K, on your disk. Repeat with an MLA model and you get one.

For the measurement problem, run one workload twice through vllm bench serve, bare and with --goodput ttft:2000 tpot:50, and compare headline throughput to goodput. That gap is what every published comparison omits; §10.3 has the discipline and LABS the runnable version.

Identity. A reusable entry needs the model and weight revision, adapter state, exact token/position/mask semantics and relevant multimodal inputs; representation compatibility additionally needs dtype, layout, quantization scales and shard mapping. Tokenizer revision belongs in input provenance even when exact IDs are the key. Tenant authorization is separate from avoiding hash collisions.

Publication. Publish a key only after a complete payload and metadata commit; verify a checksum and expected dimensions before making its slots visible to attention. A failed existence lookup can be a miss, but an unchecked partial hit cannot. Use leases or references for in-flight readers and defer reclamation until they finish.

Worked rollout. Suppose old pools P0/D0 serve revision A and new P1/D1 serve B. Route a canary request only within P1/D1, bind its request ID and retries to B, and compare quality plus latency before increasing its traffic share. Stop admissions to P0, drain its transfers and D0 generations, then reclaim old registrations and cache namespaces. On a transfer timeout, cancel the abandoned leg, release its leases, and perform at most a bounded same-version retry or local recompute. A version mismatch must never be relabeled as a successful cache hit.

This is a proposed correctness protocol, not an executed implementation of either pinned runtime. Its negative tests are wrong-version keys, truncated payloads, duplicate completion, expired leases and failure between transfer and acknowledgement.

• DistServe (Zhong et al., OSDI 2024) and Splitwise (Patel et al., ISCA 2024) — the papers that established the P/D split and most of the arithmetic in §3.
• Mooncake (Qin et al., FAST 2025) — the clearest published argument for treating the KV cache as a first-class distributed store rather than a per-server cache, and the origin of the transport that both engines now ship.
• NIXL documentation, and vLLM's connector under vllm/distributed/kv_transfer/kv_connector/v1/nixl/ — read base_worker.py end to end; it is the most honest account of what P/D actually costs.
• DeepSeek-V3.2 / DSA — the technical report for the sparse-attention mechanism whose in-tree plumbing appears in python/sglang/srt/layers/attention/nsa/ and python/sglang/srt/layers/attention/dsa/.
• vLLM issue #43186, referenced in the TODO of block W — the live discussion of the kernel divergence that forced the accuracy tolerance from 0.03 to 0.05.
• Neighbouring chapters: §13.1 for the design-decision matrix including llama.cpp and TensorRT-LLM, §13.2 for the decision procedure, and §13.4 for which of these problems is tractable as a project.