Prefill–decode disaggregation

Take a single H100 SXM serving Llama-3-8B in bf16, 64 sequences decoding, chunked prefill on with max_num_batched_tokens = 2048. A request arrives with an 8,192-token prompt. Five prefill chunks (four of 1984 tokens and a 256-token remainder) get interleaved into the decode stream. What does each one cost the sequences already in flight?

A decode step at batch 64 is weight-bound: it reads 16 GB of bf16 weights out of HBM at 3.35 TB/s, so the floor is $16 / 3.35 \times 10^3 \approx 4.8$ ms — the round figure; §0.4 tightens it to 4.48 ms by excluding the gathered embedding table, which moves nothing in the comparison below. A 2,048-token prefill chunk is compute-bound: $2NS = 2 \times 8\times10^9 \times 2048 = 3.28 \times 10^{13}$ FLOPs, and at 50 % MFU against the H100's 989 TFLOP/s dense bf16 that is 66 ms. The chunk does not replace the decode step, it is added to it.

Shrink the chunk and the spike shrinks with it — but TTFT can grow from repeated weight reads, poorer efficiency, and launch overhead; it is not generally linear in inverse chunk size, and per-chunk kernel-launch and attention-metadata overhead starts to dominate. That is the whole of the chunked-prefill tuning problem from §1.5: one knob, two objectives pulling opposite ways.

The deeper problem is that the two phases do not merely want different batch sizes, they want different machines. Take the roofline from §0.4. The H100 SXM's ridge point is

For the weight GEMMs, decode at batch $B$ does $2B$ FLOPs per bf16 weight, which is 2 bytes, so its arithmetic intensity is $I_\text{decode} \approx B$ FLOP/byte. You need $B \approx 295$ concurrent sequences before decode stops being bandwidth-bound. Prefill on a chunk of $T$ tokens has $I_\text{prefill} \approx T$, so at $T = 2048$ it is already 7× past the ridge and firmly compute-bound. One machine, one memory system, two workloads sitting on opposite sides of the knee.

Everything downstream inherits the compromise. Decode wants a high TP degree so the weight read is split across more HBM controllers; prefill gains little from TP past the point where the collectives cost more than the split saves. Decode wants every spare byte of VRAM as KV cache so $B$ can grow; prefill wants activation workspace. Colocated, you pick one number for each and both phases get a number that is wrong for them.

In the colocated lane, ITL for every already-running sequence goes 4.8 → 71 ms four times. In the disaggregated lane the decode worker never sees the prefill at all; the arriving sequence simply joins the batch at $t = 287$ ms. That is not a softening of the interference. It is its removal.

A disaggregated deployment is two independent inference clusters that share nothing except a rendezvous protocol and a fabric. A prefill pool (the P pool) runs engines configured for compute: whatever TP degree the model needs to fit, small KV pool, big token budget per batch. A decode pool (the D pool) runs engines configured for bandwidth and capacity: high TP, enormous KV pool, tiny token budget per step. A router in front of them picks one P worker and one D worker per request and hands both of them a shared correlation token. The KV cache produced on P is written directly into D's paged KV pool over RDMA, and only then does the request become schedulable on D.

Two properties fall straight out of the picture. First, the pools scale independently: P capacity is a function of prompt tokens per second, D capacity is a function of concurrent sequences, and in a real workload those two numbers move for completely different reasons. Second, the arrow between them is the entire cost of the design. Everything else in this chapter is about that arrow.

The KV cache for one token, in bytes, is the formula from §2.1:

where $L$ is layer count, $h_\text{kv}$ the number of key/value heads (8 for both Llama-3 sizes, they are GQA), $d_h$ the head dimension (128), $b$ the bytes per element (2 for bf16), and the leading 2 counts K and V. The bytes that must cross the wire for a prompt of $S$ tokens is $\beta_\text{tok} \cdot S$ — the whole prompt, because the D worker has none of it.

Now divide by link bandwidth. Three fabrics, all quoted as marketing line rate so the numbers are optimistic on purpose — real achieved bandwidth is lower, which only strengthens the conclusion:

The fourth column is the one that decides whether disaggregation is a good idea. Under tensor parallelism each rank holds only its shard of the KV heads. Llama-3-70B has $h_\text{kv} = 8$, so at TP=8 each rank owns exactly one KV head and therefore $2.68 / 8 = 0.335$ GB of the transfer — if the deployment provides independent NIC bandwidth per rank, transfers can proceed concurrently. Shared NICs, PCIe paths, and fabric contention invalidate an assumed TP-fold speedup. The transfer does not serialize across the fabric; it fans out across it. This is why the per-GPU InfiniBand device is a first-class configuration knob (--disaggregation-ib-device, which accepts a per-GPU JSON mapping in python/sglang/srt/server_args.py:L3155-L3159).

Compare against the TTFT budget. Prefill of 8,192 tokens on Llama-3-8B, one H100 at 50 % MFU, is $2 \times 8\times 10^9 \times 8192 / (0.5 \times 989\times10^{12}) = 265$ ms. For Llama-3-70B on 8×H100 at 40 % MFU it is $2 \times 70\times10^9 \times 8192 / (0.4 \times 8 \times 989\times10^{12}) = 362$ ms. So:

In the simplified projection-only model, both transfer bytes and prefill work grow linearly with prompt length S, so S cancels from their ratio below. Real viability remains length-dependent: prefill attention adds quadratic work, small transfers pay startup/handshake latency, achieved compute varies with shape, prefix hits change bytes, and queues or overlapping transfers alter end-to-end latency.

with $N$ parameters and $F$ the device's dense FLOP/s. Long prompts do not make disaggregation worse; they make it exactly as good, on a bigger absolute budget. What does make it worse is a model with a fat KV cache relative to its parameter count — which is precisely why MLA models (§3.5), whose per-token KV is a fraction of a GQA model's, are the natural fit for disaggregated serving at scale.

vLLM and SGLang arrived at genuinely different architectures here, and the difference is instructive because it is not an accident of taste — it follows from where each project put the abstraction boundary.

SGLang: a dedicated subsystem with its own event loops

SGLang treats P and D as two different programs. A server is launched in one of three modes, and the mode selects an entire scheduler event loop:

class DisaggregationMode(Enum):
    NULL = "null"
    PREFILL = "prefill"
    DECODE = "decode"

The prefill server runs event_loop_normal_disagg_prefill (python/sglang/srt/disaggregation/prefill.py:L569-L604), the decode server runs event_loop_normal_disagg_decode (python/sglang/srt/disaggregation/decode.py:L2428-L2463). Neither is the unified loop from §1.4. Each has its own queues: PrefillBootstrapQueue and disagg_prefill_inflight_queue on P, DecodePreallocQueue and DecodeTransferQueue on D.

The transport is behind an abstract interface in python/sglang/srt/disaggregation/base/conn.py. A backend supplies four classes — manager, sender, receiver, bootstrap server — and the contract between them is a five-state poll:

class KVPoll:
    Failed = 0
    Bootstrapping = 1
    WaitingForInput = 2
    Transferring = 3
    Success = 4

The two halves of the handshake are the sender's init/send and the receiver's send_metadata. Note the direction carefully — it is the receiver that pushes metadata:

    @abstractmethod
    def send_metadata(
        self,
        kv_indices: npt.NDArray[np.int32],
        aux_index: Optional[int] = None,
        state_indices: Optional[List] = None,
        decode_prefix_len: Optional[int] = None,
    ):
        """
        Notify the prefill server about the kv indices, aux index, and state_indices.
        """

D allocates its own KV pages first, then tells P the page indices to write into. The transfer is therefore an RDMA WRITE issued by P into memory D has already reserved. The state transition that unblocks P is explicit in the Mooncake manager's bootstrap thread:

    def start_prefill_thread(self):
        def bootstrap_thread():
            """This thread recvs pre-alloc notification from the decode engine"""
            # KVPoll.Bootstrapping -> KVPoll.WaitingForInput
            while True:

This single line is the load-bearing fact of the whole design: a prefill does not start until the decode side has reserved memory for its output. Backpressure is structural, not advisory.

vLLM: the KV connector, and no P/D subsystem at all

vLLM has no disaggregation/ directory. It has vllm/distributed/kv_transfer/, whose abstraction is a connector: an object with a scheduler-side half and a worker-side half that the ordinary V1 scheduler calls at fixed points in its ordinary loop. The base class docstring is the cleanest statement of the contract:

    Scheduler-side: runs in the scheduler, binds metadata, which
    is used by the worker-side to load/save KV cache.
        get_num_new_matched_tokens() - get number of new tokens
            that exist in the remote KV cache. Might be called multiple
            times for a given request and should be side-effect free.
        update_state_after_alloc() - update KVConnector state after
            temporary buffer alloc by the CacheManager.
# ...
        request_finished() - called once when a request is finished,
            with the computed kv cache blocks for the request.
            Returns whether KV cache should be freed now or if the
            connector now assumes responsibility for freeing the
            the blocks asynchronously. Also optionally returns KV
            transfer params.
# ...
    Worker-side: runs in each worker, loads/saves KV cache to/from
    the Connector based on the metadata.
# ...
        start_load_kv() - starts loading all KVs (maybe async)
        wait_for_layer_load() - blocks until layer i load is done

        save_kv_layer() - starts saving KV for layer i (maybe async)
        wait_for_save() - blocks until all saves are done

        get_finished() - called with ids of finished requests, returns
            ids of requests that have completed async sending/recving.

Disaggregation is then expressed in that interface rather than implemented alongside it. On the D node, the request arrives carrying kv_transfer_params set by an external proxy; the connector reports the whole prompt as “already computed, remotely”:

        if params is not None and params.get("do_remote_prefill"):
            # Remote prefill: get all prompt blocks from remote.
            token_ids = request.prompt_token_ids or []
            actual = self._get_remote_prefill_token_count(len(token_ids))
            count = actual - num_computed_tokens
            if count > 0:
                return count, True

The True is “asynchronously”. The scheduler allocates blocks, parks the request in RequestStatus.WAITING_FOR_REMOTE_KVS (vllm/v1/request.py:L364-L371), and the worker-side connector pulls the blocks over NIXL. On the P node, request_finished() returns True to say “do not free my blocks yet, I own them now.” Both sides converge in one function:

    def _update_from_kv_xfer_finished(self, kv_connector_output: KVConnectorOutput):
        """
        KV Connector: update the scheduler state based on the output.

        The Worker side connectors add finished_recving and
        finished_sending reqs to the output.
        * if finished_sending: free the blocks
        # if finished_recving: add to state so we can
            schedule the request during the next step.
        """
# ...
        for req_id in kv_connector_output.finished_recving or ():
            logger.debug("Finished recving KV transfer for request %s", req_id)
            assert req_id in self.requests
            req = self.requests[req_id]
            if req.status == RequestStatus.WAITING_FOR_REMOTE_KVS:
                self.finished_recving_kv_req_ids.add(req_id)
            else:
                assert RequestStatus.is_finished(req.status)
                self._free_blocks(self.requests[req_id])
        for req_id in kv_connector_output.finished_sending or ():
            logger.debug("Finished sending KV transfer for request %s", req_id)
            assert req_id in self.requests
            self._free_blocks(self.requests[req_id])

Twenty-nine lines. That is vLLM's entire disaggregation state machine in the scheduler, because everything else lives in the connector. The tradeoff is stark and symmetric:

The generality is real, not rhetorical: the same KVConnectorBase_V1 that carries P/D also carries CPU offload and remote KV stores, all registered in one table (vllm/distributed/kv_transfer/kv_connector/factory.py:L148-L242). That is why §2.6 and this chapter are about the same file in vLLM and about different files in SGLang.

Layer-wise transfer: who actually overlaps

The prize is to start moving layer $\ell$'s KV while layer $\ell+1$ is still computing, so the transfer hides entirely inside prefill. vLLM has the hook wired directly into the attention layer as a decorator:

    @wraps(func)
    def wrapper(*args, **kwargs):
        if not has_kv_transfer_group() or not is_v1_kv_transfer_group():
            return func(*args, **kwargs)

        layer_name = _resolve_layer_name(args[layer_name_index])

        # Extract attention context (metadata, layer, kv_cache, layer_slot_mapping)
        attn_metadata, _, kv_cache, _ = get_attention_context(layer_name)
        connector = get_kv_transfer_group()
        if attn_metadata is None or not connector.has_connector_metadata():
            return func(*args, **kwargs)

        # Wait for KV layer on entry
        connector.wait_for_layer_load(layer_name)

        # Execute the function
        result = func(*args, **kwargs)

        # Save KV cache layer on exit
        connector.save_kv_layer(layer_name, kv_cache, attn_metadata)

        return result

But most connectors decline it. NixlConnector.save_kv_layer is a bare pass with the comment “NixlConnector does not save explicitly” (vllm/distributed/kv_transfer/kv_connector/v1/nixl/connector.py:L326-L334), and so is MooncakeConnector's (vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py:L580-L588). The connector that does use it is MoRIIO in WRITE mode, which issues the RDMA write per layer as the layer finishes:

            return
        if self.mode == MoRIIOMode.READ:
            return
        remote_engine_id = None

        for req_id, meta in metadata.reqs_to_save.items():
            # we only need to check if dp0 in rank
            remote_engine_id = (
                str(meta.remote_host) + ":" + str(meta.remote_handshake_port)
            )

            meta.remote_engine_id = remote_engine_id

            dp0_remote_engine_id = self.get_engine_name_with_dp(remote_engine_id, 0)
            if dp0_remote_engine_id not in self._remote_agents:
                # Initiate handshake with remote engine to exchange metadata.
                with self._handshake_lock:
                    if remote_engine_id not in self._remote_agents:
                        self._background_moriio_handshake(
                            req_id, remote_engine_id, meta
                        )

                        continue
            self._write_blocks_for_req(req_id, meta, layer_name, kv_layer)

SGLang overlaps on a different axis: not per layer, but per prefill chunk. Because prefill is chunked anyway, the scheduler fires a transfer for each completed chunk from inside process_prefill_chunk:

            elif self.enable_overlap:
                # Delay KV transfer to process_batch_result_disagg_prefill when overlap is enabled to ensure results are resolved
                req.tmp_end_idx = min(
                    req.extend_range.end,
                    len(req.origin_input_ids),
                )
            else:
                self.send_kv_chunk(req)

Chunk $k$'s KV flies while chunk $k+1$ computes. With a 2,048-token chunk on Llama-3-8B, each send is 268 MB, 5.4 ms over NDR IB against a 66 ms chunk — the transfer disappears into the compute completely except for the final chunk. Within a chunk, Mooncake batches all layers into a single RDMA call rather than one per layer, on the explicit grounds that it beats threading:

    def _transfer_data(self, mooncake_session_id, transfer_blocks):
        if not transfer_blocks:
            return 0

        src_addrs, dst_addrs, lengths = zip(*transfer_blocks)
        return self.engine.batch_transfer_sync(
            mooncake_session_id, list(src_addrs), list(dst_addrs), list(lengths)
        )

There is a further optimisation gated behind SGLANG_DISAGG_PREFILL_EARLY_SEND_CACHED_PREFIX: if the P worker's radix cache already holds a device-resident prefix of this prompt, it ships those pages before the forward pass is even enqueued (python/sglang/srt/disaggregation/prefill.py:L1082-L1120). That path has to record a CUDA event on the forward stream first, because under overlap scheduling the previous step may still be writing those exact pages.

The backends

SGLang's registry names five, plus a test double:

class TransferBackend(Enum):
    MOONCAKE = "mooncake"
    MORI = "mori"
    NIXL = "nixl"
    ASCEND = "ascend"
    FAKE = "fake"

        qp_per_transfer = envs.SGLANG_MORI_QP_PER_TRANSFER.get()
        post_batch_size = envs.SGLANG_MORI_POST_BATCH_SIZE.get()
        num_worker_threads = envs.SGLANG_MORI_NUM_WORKERS.get()

        rdma_cfg = RdmaBackendConfig(
            qp_per_transfer,
            post_batch_size,
            num_worker_threads,
            poll_mode,
            False,
        )
        engine.create_backend(BackendType.RDMA, rdma_cfg)

vLLM's equivalents are registered as connectors rather than backends: NixlConnector / NixlPullConnector / NixlPushConnector, MooncakeConnector, MoRIIOConnector, all in vllm/distributed/kv_transfer/kv_connector/factory.py:L176-L222. The same three transports, wearing the shape of whichever engine they are plugged into.

Follow a single request through SGLang with the Mooncake backend, naming functions in order.

1. Router fan-out. The load balancer picks a (P, D) pair, generates a 63-bit random bootstrap_room, and posts the same request body to both servers. The load balancer makes the intent explicit — tasks = [session.post(prefill_server...), session.post(decode_server...)], with the comment “Wait for both responses to complete. Prefill should end first.” (sgl-model-gateway/bindings/python/src/sglang_router/mini_lb.py:L135-L141, room generation at :L376-L382).
2. P side registration. PrefillBootstrapQueue.add → create_sender constructs a MooncakeKVSender bound to (bootstrap_addr, bootstrap_room) and sets max_new_tokens = 1 so the prefill adder's memory estimate is honest (python/sglang/srt/disaggregation/prefill.py:L299-L324, :L377-L381). The request now sits in the bootstrap queue polling Bootstrapping.
3. D side registration and preallocation. DecodePreallocQueue.add → _create_receiver_and_enqueue builds a MooncakeKVReceiver for the same room (python/sglang/srt/disaggregation/decode.py:L662-L681). Then pop_preallocated checks the token budget, allocates real KV pages for the entire prompt, and pushes their indices to P:

            decode_req.kv_receiver.send_metadata(
                page_indices,
                decode_req.metadata_buffer_index,
                state_indices,
                **metadata_kwargs,
            )

4. P unblocks. The Mooncake bootstrap thread receives that message, flips the room's status, and P's next pop_bootstrapped() sees WaitingForInput. It calls finalize_bootstrap, which allocates a metadata buffer slot and sizes the send:

        req.time_stats.set_bootstrap_done_time()
        decode_prefix_len = req.disagg_kv_sender.pop_decode_prefix_len()
        num_kv_indices = len(req.origin_input_ids)
        req.start_send_idx = decode_prefix_len
        # Base of the staging chunk grid (suffix-relative send coordinates).
        req.disagg_decode_prefix_len = decode_prefix_len
        num_kv_indices_to_send = num_kv_indices - decode_prefix_len
        num_pages = kv_to_page_num(
            num_kv_indices_to_send,
            self.scheduler.token_to_kv_pool_allocator.page_size,
        )
        req.disagg_kv_sender.init(num_pages, req.metadata_buffer_index)
        req.pending_bootstrap = False
        return True

decode_prefix_len is the number of leading tokens D already has in its own radix cache. P skips those entirely — prefix caching on the decode side directly shrinks the wire transfer, which is the reason --disaggregation-decode-enable-radix-cache exists.

5. Prefill and streaming send. The request enters waiting_queue, gets batched, and process_prefill_chunk calls send_kv_chunk(req) after each chunk. send_kv_chunk gathers req_to_token[req_pool_idx, start:end], converts token indices to page indices, and calls sender.send(page_indices, ...), which enqueues a TransferKVChunk onto a FastQueue drained by MooncakeKVManager.transfer_worker (python/sglang/srt/disaggregation/mooncake/conn.py:L1617-L1640). Non-final chunks are truncated to a page boundary so no partial page is ever written twice.
6. Last chunk carries the sampled token. On last_chunk=True, self.disagg_metadata_buffers.set_buf(req) packs the first sampled token id, logprobs and any hidden states into an aux buffer that rides along with the final RDMA write. This is why D can start decoding immediately: it receives the KV and token 1 together and never re-runs a forward pass over the prompt.
7. P retires. process_disagg_prefill_inflight_queue polls every in-flight sender; on Success it unlocks the tree cache and streams a zero-length completion to the client so the router's prefill leg returns.
8. D admits. DecodeTransferQueue.pop_transferred sees Success, calls _commit_transfer_to_req, and the request lands in D's ordinary waiting_queue — from here on it is an ordinary decode request and §1.4 takes over.

The decode pool is full when a prefill finishes

It cannot be, and that is the point. Because D preallocates before P is unblocked, a full decode pool means pop_preallocated never reaches the request, send_metadata is never sent, and P's sender stays in Bootstrapping — it never burns a single GPU-second on a prefill whose output has nowhere to land. The queue is on the P side, in the cheapest possible place.

The escape hatch is --optimistic-prefill-attempts, which lets P start computing before bootstrap completes (python/sglang/srt/server_args.py:L3198-L3202). If a properly bootstrapped request shows up while an optimistic one is mid-flight, the optimistic one yields: optimistic_release_and_requeue releases its KV, resets start_send_idx, and pushes it back to the front of the waiting queue (python/sglang/srt/disaggregation/prefill.py:L1310-L1343). You are trading recomputation risk for latency.

Bootstrap timeouts, and the log line you will actually see

    def _check_bootstrap_timeout(self) -> Optional[KVPoll]:
        if self.init_time is None:
            return None
        elapsed = time.time() - self.init_time
        if elapsed < self.kv_mgr.bootstrap_timeout:
            return None
        logger.warning_once(
            "Some requests timed out when bootstrapping, "
            "which means prefill instances fail to receive the KV indices from the decode instance of this request. "
            "If a greater mean TTFT is acceptable, you can 'export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600' (10 minutes) to relax the timeout condition. "
        )
        self.kv_mgr.record_failure(
            self.bootstrap_room,
            f"Request {self.bootstrap_room} timed out after {elapsed:.1f}s "
            f"in KVPoll.Bootstrapping",
        )
        self.kv_mgr.update_status(self.bootstrap_room, KVPoll.Failed)
        return KVPoll.Failed

This message means one of three things, in decreasing order of likelihood: the D pool is saturated and cannot preallocate; the router sent the two halves of the request to servers that disagree about the bootstrap port; or the ZMQ path from D to P is blocked by a firewall while the HTTP bootstrap path is not. Check bootstrap_room cardinality in the P and D logs first — if P sees rooms D never sees, it is the router.

The transfer fails mid-flight

Both sides tear down symmetrically. On P, handle_inflight_transfer_failure raises the backend's failure_exception() to get a real message, unlocks the tree cache, aborts with HTTP 500 and bumps transfer_failed_reqs (python/sglang/srt/disaggregation/prefill.py:L937-L964). The error string is "Prefill transfer failed for request rank=... req.rid=... req.bootstrap_room=...". On D, pop_transferred does the same with "Decode transfer failed for request rank=..." and then must decide whether it is safe to reuse the pages:

This is the subtlest correctness problem in the subsystem. If D initiated the abort, P may still have an RDMA write in flight targeting pages D is about to hand to another request. So D defers the release until P acknowledges that its writes have drained — _defer_release, an ABORT / ABORT_ACK exchange over ZMQ, and a timeout backstop (python/sglang/srt/disaggregation/decode.py:L2283-L2295; the P-side ack handling is at python/sglang/srt/disaggregation/mooncake/conn.py:L1996-L2020). If you see KV corruption under abort-heavy load, this is the machinery to look at.

Heterogeneous TP is supported, and it is not free

Running P at TP=4 and D at TP=8 means each P rank's KV shard must be split across two D ranks. Both engines implement the remapping — SGLang in send_kvcache_slice (python/sglang/srt/disaggregation/mooncake/conn.py:L1003-L1122), vLLM in a dedicated TPMapping computed once per remote engine during handshake (vllm/distributed/kv_transfer/kv_connector/v1/nixl/tp_mapping.py:L38-L56). The cost is that the transfer stops being one contiguous block per layer and becomes many strided sub-slices, which is exactly the access pattern RDMA is worst at. Measure before assuming hetero-TP is free.

Two neighbours this is constantly confused with

The commands below use the real Mooncake backend, not fake transfer. They require two suitable GPUs (one selected for each server), installed compatible engine/transport packages, model access, and a working supported transport. A fake backend can exercise some control paths but does not transfer valid KV or establish generated-output correctness; it is not a substitute for this end-to-end experiment.

# terminal 1 - prefill server, owns the bootstrap HTTP server on 8998
CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct \
  --port 30000 --disaggregation-mode prefill \
  --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 8998

# terminal 2 - decode server
CUDA_VISIBLE_DEVICES=1 python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct \
  --port 30001 --disaggregation-mode decode \
  --disaggregation-transfer-backend mooncake

# terminal 3 - the router that injects bootstrap_room into both halves
python -m sglang_router.launch_router --pd-disaggregation \
  --prefill http://127.0.0.1:30000 8998 --decode http://127.0.0.1:30001 --port 8000

Three things to measure, in order of what they teach you:

1. Where the time goes. SGLang records bootstrap_done_time, decode_transfer_queue_entry_time and prefill_kv_transfer_finish_time per request and exports kv_transfer_latency_ms and kv_transfer_speed_gb_s (python/sglang/srt/disaggregation/prefill.py:L908-L928). If speed_gb_s is under half your NIC's line rate, look at page size and at hetero-TP slicing.
2. What backpressure looks like. Push concurrency until the D pool saturates and watch the P-side bootstrap queue grow while P's GPU utilisation falls. That falling utilisation is the design working.
3. The ITL distribution. Run the same load colocated and disaggregated, and plot the ITL histogram, not the mean. Colocated is bimodal with a mode at the chunk cost; disaggregated should be unimodal. That difference is the entire product of this chapter.

For vLLM the equivalent is a proxy plus two engines with a --kv-transfer-config naming the connector and its role; the shipped harness is examples/disaggregated/disaggregated_serving/disagg_proxy_demo.py, which documents an XpYd launch in its module docstring (examples/disaggregated/disaggregated_serving/disagg_proxy_demo.py:L3-L15).

• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024) — the paper that established the goodput framing and the placement algorithm.
• Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — the production system behind Kimi, and the origin of the Transfer Engine that is SGLang's default backend.
• SGLang #4655 — [Roadmap] Prefill and Decoding Disaggregation. The tracking issue for the entire subsystem: event loops, the Mooncake and NIXL integrations, fault tolerance, spec-decode support. The best documentation that exists for SGLang's design decisions.
• vLLM #15343 — PDController and PDWorker Prototype (1p1d). Closed unmerged, and worth reading precisely for that: the discussion is where vLLM decided that P/D orchestration belongs outside the engine.
• ai-dynamo/nixl and ROCm/mori — the two transport libraries both engines integrate. NIXL abstracts over transports (UCX, GDS, UCCL); MORI-IO is GPU-direct RDMA on AMD hardware with no abstraction layer in between.
• vllm/distributed/kv_transfer/README.md — short, and the only prose in either tree describing why the connector abstraction is shaped the way it is.

The real Mooncake experiment above requires GPUs and a supported transport. The CPU transfer-fault lab isolates a narrower prerequisite: safe ownership when completion, cancellation and room reuse occur in different orders. It ships a standard-library model with a deterministic event trace and seeded tests, not an RDMA implementation or a performance benchmark.

Cancellation is a control decision, not a memory fence

Both sides of an in-flight transfer retain storage obligations. The producer must not overwrite source KV while it is being read, and the consumer must not repurpose destination pages while they may still be written. The model represents these obligations with separate pins. Aborting the request removes destination request ownership but preserves both pins until an explicit drain acknowledgement. Source mutation uses copy-on-write on a pinned partial page, so the eventual completed snapshot stays stable.

Four abort reasons share the same safety rule: user cancellation, timeout, producer failure and consumer failure. A timeout or process-failure notification alone does not authorize page reuse. The relevant question is whether the backend guarantees that no queued or remote operation can still touch those allocations. The model's acknowledge_drained() is the place where a real implementation must supply that guarantee. Without it, retaining memory is intentional quarantine, not a proven leak that can be fixed by freeing pages.

Fence attempts, not just request names

The lab assigns each transfer attempt a monotonically increasing key and treats the room name as a reusable routing label. Duplicate live rooms are rejected. A room becomes reusable only after the old attempt drains or its published output is released; a callback for the old key cannot affect its successor. Success exposes destination KV to decode only after completion. Aborting first makes subsequent completion notifications nonpublishing; completing first instead requires ordinary output release.

Generation-qualified callbacks protect metadata but do not themselves prevent a late device write. A production protocol also needs transport-level revocation, completion or incarnation fencing before address reuse, plus bounded terminal-record retention. Worker restart, registration lifetime, partial-copy corruption, incompatible cache layouts and model-version mismatch need additional real-backend tests. The CPU exercise verifies none of those by implication.

Acceptance criteria for a backend adapter

1. For each abort reason, retain source and destination allocations until the transport proves quiescence; do not publish partially transferred KV.
2. Inject duplicate completion, duplicate cancellation, delayed drain and old-attempt callbacks after room reuse. Check that the new attempt's ownership and output are unchanged.
3. Fail destination allocation midway. Verify that all temporary pages and source pins are rolled back, while the producer's original KV remains readable.
4. Cancel the producer while its request is being preempted. Verify that replay cannot reuse pinned pages, then succeeds once the transport has drained.
5. Separate eventual cleanup from safety. A missing acknowledgement must not cause early reuse; an operational test must also demonstrate the backend's actual recovery path and deadline.

The included tests cover the abstract contracts for all four abort reasons and compare published rows with an independent token-history oracle. Port these assertions to backend-specific fixtures before drawing conclusions about real engines, failure recovery time or available serving capacity.