Head-to-head design comparison

Here is the inversion, from the two harnesses at the pinned SHAs.

    if not (0.0 <= input_range_ratio < 1.0):
        raise ValueError("input_range_ratio must be in [0, 1).")
    if not (0.0 <= output_range_ratio < 1.0):
        raise ValueError("output_range_ratio must be in [0, 1).")
    num_special_tokens = int(tokenizer.num_special_tokens_to_add())
    real_input_len = max(0, int(input_len) - num_special_tokens)
    input_low = math.floor(real_input_len * (1 - input_range_ratio))
    input_high = math.ceil(real_input_len * (1 + input_range_ratio))

def compute_random_lens(full_len: int, range_ratio: float, num: int) -> List[int]:
    # full_len=0 is valid for embedding benchmarks where no output tokens are generated
    if full_len <= 0:
        return [0] * num
    return np.random.randint(
        max(int(full_len * range_ratio), 1),
        full_len + 1,
        size=num,
    ).tolist()

In vLLM, range_ratio is a fractional half-width around the mean: $r=0$ pins every length to $L$, and $r=1$ is rejected outright. In SGLang, range_ratio is the lower bound as a fraction of $L$: $r=1$ pins every length to $L$, and $r=0$ gives uniform on $[1,L]$. Both default to 0.0. A reader who copies a command from one project's README into the other's harness gets a mean input length that differs by a factor of two, and every number downstream — TTFT, prefix hit rate, tokens/s — moves with it.

This is not a bug in either project. It is what happens when two teams solve the same problem from opposite ends and nobody is obliged to reconcile the vocabulary. The book has been accumulating instances of that for twelve parts. This chapter puts them in one table.

The thesis is short: vLLM and SGLang differ far less in what they do than in where they put the complexity and what they are willing to be wrong about. Both page the KV cache, both batch continuously, both call FlashAttention and FlashInfer, both cache prefixes, both capture CUDA graphs, both are moving their serving layer into Rust. Where they diverge is in the shape of the code that surrounds those mechanisms — and each shape has a characteristic failure mode that you can predict from the shape alone.

The cleanest way to see the difference is to draw both engines at the same level of abstraction and look at where the arrows cross a process boundary.

vLLM decides centrally and ships the decision. The scheduler in the engine-core process produces a SchedulerOutput, which is broadcast over a shared-memory message queue to every worker; workers execute what they are told.

    def execute_model(  # type: ignore[override]
        self, scheduler_output: SchedulerOutput, non_block: bool = False
    ) -> ModelRunnerOutput | None | Future[ModelRunnerOutput | None]:
        return self.collective_rpc(
            "execute_model",
            args=(scheduler_output,),
            unique_reply_rank=self.output_rank,
            non_block=non_block,
            timeout=envs.VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS,
            kv_output_aggregator=self.kv_output_aggregator,
            ec_output_aggregator=self.ec_output_aggregator,
        )

SGLang replicates the decision. Rank 0 receives requests and broadcasts the inputs; every TP rank then runs an identical copy of the scheduler over them and arrives at the same batch independently.

    @scheduler_nvtx_method("scheduler.recv_requests")
    def recv_requests(
        self,
    ) -> List[Union[TokenizedGenerateReqInput, TokenizedEmbeddingReqInput, Any]]:
        """Receive results at tp_rank = 0 and broadcast it to all other TP ranks."""

Only rank zero holds the sockets. init_ipc_channels computes is_rank_zero from pp_rank == 0 and attn_tp_rank == 0 and attn_cp_rank == 0, and every other rank returns before recv_from_tokenizer is assigned (python/sglang/srt/managers/scheduler.py:L740-L768). One process reads the wire; $N$ processes reach the same conclusion about what to do with what it read.

Centralized planning communicates a plan; replicated planning communicates inputs and derives the plan locally. SGLang still has broadcast and synchronization costs, and rank-local physical slot IDs can legitimately differ. Correctness requires agreement on ordered logical requests, positions and collective-compatible operations, plus a valid local mapping on each rank. Divergence can corrupt outputs, raise or hang depending on what differs; separate addresses alone are not an error.

Take that observation one level down. Every engine has to solve the same list of problems; what differs is which module absorbs the irregularity. Call it the complexity budget. Each project spends it somewhere, and the place it spends it is where its bugs live.

vLLM spends it in the block layer. Its allocator is block-structured — DEFAULT_BLOCK_SIZE: ClassVar[int] = 16 (vllm/config/cache.py:L59) — so the KV cache is indexed by a block table of one int32 per block. At max_num_reqs=256 and a 32,768-token window that is $256 \times 2048 \times 4 = 2$ MiB of metadata addressing the 52.32 GiB KV pool of a Llama-3-8B server (derived; see §2.2). The price is that nothing is free at token granularity: a prefix that diverges mid-block may require recomputation of the unshared partial block (copy-on-write is a distinct implementation choice, not implied by paging), a hash follows the configured block granularity, and eviction order is a convention the callers must honour rather than something the data structure enforces.

    and then freed, it will be appended back with the eviction order:
    1. The least recent used block is at the front (LRU).
    2. If two blocks have the same last accessed time (allocated by the
       same sequence), the one with more hash tokens (the tail of a block
       chain) is at the front.
    Note that we maintain this order by reversing the block order when free
    blocks of a request. This operation is outside of this class.

Read that last sentence again. Leaf-first eviction — free the tail of a chain before its prefix — is correct in vLLM only because SingleTypeKVCacheManager.free happens to call self.block_pool.free_blocks(reversed(...)) (vllm/v1/core/single_type_kv_cache_manager.py:L516-L524). A caller that forgets the reversed() compiles, runs, and quietly degrades the hit rate.

SGLang spends it in the index. On CUDA its default page is a single token:

        return {"page_size": 64}
    if not is_musa():
        return {"page_size": 1}
    return {"page_size": 64}

One slot per token means zero internal fragmentation and prefix sharing at exact token boundaries with no copy-on-write machinery at all. It also means the req_to_token index is one int32 per token: $257 \times 32768 \times 4 = 32.1$ MiB at the same 256×32k operating point, allocated on the GPU where it competes with the KV cache itself (derived, §2.2). The free list is a GPU tensor sliced with self.free_pages[need_size:] rather than a CPU linked list (python/sglang/srt/mem_cache/allocator/token.py:L44-L66), which is why SGLang's allocator has no per-block Python object to walk — and also why "which blocks are free" is not a thing the CPU can cheaply inspect.

And in SGLang the eviction invariant is structural. The radix tree keeps an explicit set of evictable leaves and pops from a heap over it:

    def _evict_device_start(self, request_cnt: int) -> None:
        self._ensure_eviction_strategy()
        self._evict_device_request_cnt = request_cnt
        self._evict_device_last_node = None
        self._evict_device_heap = [
            (self.session_ref_eviction_strategy(n), n)
            for n in self.tree_core.evictable_device_leaves
        ]
        heapq.heapify(self._evict_device_heap)
        # ... the signature of _evict_device_next_node elided ...
    ) -> Optional[NodeId]:
        ct = self.component_type
        lv = self._evict_device_last_node
        if (
            lv is not None
            and lv.parent is not None
            and lv.parent in self.tree_core.evictable_device_leaves
        ):
            heapq.heappush(
                self._evict_device_heap,
                (self.session_ref_eviction_strategy(lv.parent), lv.parent),
            )
        self._evict_device_last_node = None
        while tracker[ct] < self._evict_device_request_cnt and self._evict_device_heap:
            _, x = heapq.heappop(self._evict_device_heap)
            if x not in self.tree_core.evictable_device_leaves:
                continue
            self._evict_device_last_node = x
            return x.id

That is the live path — FullComponent inside UnifiedRadixCache, not the RadixCache the blog posts point at (§12.3). A parent becomes a candidate only once its last child is gone, because it enters evictable_device_leaves only then and _evict_device_next_node pushes it back onto the heap after its child is popped. You cannot evict a prefix that something still depends on, because the data structure will not offer it to you. Same policy as vLLM; enforced instead of documented.

The preemption row is the sharpest single difference, so it is worth the arithmetic. vLLM's _preempt_request frees the victim's blocks and returns it to the head of the waiting queue (vllm/v1/core/sched/scheduler.py:L1347-L1388). Freed blocks that carry a hash go to the back of the free queue and keep that hash, so unless the pool actually turns over, the victim's prefix is still cached when it resumes. SGLang's release_req does the opposite:

    # TODO (csy): for preempted requests, we may want to insert into the tree
    release_kv_cache(req, tree_cache, is_insert=False)
    # NOTE(lsyin): we should use the newly evictable memory instantly.
    num_tokens = remaing_req_count * envs.SGLANG_RETRACT_DECODE_STEPS.get()
    evict_from_tree_cache(tree_cache, num_tokens)

is_insert=False suppresses insertion of the retracted request's new suffix; it does not erase already cached prefixes, including its own prior chunks. In either engine actual reuse depends on pressure, active references and eviction before resume. The illustrative recompute costs 3.7 ms at $f=0.99$ and 372 ms at $f=0$ compare assumed retention fractions, not engine guarantees or observed developer motivation. Measure each engine's contiguous reusable prefix after preemption.

Here is the deliverable. Two tables, same seven columns, split only so the cells stay readable.

This is the point in a comparison article where you expect a chart. There isn't one, and the reason is worth more than the chart would have been.

First, I have no GPU. Every number in this chapter is either cited to source or derived by arithmetic from FORMULAS and published model shapes, and both kinds are labelled. Presenting either as measured would be fabrication.

Second — and this holds even for someone with eight H100s — the two projects' own harnesses do not measure the same quantity. The --random-range-ratio inversion in §1 is the loud one. The quiet one is inter-token latency. vLLM records one ITL sample per SSE chunk:

                                # Decoding phase
                                else:
                                    output.itl.append(timestamp - most_recent_timestamp)

vLLM's own comment elsewhere in the harness concedes that this is not per token: it counts output length with the tokenizer instead of len(outputs[i].itl) "since multiple output tokens may be bundled together" (vllm/benchmarks/serve.py:L595-L598). SGLang's native backend divides the chunk gap by the number of new tokens and emits that many identical samples:

                                # Decoding phase
                                else:
                                    num_new_tokens = output_len - last_output_len
                                    if num_new_tokens == 0:
                                        continue
                                    chunk_gap = timestamp - most_recent_timestamp
                                    adjust_itl = chunk_gap / num_new_tokens
                                    output.itl.extend([adjust_itl] * num_new_tokens)

These produce different distributions from identical server behaviour. Under speculative decoding, where one step emits several tokens, vLLM's p99 ITL includes the whole multi-token gap as a single sample while SGLang's is smoothed by construction. A p99 ITL comparison across the two harnesses is not a comparison of the engines; it is a comparison of two definitions of the metric. §10.2 works through the rest.

The differences make a better story, but the convergence is more instructive, because it tells you which ideas are settled.

Both engines now page the KV cache, batch continuously, chunk prefills, cache prefixes, capture CUDA graphs, dispatch to FlashAttention and FlashInfer, support FP8 and 4-bit weights, disaggregate prefill from decode, and speak the OpenAI API. Both READMEs list nearly the same feature set in nearly the same order: vLLM's "vLLM is fast with" bullets (repo-root README.md, lines 28–38) and SGLang's "Fast Runtime" bullet (repo-root README.md, line 68) name paged attention, continuous batching, chunked prefill, prefix caching, speculative decoding, and prefill-decode disaggregation between them. Neither project would have written that list in 2023.

The sharpest evidence of convergence is that SGLang has adopted vLLM's KV-event schema field for field:

class BlockStored(KVCacheEvent):
    block_hashes: list[ExternalBlockHash]
    parent_block_hash: ExternalBlockHash | None
    token_ids: list[int]
    block_size: int

    lora_id: int | None

class BlockStored(KVCacheEvent):
    block_hashes: list[int]
    parent_block_hash: Optional[int]
    token_ids: list[int]
    block_size: int
    lora_id: Optional[int]
    medium: Optional[str] = None

Same class name, same field names, same order, same ZMQ publisher pattern. An engine that allocates at token granularity has no natural notion of a "block hash" at all — it adopted one so that a router written against vLLM's event stream works against it too. This is what a de facto standard looks like while it is forming.

One more convergence, less flattering to both: dead code. The CUDA PagedAttention kernel that gave vLLM its name is gone from the tree; docs/design/paged_attention.md:L3-L5 now opens with its own tombstone —

!!! warning
    This is a historical document based on the [original paper for vLLM](https://arxiv.org/abs/2309.06180).
    It no longer describes the code used in vLLM today.

— and the file it points at, attention_kernels.cu, is not in csrc/attention/ any more — that directory holds six headers and no .cu, and the only surviving hand-written paged-attention kernel is csrc/rocm/attention.cu. On the SGLang side, try_jump_forward — the compressed-FSM trick from the 2024-02 blog post — has exactly one caller in the whole tree, and that caller is ReasonerGrammarObject.try_jump_forward (python/sglang/srt/constrained/reasoner_grammar_backend.py:L226-L229) delegating to the wrapped grammar object. Nothing in managers/ or model_executor/ ever calls it. Jump-forward decoding is not wired to the scheduler at 7d89325.

The radix-cache family is the same story at larger scale. default_radix_cache_factory never constructs RadixCache, HiRadixCache, MambaRadixCache, or SWARadixCache: the default path returns a ChunkCache variant, PureSWARadixCache, LMCRadixCache, RadixCacheCpp, or — in the common case — UnifiedRadixCache (python/sglang/srt/mem_cache/registry.py:L79-L143). Plain RadixCache survives as a base class and as create_simulated, a constructor whose docstring says "for simulation purpose" (python/sglang/srt/mem_cache/radix_cache.py:L335-L350). Both projects carry the fossils of their own founding papers.

The honest summary: three of the five columns above are mostly unread, and that is the correct state for a book that refuses to describe an API from memory. If you need those cells filled, the method is the one this book uses throughout — pin a SHA, open the file, quote the lines, and cite the range.

One request, 8,192 prompt tokens, Llama-3-8B on a single H100, decoding happily. The pool runs dry. Here is the code path in each engine, in order.

vLLM. The cited scheduler path selects a victim, releases request ownership, marks PREEMPTED, resets computed-token bookkeeping and requeues it. The block-pool path prioritizes hashless versus hashed reusable entries differently, preserving the possibility of a later hit. Subsequent allocations can overwrite those blocks, and outstanding holders can delay reuse. Thus the scenario's 512 prompt blocks are not guaranteed to survive; only if measured retention is near $f=0.99$ does the illustrative 3.7 ms tail-recompute model apply.

SGLang. The cited retraction path calls release_kv_cache(..., is_insert=False), releases request-held suffix state subject to ownership rules, and may evict tree state to satisfy the current deficit. It then resets request bookkeeping. Existing cached prefixes can still match on resume; the flag does not imply a zero hit rate. The 372 ms model applies only if no relevant prefix survives. Retraction policy, prefix reuse and physical reclamation are three separate observations.

The data structures make different retention operations convenient, but they do not fix the retained fraction. Test a grid of memory pressure, reusable-prefix length, active shared references and delayed resume. For each engine record blocks available before and after preemption, actual contiguous hit length, recompute tokens and SLO goodput. Compare scenarios with equal retention before attributing a difference to scheduling or kernel performance.

The general pattern behind all four: a difference you observe between the engines is usually a difference in what the project was willing to be wrong about. vLLM is willing to be wrong about speed — it will run a slower kernel rather than fail. SGLang is willing to be wrong about availability — it will fail rather than run something it has not tuned. vLLM is willing to be wrong about determinism; SGLang cannot be. SGLang is willing to be wrong about preemption cost; vLLM is not. Read any table cell in §4 through that lens and the "why" column mostly writes itself.

Every structural claim in this chapter is reproducible without a GPU. Point these at your own checkouts and confirm the shape before you trust the prose.

V=~/Documents/other_git_repos/vllm
S=~/Documents/other_git_repos/sglang

# 1. Allocation granularity: 16 vs 1
grep -n "DEFAULT_BLOCK_SIZE" $V/vllm/config/cache.py
sed -n '2378,2397p' $S/python/sglang/srt/arg_groups/overrides.py

# 2. Eviction: convention vs invariant
sed -n '236,243p' $V/vllm/v1/core/kv_cache_utils.py
sed -n '190,223p' $S/python/sglang/srt/mem_cache/unified_cache/components/full_component.py

# 3. Determinism: vLLM's own TODO
grep -rn "make v1 scheduling deterministic" $V/vllm/

# 4. Compilation defaults
grep -n "optimization_level: OptimizationLevel" $V/vllm/config/vllm.py
grep -n 'tc_compiler: str' $S/python/sglang/srt/model_executor/cuda_graph_config.py

# 5. Which radix cache actually runs
grep -n "return .*RadixCache\|return .*ChunkCache" $S/python/sglang/srt/mem_cache/registry.py

# 6. Dead code: jump-forward has no scheduler caller
grep -rn "try_jump_forward" $S/python/ | grep -v "def try_jump_forward"

# 7. Rust convergence, by the numbers
find $V/rust -name '*.rs' | wc -l; find $V/rust -name '*.rs' -exec cat {} + | wc -l
find $S/sgl-model-gateway -name '*.rs' | wc -l
find $S/sgl-model-gateway -name '*.rs' -exec cat {} + | wc -l

Then the one that matters most: before you ever quote a cross-engine benchmark, diff the two harnesses' definitions of the metric you are quoting. vllm/benchmarks/lib/endpoint_request_func.py against python/sglang/benchmark/serving.py for ITL, and vllm/benchmarks/datasets/utils.py against python/sglang/benchmark/datasets/common.py for the workload. If they disagree, the number is not a comparison. Lab 11 is the one harness, two servers version.

• Efficient Memory Management for Large Language Model Serving with PagedAttention — the vLLM paper. Read it alongside docs/design/paged_attention.md, whose own warning that it "no longer describes the code used in vLLM today" is the most useful sentence in either project's documentation.
• SGLang: Efficient Execution of Structured Language Model Programs — RadixAttention and the compressed-FSM work, including jump-forward decoding, which is in the paper and not in the scheduler.
• Fast and Expressive LLM Inference with RadixAttention and compressed finite state machine — the two SGLang posts whose features this chapter tracks into and out of the tree.
• SGLang v0.4: zero-overhead batch scheduler, cache-aware load balancer — the origin of the replicated-scheduler design and of sgl-model-gateway's approximate radix tree.
• docs/design/model_runner_v2.md in vLLM — the clearest statement either project has published of how it handles its own technical debt: "we discovered several fundamental design mistakes and accumulated significant technical debt… we implemented Model Runner V2 (MRV2) from first principles."
• python/sglang/srt/mem_cache/README.md in SGLang — the mirror image: a layer diagram and a "where does my class go?" table, with the unification tracked in issues #25371 and #20415.
• TensorRT-LLM docs, TGI docs, llama.cpp README — the three columns this chapter did not read at a pinned SHA.

Next: §13.2 turns this table into a decision procedure — workload shape, model family, hardware, and team capacity, in that order, with no benchmark screenshots. Then §13.3 takes the rows where both columns say "none" — worker-level fault recovery, deterministic scheduling, the KV cache as a distributed system — and asks what it would take to fix them.