Choosing an engine for a workload

A team serving a document-QA product reads that SGLang's radix tree gives near-perfect prefix reuse on shared-prefix workloads, benchmarks it on Llama-3-8B against their chat traffic, sees the win, and ships. Six weeks later they add a reranking pass — one request scores many candidate passages against one query — and enable it with --enable-mis. Throughput on that path is a fraction of what the benchmark predicted, and there is no regression to find. Three subsystems turned themselves off at startup, each with a logger.warning nobody read.

    def _handle_multi_item_scoring(self):
        """Setup and validate multi-item scoring constraints.

        Auto-disables settings incompatible with MIS mechanics (CUDA graph,
        radix cache, chunked prefill). Asserts on attention backend since
        changing it silently could surprise users who intentionally picked
        a non-flashinfer backend.
        """
        if not self.enable_mis:
            return

        if self.cuda_graph_config.decode.backend != Backend.DISABLED:
            logger.warning("CUDA graph is disabled because --enable-mis is set.")
        self.cuda_graph_config.decode.backend = Backend.DISABLED
        self.cuda_graph_config.prefill.backend = Backend.DISABLED

        if not self.disable_radix_cache:
            logger.warning("Radix cache is disabled because --enable-mis is set.")
            self.disable_radix_cache = True

        if self.chunked_prefill_size != -1:
            logger.warning("Chunked prefill is disabled because --enable-mis is set.")
            self.chunked_prefill_size = -1

CUDA graphs, the radix cache, and chunked prefill, in that order. The docstring gives the reason as written — these settings are "incompatible with MIS mechanics" — and the assertion just below the quoted range requires FlashInfer "for custom attention mask support". A correct decision, correctly logged, that deletes the reason the engine was chosen.

The expensive part is the second one, and it is not simply "no cache". SGLang's cache selection is a chain, and one branch of it swaps the tree for a different class rather than for nothing:

    if ctx.effective_chunked_prefill_size is not None and ctx.disable_radix_cache:
        if not ctx.is_hybrid_swa:
            from sglang.srt.mem_cache.chunk_cache import ChunkCache

            return ChunkCache(params)
        if ctx.full_tokens_per_layer == 0:
            from sglang.srt.mem_cache.chunk_cache import PureSWAChunkCache

            return PureSWAChunkCache(params)
        from sglang.srt.mem_cache.chunk_cache import SWAChunkCache

        return SWAChunkCache(params)

ChunkCache is exactly what its docstring says: "used when radix cache is disabled" (python/sglang/srt/mem_cache/chunk_cache.py:L35-L41). Its match_prefix returns an empty index tensor and its insert is a documented no-op — "ChunkCache does not support prefix caching" (python/sglang/srt/mem_cache/chunk_cache.py:L67-L77). The tree is not degraded; it is absent, replaced by a scratchpad holding one request's own blocks.

And there is a third thing, independent of any of this: on the original chat path, the tree was probably never being scheduled for. SGLang's cache-aware policies are opt-in.

    schedule_policy: A[
        str,
        Arg(
            help="The scheduling policy of the requests.",
            choices=[
                "lpm",
                "random",
                "fcfs",
                "dfs-weight",
                "lof",
                "priority",
                "routing-key",
            ],
        ),
        NS("schedule"),
    ] = "fcfs"

The default is fcfs. The radix cache still services hits opportunistically, but the scheduler does not order the waiting queue to create them — §2.4 works through why the ordering, not the data structure, is what produces the hit rate under load.

None of this is an SGLang defect; vLLM has the mirror image of every one, and §5 quotes one. The point is the shape of the failure. The decision was made on an axis — prefix reuse — that a later feature flag switched off, on a path the original benchmark never covered. No comparison chart can show you that, because what disabled the mechanism is a property of your flags, not of the engine.

Four things are usually offered as "how to choose": workload, model, hardware, team. They are not peers. Two eliminate: if your architecture is not in the registry or your accelerator has no backend, no benchmark matters. One discriminates: among engines that both work, workload shape decides which mechanism is load-bearing, and therefore which engine's version of it you are buying. One decides whether the decision survives: team capacity determines whether you can still operate the thing after the person who chose it leaves.

Order matters because the common failure is running them backwards: pick on the axis, meet the gate in production. And note the uncomfortable consequence — for the modal case, a Llama- or Qwen-family dense model on H100s, both gates pass, so the comparison chart everyone reads is answering the one question that was never in doubt.

The rest of the chapter walks the four in that order, then folds them into an explicit question list in §8.

Almost every engine comparison is, underneath, a comparison of prefill efficiency: prefix caching, chunked prefill, cache-aware scheduling, P/D disaggregation. So the first quantity to compute is the fraction of your GPU time that prefill occupies. Call it $\phi$; everything the prefill machinery can buy you is bounded by it.

Let $T_p$ be the GPU time one request's prefill costs and $T_d$ the GPU time its decode costs, both amortised to a per-request share of a batched step. Let $\eta \in [0,1]$ be the fraction of prefill tokens a perfect prefix cache would serve from an existing hit. Then

Amdahl's law with prefill as the accelerable part. It bounds the mechanism, not either engine, and the gap between two implementations of that mechanism is some fraction of it. If the ceiling is 1.06×, radix-versus-hash is an argument about a couple of percent.

Working $\phi$ for three real workloads

Llama-3-8B on one H100 SXM, with the constants Part 10 uses throughout: 7.505×10⁹ non-embedding parameters (15.01 GB in bf16), 3.35 TB/s of HBM, 989.4 TFLOP/s of bf16 peak, 128 KiB of KV per token (§2.1). Prefill is compute-bound: $T_p = 2NP/F$. Decode is bandwidth-bound: a step costs $(W + \text{KV})/B_{\text{HBM}}$, and one request's share of $O$ output tokens is $O \, t_{\text{step}} / B$.

Now put those through the ceiling at an optimistic $\eta = 0.9$ — nine tenths of prompt tokens repeating something already cached, roughly what a stable system prompt plus a stable tool schema plus a growing conversation gives you.

This is the single most useful number in the chapter. On a decode-heavy chat product a perfect prefix cache is worth 6%, so the radix-versus-hash literature is a rounding error and you should decide on hardware fit and operability. On an agent loop with 8k prompts and short replies the same mechanism is worth 2.7×, and nothing else is worth deciding on.

Use $\eta$ as the fraction of baseline prefill cost actually removable in the fixed-resource model. A reusable-token fraction approximates it only when that cost is nearly linear and shared states eliminate the corresponding work. Suffix attention still reads cached keys; lookup, eviction, admission and changed batch sizes can alter the measured gain. At fixed other costs, $1/(1-\phi\eta)$ gives 1.25 at $\phi\eta=0.2$ and 1.667 at 0.4. It is not a universal engine-speedup bound under changing resources.

Four sub-axes, each tied to a mechanism that one engine exposes differently from the other. None of them is "which engine is faster".

Shared prefix: the discriminator is the scheduler, not the tree

Both engines cache prefixes; §2.3 and §2.4 own the structures. What differs at the decision level is how much of the queue-ordering problem each one lets you express. SGLang's policy enum has six members, split by exactly one question — does the policy consult the cache:

class CacheAwarePolicy(Enum):
    """Scheduling policies that are aware of the tree cache."""

    LPM = "lpm"  # longest prefix match
    DFS_WEIGHT = "dfs-weight"  # depth-first search weighting


class CacheAgnosticPolicy(Enum):
    """Scheduling policies that are not aware of the tree cache."""

    FCFS = "fcfs"  # first come first serve
    LOF = "lof"  # longest output first
    RANDOM = "random"
    ROUTING_KEY = "routing-key"  # prioritize by routing key frequency in running batch

vLLM has two, neither of which consults the cache:

class SchedulingPolicy(Enum):
    """Enum for scheduling policies."""

    FCFS = "fcfs"
    PRIORITY = "priority"

That is the real shared-prefix decision, and it cuts both ways. If your prefixes branch — one document, many questions; one agent scaffold, many trajectories — reordering the queue so siblings run together is worth more than the data structure, and --schedule-policy lpm is a lever vLLM does not have. If they are flat, LPM has nothing to reorder and vLLM's hash chain hits the same prefix with less machinery. Two warnings before reaching for LPM: it is not the default, and it silently downgrades to FCFS past 128 queued requests (§10.3).

Long prefill versus long decode: whose TTFT do you protect

Chunked prefill (§1.5) is on by default in both. The interesting difference is the knob each exposes for the conflict between a long prefill and the decodes it delays. vLLM lets you declare a prompt "long" and cap how much of the token budget it may take:

    long_prefill_token_threshold: int = Field(default=0, ge=0)
    """For chunked prefill, a request is considered long if the prompt is
    longer than this number of tokens. 0 disables the cap (default)."""

SGLang instead lets you guarantee decode progress between prefill batches:

    prefill_decode_interval: A[
        int,
        "The number of decode rounds to run after a prefill batch before scheduling the next prefill. In data-parallel attention mode, the interval is synchronized across all DP ranks. Set to 0 to disable.",
        NS("schedule"),
    ] = 0

These are duals, and both default to 0, meaning off. vLLM's threshold bounds the damage one outlier prompt does; SGLang's interval bounds the rate at which prefill is admitted at all — the stronger guarantee for inter-token latency, the more expensive one for TTFT. If your SLO is written on ITL jitter under a mixed short/long workload, SGLang's knob maps onto it directly; if your problem is one pathological 200k-token prompt among 2k-token neighbours, vLLM's does.

Strict TTFT versus throughput-maximising batch

This axis chooses a configuration, not an engine — but it changes what a comparison means. A TTFT-bound deployment runs at low utilisation by construction, so per-step overhead and CUDA-graph coverage dominate; a saturated offline batch is decided by scheduler packing and memory efficiency. Engines routinely swap places between those regimes (§10.3). Decide which regime you are in before reading anyone's numbers, and discard every number taken in the other.

Bursty versus steady arrivals

Burstiness makes admission and preemption first-order, and preemption cost is where the two engines genuinely diverge — §13.1 has the row and the derived numbers, §1.4 the mechanism. If your traffic is bursty enough that preemption is routine, that row outweighs any prefix-cache comparison, because the cost lands exactly when your SLO is already in trouble. If arrivals are steady and you run below the knee, preemption is a tail event and the row is trivia.

Gate 1: is your architecture actually served

The two registries have opposite failure modes, which is more decision-relevant than the raw counts. vLLM's is an explicit table of architecture string to module:

_TEXT_GENERATION_MODELS = {
    # [Decoder-only]
    "AfmoeForCausalLM": ("afmoe", "AfmoeForCausalLM"),
    "ApertusForCausalLM": ("apertus", "ApertusForCausalLM"),
    "ArceeForCausalLM": ("arcee", "ArceeForCausalLM"),
    "ArcticForCausalLM": ("arctic", "ArcticForCausalLM"),

SGLang discovers models by importing every module in the package and looking for an EntryClass:

def import_model_classes(package_name: str, strict: bool = False):
    model_arch_name_to_cls = {}
    package = importlib.import_module(package_name)
    for _, name, ispkg in pkgutil.iter_modules(package.__path__, package_name + "."):
        if not ispkg:
            if name.split(".")[-1] in envs.SGLANG_DISABLED_MODEL_ARCHS.get():
                logger.debug(f"Skip loading {name} due to SGLANG_DISABLED_MODEL_ARCHS")
                continue

            try:
                module = importlib.import_module(name)
            except Exception as e:
                if strict:
                    raise
                logger.warning(f"Ignore import error when loading {name}: {e}")
                continue

Read the except Exception. A module that fails to import — a missing dependency, a kernel package that did not build — warns and vanishes from the registry, and the symptom is "architecture not supported" for a model that is in the tree. vLLM's static table cannot fail that way, at the cost that adding a model means editing a central file.

The model family also silently reconfigures the engine. vLLM derives whether prefix caching is even legal from the model's attention type and pooling config:

    def is_prefix_caching_supported(self) -> bool:
        attn_type = self.attn_type

        if pooler_config := self.pooler_config:
            # for pooling models
            if attn_type == "encoder_only":
                logger.debug(
                    "Pooling models with bidirectional attn "
                    "do not support prefix caching."
                )
                return False

Encoder-only bidirectional attention generally prevents reusing a prefix's cached states across different suffixes because those states depend on the full sequence. This is not a statement about all embeddings: decoder-based embedding models can be causal, and repeated identical complete inputs can still be cached at an appropriate level. Check the checkpoint's attention, pooling semantics and runtime support separately; task name alone is insufficient.

SGLang's version of the same thing is per-architecture rather than per-attention-type, and it fires on multimodal models whose processors cannot be chunked:

        if (
            model_config.is_multimodal
            and not model_config.is_multimodal_chunked_prefill_supported
        ):
            self.chunked_prefill_size = -1
            logger.info(
                f"Automatically turn off --chunked-prefill-size as it is not supported for "
                f"{hf_config.model_type}"
            )

Note what this one does not do: it disables chunked prefill but leaves the radix cache alone, so a VLM keeps its tree and loses only the prefill-admission machinery. The cascade that costs you the tree needs disable_radix_cache too, set automatically at this SHA for EmbeddingGemma (python/sglang/srt/server_args.py:L3927-L3929), HRM-Text with prefix_lm (L3876-L3878), and --enable-mis. Read the startup log; do not reason about it.

Gate 2: hardware

vLLM has five built-in platforms and an entry-point group for everything else, with out-of-tree plugins taking precedence over built-ins:

builtin_platform_plugins = {
    "tpu": tpu_platform_plugin,
    "cuda": cuda_platform_plugin,
    "rocm": rocm_platform_plugin,
    "xpu": xpu_platform_plugin,
    "cpu": cpu_platform_plugin,
}

SGLang instead carries hardware backends in-tree: python/sglang/srt/hardware_backend/ holds cpu, gpu, mlx, musa, npu, and xpu subtrees, and the attention-backend list names the accelerators directly:

ATTENTION_BACKEND_CHOICES = [
    # Common
    "triton",
    "torch_native",
    "flex_attention",
    "dsa",
    "nsa",  # Deprecated alias for "dsa"
    "dsv4",
    "compressed",  # Deprecated alias for "dsv4"
    # ...
    # AMD specific
    "aiter",
    "wave",
    # Other platforms
    "intel_amx",
    "ascend",
    "intel_xpu",
]

So: TPU is a vLLM built-in with no SGLang counterpart at this SHA. Ascend NPU is in-tree in SGLang (an ascend backend plus an npu/ tree with its own allocator, memory pool, MoE and quantization) and an out-of-tree plugin on vLLM. Apple Silicon appears in SGLang via mlx, not in vLLM. AMD and Intel XPU are first-class in both. Off NVIDIA and AMD the gate is usually decisive — in different directions.

The single-24GB-card case

Both engines tier their defaults by device memory, and there you can read each project's stated target. vLLM has three tiers and a candid TODO:

        if device_memory >= 160 * GiB_bytes:
            # for GPUs like B200/B300 with >= 160GB memory, use the largest defaults
            default_max_num_batched_tokens = {
                UsageContext.LLM_CLASS: 16384,
                UsageContext.OPENAI_API_SERVER: 16384,
            }
            default_max_num_seqs = {
                UsageContext.LLM_CLASS: 1024,
                UsageContext.OPENAI_API_SERVER: 1024,
            }
        elif device_memory >= 70 * GiB_bytes and "a100" not in device_name:
            # For GPUs like H100 and H200, use larger offline defaults.
            default_max_num_batched_tokens = {
                UsageContext.LLM_CLASS: 16384,
                UsageContext.OPENAI_API_SERVER: 8192,
            }
            default_max_num_seqs = {
                UsageContext.LLM_CLASS: 1024,
                UsageContext.OPENAI_API_SERVER: 1024,
            }
        else:
            # TODO(woosuk): Tune the default values for other hardware.
            default_max_num_batched_tokens = {
                UsageContext.LLM_CLASS: 8192,
                UsageContext.OPENAI_API_SERVER: 2048,
            }
            default_max_num_seqs = {
                UsageContext.LLM_CLASS: 256,
                UsageContext.OPENAI_API_SERVER: 256,
            }

SGLang has six, and the bottom two are named after consumer cards:

        if gpu_mem is not None:
            if gpu_mem < 20 * 1024:
                # T4, 4080
                # (chunked_prefill_size 2k, max_bs 8)
                if self.chunked_prefill_size is None:
                    self.chunked_prefill_size = 2048
                if decode_cuda_graph_config.max_bs is None:
                    decode_cuda_graph_config.max_bs = 8
            elif gpu_mem < 35 * 1024:
                # A10, 4090, 5090
                # (chunked_prefill_size 2k, max_bs 24 if tp < 4 else 80)
                if self.chunked_prefill_size is None:
                    self.chunked_prefill_size = 2048
                if decode_cuda_graph_config.max_bs is None:
                    if self.tp_size < 4:
                        decode_cuda_graph_config.max_bs = 24
                    else:
                        decode_cuda_graph_config.max_bs = 80

Read them as declarations of intent. vLLM collapses everything below 70 GB into one bucket with an explicit "tune the default values for other hardware" note; SGLang names T4, 4080, A10, 4090 and 5090 and picks a CUDA-graph batch ceiling for each. On a single 24 GB card, SGLang has thought about you more recently.

The quantization surface says it louder. SGLang's list includes the formats that matter when weights do not fit:

QUANTIZATION_CHOICES = [
    "awq",
    "fp8",  # MOE + linear online quantization.
    "mxfp8",  # MOE + linear online quantization.
    "gptq",
    "marlin",
    "gptq_marlin",
    "awq_marlin",
    "bitsandbytes",
    "gguf",
    # ...
    # Apple Silicon MLX backend — on-the-fly quantization of fp16 weights at load
    # time via mlx.nn.quantize. Only takes effect when SGLANG_USE_MLX=1.
    "mlx_q4",  # 4 bits, group_size=64 (mlx-community default)
    "mlx_q8",  # 8 bits, group_size=64
    "unquant",
    "humming",
]

vLLM's list, at this SHA, does not:

QuantizationMethods = Literal[
    "awq",
    "auto_awq",
    "fp8",
    "fbgemm_fp8",
    "fp_quant",
    "modelopt",
    "modelopt_fp4",
    "modelopt_mxfp8",
    "modelopt_mixed",
    "auto_gptq",
    "gptq",
    "gptq_marlin",
    "awq_marlin",
    "humming",
    "compressed-tensors",
    "experts_int8",
    "quark",
    "moe_wna16",
    "torchao",
    "inc",
    "mxfp4",
    "gpt_oss_mxfp4",
    "deepseek_v4_fp8",
    "online",

For a single 24 GB card the honest answer is often neither, and §9 says why.

One node, and more than one

On one 8×80 GB node both engines are mature and the gate passes; decide on the axis. Beyond one node the question is what ships in the box: SGLang ships a Rust gateway (sgl-model-gateway/) with cache-aware routing built in, vLLM ships no router and exports KV-cache events for someone else's to consume. §9.4 owns that comparison. In one line: SGLang gives you a policy you did not write; vLLM gives you truth you must write a policy against.

Two questions: how much surface must an operator hold in their head, and where can an engineer cut in when the answer is "we need X and it is not there".

On surface area, the two projects made opposite structural choices. SGLang derives its entire CLI from one dataclass:

    def add_cli_args(parser: argparse.ArgumentParser):

        # Auto-derived from Annotated[..., Arg(...)] field metadata.
        add_cli_args_from_dataclass(parser, ServerArgs)

ServerArgs carries 475 annotated fields in a single 9,400-line class body (between class ServerArgs at L458 and the last field at L9858). vLLM splits the equivalent surface across 31 typed config modules in vllm/config/, roughly 14,000 lines. Neither is small. SGLang's is greppable in one file with no type-level grouping; vLLM's is grouped and individually documented, and requires you to know which of 31 files owns the flag.

On extension, the seams are genuinely different in kind. vLLM's are entry-point groups resolved at import time, one per process boundary:

# Default plugins group will be loaded in all processes(process0, engine core
# process and worker processes)
DEFAULT_PLUGINS_GROUP = "vllm.general_plugins"
# IO processor plugins group will be loaded in process0 only
IO_PROCESSOR_PLUGINS_GROUP = "vllm.io_processor_plugins"
# Platform plugins group will be loaded in all processes when
# `vllm.platforms.current_platform` is called and the value not initialized,
PLATFORM_PLUGINS_GROUP = "vllm.platform_plugins"
# Stat logger plugins group will be loaded in process0 only when serve vLLM with
# async mode.
STAT_LOGGER_PLUGINS_GROUP = "vllm.stat_logger_plugins"
# Endpoint plugins group is loaded in the API server front end process only.
# Each entry point resolves to a factory returning an `EndpointPlugin`
# (see `vllm/plugins/endpoint_plugins/interface.py`).
ENDPOINT_PLUGINS_GROUP = "vllm.endpoint_plugins"

SGLang's are in-process registration functions called at import: register_radix_cache_backend for the prefix cache (§12.3), SpeculativeAlgorithm.register for a decoding algorithm, and the add_*_choices helpers at python/sglang/srt/server_args.py:L397-L454, which let an out-of-tree package extend the CLI's own choice lists before ServerArgs is built. §11.5 and §12.4 audit both.

The uncomfortable truth is that both engines are large enough that "we will extend it" is a staffing commitment, not a flag. If nobody on the team can read a scheduler, decide on the gates and the axis, configure conservatively, and treat both extension-point chapters as reference rather than plan. One factor this book cannot cite: vLLM has the larger operator population, so more of your failure modes are already someone else's answered issue. Treat that as a prior, not evidence.

Eight questions, in order. Stop at the first that gives an answer.

Concrete case. A coding agent: 4×H100-80GB on one node, an MoE model in the 30B-total / 3B-active class, prompts averaging 12k tokens (a stable ~2k system prompt and tool schema, a repository context block that is stable within a session, and a growing tool-call transcript), replies averaging 400 tokens, arrivals bursty around a working day, SLO written as p95 time-to-first-token under 3 s.

1. Q1 — architecture. Both trees carry the Qwen3-MoE family, but do not stop at the family: check the exact architectures[0] from the checkpoint's config.json against vLLM's _TEXT_GENERATION_MODELS and SGLang's discovered set, and start both servers to confirm SGLang swallowed no import error. both pass
2. Q2 — hardware. NVIDIA datacentre, one node. Both first-class. Both have H100 tuned MoE configs in their fused-MoE config directories. both pass
3. Q3 — silent disable. Text-only, so the multimodal chunked-prefill auto-off does not fire; not a pooling model, so vLLM's prefix-caching gate does not fire. Read the startup log anyway. clean
4. Q4 — the ceiling. Measure prefill and decode work for this checkpoint and batch distribution. The prompt/output ratio does not establish the phase fraction, and active experts per token do not equal all experts touched by an aggregate batch. No value above 0.75, or resulting 2.7× ceiling, is established here.
5. Q5 — branching. Heavily branching: one repository context, many tool-call trajectories diverging from it, many turns per session. This is the case SGLang's tree and LPM ordering were built for.
6. Q6 — SLO. p95 TTFT is a prefill-side SLO, the same side the axis is on. Consistent. Had it been ITL, SGLang's --prefill-decode-interval would be the lever; here it is not.
7. Q7 — team. Include operating cost and ownership throughout; Q4 remains unmeasured and cannot settle this choice.
8. Q8 — verify. Below.

Screening hypothesis, not an established winner. Include SGLang with explicitly configured LPM as one candidate for this branching workload, then compare it to vLLM on identical traces and quality requirements. The MoE prefill fraction is unmeasured, so no engine recommendation follows from $\phi>0.75$ here. Measure phase work, actual saved cost, queue depth, p95 TTFT, goodput and operational behavior. Failure of a cache hypothesis does not make all remaining engine differences disappear.

llama.cpp / Ollama — when the machine is yours and the weights do not fit. The condition: a single consumer card, a laptop, or CPU-only, with a model too large for VRAM. Its docs describe a "Fast, lightweight, pure C/C++ HTTP server" with continuous batching on by default, -np, --parallel N for "number of server slots", --cache-prompt enabled by default, and -ngl, --gpu-layers for the "max. number of layers to store in VRAM" (server README). Note what that has and vLLM at a556f3f does not: GGUF weights and a first-class partial-offload story. It is not single-user — it batches. It optimises for one machine you own rather than a fleet you rent.

TensorRT-LLM — when you are NVIDIA-only and want NVIDIA's own kernels. NVIDIA's overview describes a "comprehensive open-source library for accelerating and optimizing inference performance of the latest large language models (LLMs) on NVIDIA GPUs", "Architected on Pytorch", with "Seamless distributed inference with tensor, pipeline, and expert parallelism across multiple GPUs and nodes" (overview). The ahead-of-time engine build that used to define it is gone: a migration guide states that "The TensorRT engine backend has been removed", that "PyTorch is now the sole execution backend for TensorRT LLM", and that there is "no engine-build step — HuggingFace checkpoints load directly" (migration guide). So the old winning condition — a shape-stable model you were willing to compile for — no longer describes it. What remains is the narrower one: NVIDIA hardware you control, and a preference for kernels and parallelism strategies maintained by the vendor of the silicon. §13.1 has the same reconciliation in its table.

A hosted API — when your utilisation is low. Most teams should price this and almost none do. One 8×H100 node at a rented $2–3 per GPU-hour is $16–24 per hour and $12k–17k per month whether or not a request arrives. Divide by your actual monthly token volume for your true cost per million tokens, then compare a provider's list price. A deployment at 10% utilisation pays 10× its marginal cost, and no engine choice recovers that — only batching traffic into fewer, busier replicas does. Self-hosting wins on sustained high utilisation, data residency, models nobody hosts, and latency floors an API cannot promise; not on price at low volume. §10.5 turns a curve into exactly this number.

TGI is no longer a live option. Hugging Face's own documentation opens with a caution: "text-generation-inference is now in maintenance mode. Going forward, we will accept pull requests for minor bug fixes, documentation improvements and lightweight maintenance tasks", and recommends "vllm, SGLang, as well as local engines with inter-compatibility such as llama.cpp or MLX" (TGI index, fetched while writing). It is a historical note, not a column in a decision matrix.

Six mechanical ways a published comparison misleads, each with its control. Every one is a knob in a harness both projects ship.

There is a seventh that is specific to this chapter's subject. The default synthetic dataset in vLLM's harness has no shared prefix at all:

    # Default values copied from benchmark_serving.py for the random dataset.
    DEFAULT_PREFIX_LEN = 0
    DEFAULT_RANGE_RATIO = 0.0
    DEFAULT_INPUT_LEN = 1024
    DEFAULT_OUTPUT_LEN = 128

DEFAULT_PREFIX_LEN = 0: --dataset-name random with default flags measures $\eta = 0$, the exact worst case for the mechanism that per §3 decides an agent workload. A comparison run that way cannot see the axis you care about, and most published comparisons are run that way. (DEFAULT_RANGE_RATIO = 0.0 is a second problem; §10.1 owns it.)

So measure $\phi$ and $\eta$ first, then run the two arms that differ. Both harnesses ship a shared-prefix generator: vLLM's PrefixRepetitionRandomDataset (vllm/benchmarks/datasets/datasets.py:L4380) and SGLang's generated_shared_prefix with Zipf-distributed prefix groups (python/sglang/benchmark/datasets/generated_shared_prefix.py:L23-L41).

# Exploratory workload sensitivity only: these datasets have different defaults.
# Their delta does NOT isolate caching or measure a theoretical ceiling.
# Start the server with VLLM_SERVER_DEV_MODE=1 for reset_prefix_cache.
# For a causal cache test, replay IDENTICAL saved/tokenized requests and
# arrival times against cache-enabled and cache-disabled server configurations.
vllm bench serve --model $MODEL --dataset-name random \
  --random-input-len 8192 --random-output-len 128 \
  --request-rate 6 --num-prompts 2000 --seed 1 \
  --save-result --result-filename eta0.json

curl -X POST localhost:8000/reset_prefix_cache      # needs VLLM_SERVER_DEV_MODE=1

vllm bench serve --model $MODEL --dataset-name prefix_repetition \
  --request-rate 6 --num-prompts 2000 --seed 1 \
  --save-result --result-filename etahigh.json

# Do not decide from this unmatched pair; run the matched cache-on/off test
# Keep exact tokenized inputs, output limits, arrivals and quality checks fixed.

# Run these servers SEQUENTIALLY on the same GPUs, stopping one first.
# One client, both servers, matched semantics, open loop, n >= 3.
# Cross-harness numbers are NOT comparable - see section 10.2.
VLLM_SERVER_DEV_MODE=1 vllm serve $MODEL \
  --max-num-seqs 256 --max-num-batched-tokens 8192 --enable-prefix-caching

python -m sglang.launch_server --model-path $MODEL \
  --max-running-requests 256 --chunked-prefill-size 8192 \
  --schedule-policy lpm --port 30000

# Then point ONE harness at each in turn, with identical arrival schedule
# and seed, and sweep request-rate to get a curve rather than a point.

The full protocol is in honest benchmarking and Lab 11. The unmatched dataset runs above are exploratory only. For cache attribution, use a two-by-two design: engine A/B crossed with cache on/off, replaying the same tokenized requests, output constraints and intended arrivals. Start each arm from a documented empty state, then apply the same warmup; verify the resolved cache flag, actual hit counts and quality. Repeat paired runs in alternating order, report achieved savings and SLO goodput with uncertainty, and avoid calling a measured delta a theoretical ceiling.

• §13.1 is the comparison matrix this chapter deliberately does not duplicate: one row per design decision, with the "why they chose this" column that makes the tradeoffs legible. Read it before running the procedure here, not instead of it.
• §10.1 for measuring $\eta$ and the token histogram that feeds $\phi$; §10.2 for why the two harnesses' numbers cannot share a table; §10.3 for the protocol; §10.5 for turning a curve into cost per million tokens, which is the input to the hosted-API comparison in §9.
• §2.3 and §2.4 for the two cache mechanisms, and §1.5 for the prefill-admission machinery whose knobs §4 compares.
• §11.5 and §12.4 for the extension seams behind the team-capacity section, including what each project explicitly refuses to make extensible.
• llama.cpp server README, TensorRT-LLM overview, and the TGI index — the three documents behind §9. Fetch them yourself; all three moved in the last year and the TGI one in particular changed what it recommends.
• The pinned trees themselves are the reference for Q1 and Q2: vllm/model_executor/models/registry.py, python/sglang/srt/models/registry.py, vllm/platforms/, and python/sglang/srt/hardware_backend/. Every count in the table in §5 is reproducible with ls and grep, and you should reproduce it at your own SHA rather than trusting mine.