What to build next

Your first pull request against either project will most likely fail in a way that has nothing to do with your code. In SGLang, nothing happens at all:

We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests.
Users with permission are listed in the [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json)

**PR authors** can use `/rerun-failed-ci` on their own PRs even if they are not listed in `CI_PERMISSIONS.json`. Selective reruns have additional rules because they execute PR code on self-hosted runners; see the permission table below.

For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands:

No label, no CI, no signal, and the label is applied by a list of people you are not on. vLLM's version of the same wall is quieter:

- Note that not all CI checks will be executed due to limited computational
  resources. Reviewers with write access and configured trusted contributors
  can comment `/ci run` for upstream CI or `/amd-ci run` for AMD CI only when
  CI signals are needed before a PR is ready. After the PR is approved or has
  the `ready` label, the PR author can use `/ci run`, `/ci retry`, `/ci cancel`,
  or the corresponding `/amd-ci` variants. New commits do not start upstream
  CI automatically.
...
vLLM uses GitHub's [pull request limit](https://github.blog/open-source/maintainers/how-pull-request-limits-are-cutting-down-the-noise/)
for contributors without write access. The current cap is 6 open PRs. If this
blocks well-intentioned critical work, contact a committer to request bypass
list access.

Six open PRs, and pushing a commit does not start CI. Both projects merge dozens of changes a day and both have decided that contributor enthusiasm is cheaper than GPU minutes. That is the first thing a finished reader gets wrong: treating the engineering as the hard part.

The second thing is worse, because it takes eight weeks to show up. You build your feature out of tree, against a file you understand well, and one morning it no longer imports. Count the damage before you start. In the twenty-nine weeks ending at the pinned SHA, vllm/v1/attention/backends/ absorbed 384 commits and vllm/v1/core/sched/interface.py absorbed seven. Same repository, same window, a factor of 55. Which of those two your project sits on top of matters more than how clever it is.

Rungs on this ladder are not graded by lines of code. They are graded by three things that all point the same way: how many processes your change spans (one is a weekend, three with a network in between is a month — §11.2 and §12.1 drew those boundaries), how fast upstream moves under the files you touch, and whether a registered seam exists or you have to fork. A project that stays inside one process, sits on a slow file, and registers through a real seam is a weekend. Break any one of the three and it is a week. Break all three and it is a quarter, whether you planned for one or not.

Define the quantity the ladder is really measuring. Let $F$ be the set of upstream files your project depends on textually — the ones whose contents you read, subclass, or call into. Let $r$ be the rate at which upstream commits touch $F$, in commits per week. Let $T$ be the duration of your project in weeks. Then the number of upstream changes you will have to reconcile before you can merge or ship is

$R$ is rebase debt: commits accrued under you while you were working. It is an upper bound on conflicts, not a count of them — most commits touching a file will not collide with your change — but it is the right hazard rate, because you have to read every one of them to know which did.

Both $r$ terms are measurable from the checkouts you already have. Counting the twenty-nine weeks from 1 February 2026 to the pinned commit dates (2026-08-21 in both repositories):

# whole repo, then one path, over the same window
git log --since=2026-02-01 --oneline | wc -l
git log --since=2026-02-01 --oneline -- vllm/v1/attention/backends | wc -l
git log --since=2026-02-01 --oneline -- vllm/v1/core/sched/interface.py | wc -l

The scheduler implementation changed 115 times and its interface seven times in the cited window, giving a roughly 16.4 ratio of file-touch rates. Under the toy $R=rT$ model a month produces about 17 versus 1 touches. This is not a 16-fold maintenance result: an interface implementation still depends on method semantics, SchedulerOutput layouts, ownership, async behavior and tests. Include those dependencies in the comparison, and do not treat a non-public ABC as a stability promise.

class SchedulerInterface(ABC):
    @abstractmethod
    def __init__(
        self,
        vllm_config: "VllmConfig",
        kv_cache_config: "KVCacheConfig",
        structured_output_manager: "StructuredOutputManager",
        block_size: int,
        hash_block_size: int,
        mm_registry: MultiModalRegistry = MULTIMODAL_REGISTRY,
        include_finished_set: bool = False,
        log_stats: bool = False,
    ) -> None:
        raise NotImplementedError

    @abstractmethod
    def schedule(self, throttle_prefills: bool = False) -> "SchedulerOutput":

The abstract base is not a courtesy. It is a rate limiter on your rebase debt, and it is the reason the ladder's rungs are where they are. The same shape holds on the SGLang side: python/sglang/srt/mem_cache/base_prefix_cache.py took 39 commits while the attention layer directory took 660.

Four projects that edit no upstream file. Direct patch-rebase work may be zero, but they still depend on APIs, schemas, metric semantics, packaging and runtime behavior, so maintenance risk is not zero. Each replaces a claim with a measured number or independently checked diff.

Worked trace: where the instrumentation hooks already are

The first project is worth walking end to end, because most of it is already written and the interesting work is arithmetic rather than plumbing. Ask vLLM to profile without configuring it and it tells you the exact incantation:

    def profile(self, is_start: bool = True, profile_prefix: str | None = None):
        # Check if profiling is enabled
        if self.profiler_config is None or self.profiler_config.profiler is None:
            raise RuntimeError(
                "Profiling is not enabled. Please set --profiler-config to enable "
                "profiling. Example: "
                "'--profiler-config.profiler=torch --profiler-config.torch_profiler_dir"
                "=YOUR_DIR_PATH_TO_DUMP_TRACE'"
            )

From there to a trace is four hops in the worker process: Worker.profile() builds a TorchProfilerWrapper; the engine's busy loop calls Worker.execute_model(), which wraps the model call in annotate_profile(); that helper classifies the iteration's requests via compute_iteration_details() in vllm/v1/utils.py; and it opens an annotation range around self.model_runner.execute_model():

    def annotate_profile(self, scheduler_output):
        # add trace annotation so that we can easily distinguish
        # context/generation request numbers in each iteration.
        # A context request is a request that has not yet generated any tokens
        if not self.profiler:
            return nullcontext()

        self.profiler.step()
        if not self.profiler.is_running:
            return nullcontext()
# ...
        with self.annotate_profile(scheduler_output):
            output = self.model_runner.execute_model(
                scheduler_output, intermediate_tensors

Turn on one flag and the annotation stops being a label and becomes a roofline measurement:

    detailed_trace_annotation: bool = False
    """If `True`, uses detailed annotations with roofline metrics (sk, sqsq,
    sqsk) in profiler trace events. If `False`, uses simple annotations with
    only context/generation request counts and token counts.
    Disabled by default."""

        if self.vllm_config.profiler_config.detailed_trace_annotation:
            # Compute roofline-model metrics per request, split by phase
            # (context vs generation). These help estimate compute and
            # memory intensity from the trace.
            #
            # Per-request quantities:
            #   query_len = number of scheduled (new) tokens for this request
            #   seq_len   = total sequence length (computed + scheduled tokens)
            #
            # Aggregated across requests in each phase
            # (ctx_=context, gen_=generation):
            #   seq_len_sum = sum of seq_len   (total KV length)
            #   qq_compute  = sum of query_len*query_len
            #                 (proxy for QK^T compute cost)
            #   qk_compute  = sum of query_len*seq_len
            #                 (proxy for QK^T compute cost for decode and
            #                  chunked prefill)
            #   total_scheduled_tokens = scheduled tokens across all requests

That comment is the arithmetic-intensity model of §0.4, computed per iteration by the engine itself: seq_len_sum is the total KV length the step will read, qk_compute is $\sum_i q_i s_i$, the proxy for $QK^\top$ work, and the split into ctx_ and gen_ is exactly the prefill/decode split of §1.1. Two more fields make the capture surgical rather than a firehose: delay_iterations and max_iterations in the same config (vllm/config/profiler.py:L118-L123) let you skip past warmup and keep exactly one step. Note that the hook is at the worker level, so it wraps whichever model runner the config selected — vllm/v1/worker/gpu_worker.py:L424-L438 instantiates V2 from vllm/v1/worker/gpu/model_runner.py or V1 from vllm/v1/worker/gpu_model_runner.py, both classes named GPUModelRunner. §13.1 warns about exactly that name collision; the profiling project is unaffected by it, and the next one is not.

SGLang puts the same capability behind a very different seam — an admin HTTP route rather than a launch-time config object:

class ProfileReq(BaseReq, kw_only=True):
    req_type: ProfileReqType = ProfileReqType.START_PROFILE
    # The output directory
    output_dir: Optional[str] = None
    # Specify the steps to start the profiling
    start_step: Optional[int] = None
    # If set, it profile as many as this number of steps.
    # If it is set, profiling is automatically stopped after this step, and
    # the caller doesn't need to run stop_profile.
    num_steps: Optional[int] = None
    # The activities to record. The choices are ["CPU", "GPU", "MEM", "RPD"]
    activities: Optional[List[str]] = None
    # Whether profile by stages (e.g., prefill and decode) separately
    profile_by_stage: bool = False
    # Whether to record source information (file and line number) for the ops.

POST /start_profile with start_step and num_steps does at runtime what vLLM asks you to decide before the process starts, and profile_by_stage splits prefill from decode for you. The tradeoff recurs throughout this book: vLLM puts the knob in a typed configuration surface covered by a deprecation policy, SGLang puts it on a live endpoint you can curl. Neither is wrong.

How you know you got the other three right

A weekend project with no oracle is a weekend wasted, and all three of the remaining ones have an oracle sitting in the trees.

The allocator. SGLang's scheduler ships a conservation check over its pools. Make your toy allocator satisfy the same equation after every operation and most of the bugs that matter become impossible:

    @staticmethod
    def _check_pool_invariant(
        pool_name: str,
        available: int,
        evictable: int,
        protected: int,
        session_held: int,
        total: int,
        uncached: int = 0,
    ) -> Tuple[bool, str]:
        """Check: available + evictable + protected + session_held + uncached == total."""
        total_accounted = available + evictable + protected + session_held + uncached
        leak = total_accounted != total
        msg = (
            f"[{pool_name}] {total=}, {available=}, {evictable=}, "
            f"{protected=}, {session_held=}, {uncached=}"
        )
        return leak, msg

For eviction order, vLLM's own unit test is a ready-made specification. It allocates 6 then 3 blocks out of a ten-block usable pool, frees both requests, and asserts the exact free-list contents — partial blocks at the head, because they carry no hash and are worthless to a future prefix hit:

    # 10 - (6 + 3) == 1
    assert manager.block_pool.free_block_queue.num_free_blocks == 1

    manager.free(req0)
    # partial blocks (without hash) at head, other at tail (LRU policy):
    assert [
        b.block_id for b in manager.block_pool.free_block_queue.get_all_free_blocks()
    ] == [6, 10, 5, 4, 3, 2, 1]
    manager.free(req1)
    assert manager.block_pool.free_block_queue.num_free_blocks == 10
    assert [
        b.block_id for b in manager.block_pool.free_block_queue.get_all_free_blocks()
    ] == [6, 10, 5, 4, 3, 2, 1, 9, 8, 7]

Copy those assertions verbatim against your own allocator. If your free list comes back in block-id order rather than [6, 10, 5, 4, 3, 2, 1], you have built an allocator and not a cache — §2.2 explains what that ordering is protecting.

Online softmax. The oracle is not torch.softmax — it is the rescale that both engines use to merge two independently-softmaxed chunks. vLLM's version, from the kernel that stitches a prefix-cache hit onto a fresh suffix:

  const float max_lse = fmaxf(p_lse, s_lse);
// ...

  p_lse = p_lse - max_lse;
  s_lse = s_lse - max_lse;
  const float p_se = expf(p_lse);
  const float s_se = expf(s_lse);
  const float out_se = p_se + s_se;
  const float p_scale = p_se / out_se;

That is the whole idea: subtract the joint maximum from both log-sum-exps, exponentiate, and take the convex combination. Then test yours against the edge case the vLLM authors had to fix, documented immediately above at csrc/libtorch_stable/attention/merge_attn_states.cu:L108-L115 — with chunked prefill and no prefix hit, both LSEs can be $-\infty$ at once, and continuing the arithmetic yields NaN instead of the zero-attention answer. Reproduce it before you handle it.

The hit-rate harness. The oracle here is a disagreement. vLLM exports two monotonic Prometheus counters, vllm:prefix_cache_queries and vllm:prefix_cache_hits, both in tokens, and you difference them yourself. Requests that were preempted and re-prefilled are kept out of both, in separate fields:

    def record(self, num_tokens: int, num_hits: int, preempted: bool) -> None:
        """Aggregate request information into the stats."""
        if preempted:
            # Previously preempted request
            self.preempted_requests += 1
            self.preempted_queries += num_tokens
            self.preempted_hits += num_hits
        else:
            # New request
            self.requests += 1
            self.queries += num_tokens
            self.hits += num_hits

SGLang reaches the same policy by subtraction rather than segregation, and reports a windowed rate rather than a counter:

            effective_input_tokens = (
                prefill_stats.log_input_tokens
                - prefill_stats.reprocessed_log_input_tokens
            )
            effective_hit_tokens = (
                prefill_stats.log_hit_tokens - prefill_stats.reprocessed_log_hit_tokens
            )
            total_tokens = effective_input_tokens + effective_hit_tokens
            cache_hit_rate = (
                effective_hit_tokens / total_tokens if total_tokens > 0 else 0.0
            )
            self.recent_cache_hit_rate = self.cache_hit_rate_window.add(
                effective_hit_tokens,
                total_tokens,

The comment that pins the semantics sits at the accounting site rather than here: "reprocessed_log_* is a subset of log_*; metrics_reporter subtracts it when computing the first-attempt prefix cache hit rate" (python/sglang/srt/managers/schedule_policy.py:L905-L911). Both engines therefore report a first-attempt hit rate, but one is a monotonic token counter you differentiate yourself and the other is a pre-smoothed windowed ratio. Plot them on one axis without saying which is which and you have produced a chart of nothing. That is the lesson of the weekend, and §13.1 generalises it: two systems agreeing on a metric's name is not evidence that they agree on the metric.

A week buys you one real seam, used properly, with a test a reviewer would accept. §11.5 and §12.4 inventory all of them; what follows is which four are worth a week and what each teaches that reading cannot.

A custom attention backend

vLLM's registry takes an enum member and a dotted path, and the docstring is the specification:

    Examples:
        # Override an existing attention backend
        @register_backend(AttentionBackendEnum.FLASH_ATTN)
        class MyCustomFlashAttn:
            ...

        # Override an existing mamba attention backend
        @register_backend(MambaAttentionBackendEnum.LINEAR, is_mamba=True)
        class MyCustomMambaAttn:
            ...

Registering is one call. The week goes on everything else: the metadata builder that turns a SchedulerOutput into the layout your kernel wants, the CUDA-graph compatibility declaration, and the fact that §3.4's block table is not what most published kernels expect. SGLang's equivalent seam is two registrations, not one — @register_attention_backend(name) plus add_attention_backend_choices, because --attention-backend's argparse choices list is a separate module constant; miss the second and argparse rejects you before the engine starts (§12.4).

How you know you got it right: vLLM's own registry test is 169 lines and registers a dummy backend by class path, then asserts the enum reports itself overridden and resolves to your class. Copy its shape.

def test_register_custom_backend_with_class_path():
    # Register with explicit class path
    register_backend(
        backend=AttentionBackendEnum.CUSTOM,
        class_path="tests.test_attention_backend_registry.CustomAttentionBackend",
        is_mamba=False,
    )

    # Check that CUSTOM backend is registered
    assert AttentionBackendEnum.CUSTOM.is_overridden(), (
        "CUSTOM should be overridden after registration"
    )

    # Get the registered class path
    class_path = AttentionBackendEnum.CUSTOM.get_path()
    assert class_path == "tests.test_attention_backend_registry.CustomAttentionBackend"

Correctness past registration is a numerics problem needing a GPU and a reference: run the same prompts through your backend and through FLASH_ATTN at temperature 0 and diff the logits, not the text — §10.4 explains why text-level diffs will lie to you.

A scheduler policy

vLLM lets you pass a class or a dotted path, and then tells you exactly how much of a guarantee you are getting:

        # The first half of this warning can be removed once the Scheduler interface is
        # finalized and we can maintain support for scheduler classes that implement it
        logger.warning_once(
            "Using custom scheduler class %s. This scheduler interface is not public "
            "and compatibility may not be maintained. If you have subclassed Scheduler "
            "instead of AsyncScheduler, you will see degraded performance due to async "
            "scheduling being disabled.",
            self.scheduler_cls,  # type: ignore[arg-type]
        )
        if not isinstance(self.scheduler_cls, str):
            return cast(type["SchedulerInterface"], self.scheduler_cls)
        return resolve_obj_by_qualname(self.scheduler_cls)

Two warnings in one message: the interface is not public — which is why §3's rebase debt matters here — and subclassing Scheduler rather than AsyncScheduler silently costs you async scheduling. A week is enough to build a priority or fair-share policy over the admission logic of §1.4 and §11.3. It is not enough to keep it working for a year, so plan to upstream it.

SGLang has no equivalent seam, and the inversion is worth stating plainly: SGLang ships more scheduling policies than vLLM and no way to add one, while vLLM ships two and a seam (§13.2 chooses between the built-ins; this is about adding a seventh). --schedule-policy resolves through two enums, CacheAwarePolicy and CacheAgnosticPolicy, dispatched by an if-chain that raises on an unknown name (python/sglang/srt/managers/schedule_policy.py:L200-L215, L254-L288), so a new policy is a patch to that file. The 14.6-commits/week count above belongs to managers/scheduler.py, not schedule_policy.py; measure the actual file before assigning it a rate. The general escape hatch is the hook registry, which can REPLACE any dotted path including Scheduler.schedule; §12.4 covers the mechanism and its collision policy. Treat that as a research tool, not a deployment strategy.

A structured-decoding backend, and a quantization method

These two are the clearest demonstration in the book that neither project set out to design a plugin architecture. For grammars, SGLang has a real registry — register_grammar_backend(name, init_func) — and vLLM has an if-chain. For quantization it is exactly reversed: vLLM has @register_quantization_config with a doctest in its docstring, and SGLang has a module dict plus a single out-of-tree escape via the platform interface (§12.4). Pick the engine by which seam exists, not by preference.

If quantization is the project, SGLang's contribution guide is unusually prescriptive about the shape it wants, and names two merged PRs as the reference refactors:

## Adding or Refactoring a Quantization Method

1. Define the config entry point and register it through `python/sglang/srt/layers/quantization/__init__.py` when needed.
2. Add explicit scheme selection helpers such as `get_linear_scheme` and `get_moe_scheme`.
3. Move layer-specific weight creation and weight loading into scheme classes.
4. Move GPU (CUDA/HIP/XPU), NPU, or other hardware kernel calls into backend kernel modules.
5. Keep Linear, MoE, embedding, and non-linear module handling explicit. Do not assign a Linear quantization method to a module type that needs different semantics.
6. Preserve compatibility for existing quantized checkpoints and runtime flags.
7. Add tests that cover both config parsing and execution paths touched by the change.

For examples, see the AWQ and GPTQ refactors:

- [PR #21126](https://github.com/sgl-project/sglang/pull/21126): splits AWQ schemes, weight initialization, and backend kernel calls.
- [PR #26402](https://github.com/sgl-project/sglang/pull/26402): applies the same scheme/kernel split to GPTQ.

Read PR 21126 and 26402 before writing anything — they are the difference between a method that lands and one that gets a "please split the scheme from the kernel" review. §4.2 covers what those schemes compute; the guide covers where the code goes.

How you know, for both: the accuracy gate. SGLang's guide wants a launched model, a /generate round trip, and an accuracy test sized to the blast radius — and labels its own GSM8K run a sanity check with 1–5% variance, not an accuracy result (docs/docs/developer_guide/contribution_guide.mdx:L84-L102).

A month is what it costs when the change spans processes, or crosses a network, or sits on a seam that is real but young. Three worth doing.

A speculative-decoding proposer

Both engines will load a proposer you wrote, and the two designs are a clean study in opposite bets. vLLM duck-types on one method and reads the class path out of a field that normally names a model:

def create_custom_proposer(vllm_config: VllmConfig):
    """Load and instantiate a user-provided proposer class.

    The class path is read from ``speculative_config.model``
    (e.g., ``"my_module.MyCustomProposer"``).  The class is
    imported, instantiated with *vllm_config*, and returned
    directly so the caller can use it without any wrapper.

    The returned object must expose a callable ``propose`` method.
    """

The dispatch is the first branch in the model runner's drafter selection, so a custom class shortcuts every built-in method:

            )
            if self.speculative_config.method == "custom_class":
                self.drafter = create_custom_proposer(  # type: ignore[assignment]
                    self.vllm_config
                )

That branch is in the V1 runner. The V2 runner — the default for the architectures vllm/config/vllm.py:L649-L699 designates, which is where §13.1's name-collision warning bites — builds its speculator from a separate factory that does not know the method exists:

    elif speculative_config.use_eagle():
        from vllm.v1.worker.gpu.spec_decode.eagle.speculator import (
            EagleSpeculator,
        )

        return EagleSpeculator(vllm_config, device)
    else:
        raise NotImplementedError(f"{speculative_config.method} is not supported yet.")

SGLang's register_algorithm (python/sglang/srt/speculative/spec_registry.py:L222-L243) takes the opposite bet: a registry that refuses duplicate and reserved names and reflects over its own enum to raise TypeError if your class is missing an is_* or supports_* method that the scheduler branches on — a guard added after two such methods went missing silently (§12.4). vLLM checks that propose exists and is callable, and nothing else. So the vLLM seam is faster to satisfy and fails at the first bad forward pass; the SGLang seam is slower and fails at import. If you have ever debugged an acceptance rate that was quietly zero, you know which failure you would rather have.

What it teaches: not the proof from §6.2 — the engines implement rejection sampling for you — but the bookkeeping of §6.4: KV slots for tokens that may be rejected, position IDs for a tree, a draft that must not stall the batch. How you know: acceptance length plus a greedy-equivalence check (Lab 08). Exact greedy token equality is a strong oracle only when target logits, argmax tie rules and numerical paths are matched. Different batch/kernel paths can perturb near-tied logits. Check the target top-two margin and logit error, then distinguish a bookkeeping defect from numerical-path divergence; distributional speculative sampling needs separate tests.

A KV connector for prefill–decode disaggregation

The highest-leverage month-sized project in either tree: §1.6's architecture is only as good as the transport under it, and the transport is genuinely pluggable:

    @classmethod
    def register_connector(cls, name: str, module_path: str, class_name: str) -> None:
        """Register a connector with a lazy-loading module and class name."""
        if name in cls._registry:
            raise ValueError(f"Connector '{name}' is already registered.")

        def loader() -> type[KVConnectorBase]:
            module = importlib.import_module(module_path)
            return getattr(module, class_name)

        cls._registry[name] = loader

A lazy loader keyed by name, with a hard error on duplicates. The interface is KVConnectorBase_V1, which §11.5 walks. It is a month rather than a week because a connector has two halves in two processes — a scheduler-role half that decides what to send and a worker-role half that moves bytes — and they must agree about block identity while the scheduler is free to preempt underneath them. Note the churn number from §3: the connector base moved 12 times in 29 weeks, one of the slowest files in either repo. The Python interface has settled even though everything built on it is new — which is not the same as the wire contract having settled, and §13.3 is the chapter on how far it has not.

How you know you got it right: a leak check first — run the workload, drain it, and assert the block pools return to their initial free counts, using the invariant quoted in §4. Output equality is the weaker test than it looks: upstream's own P/D accuracy gate had to be widened to absorb kernel-path divergence across the boundary, which §13.3 documents. Demand bit-equality only where both sides run the same kernels; otherwise you have inherited §13.3's accuracy-tolerance problem along with the project.

A cache-aware routing policy

SGLang's router is Rust, and the seam is a trait:

#[async_trait]
pub trait LoadBalancingPolicy: Send + Sync + Debug {
    /// Select a single worker from the available workers
    ///
    /// This is used for regular routing mode where requests go to a single worker.
    /// Now uses Arc<dyn Worker> for better performance and to avoid unnecessary cloning.
    ///
    /// # Arguments
    /// * `workers` - Available workers to select from
    /// * `info` - Additional information for routing decisions
    async fn select_worker(
        &self,
        workers: &[Arc<dyn Worker>],
        info: &SelectWorkerInfo<'_>,
    ) -> Option<usize>;

Implementing it is a week. What makes it a month is that construction goes through a match on a config enum, not a registry:

impl PolicyFactory {
    /// Create a policy from configuration
    pub fn create_from_config(config: &PolicyConfig) -> Arc<dyn LoadBalancingPolicy> {
        match config {
            PolicyConfig::Random => Arc::new(RandomPolicy::new()),
            PolicyConfig::RoundRobin => Arc::new(RoundRobinPolicy::new()),
            PolicyConfig::PowerOfTwo { .. } => Arc::new(PowerOfTwoPolicy::new()),
            PolicyConfig::CacheAware {

So a new policy is upstream work by construction: a variant on PolicyConfig, an arm here, and a PR. Before writing one, read the existing cache-aware policy's header, which states its algorithm and its two-regime switch in twelve lines:

    1. Cache-Aware Routing (Approximate Tree)
    2. Load Balancing (Shortest Queue with Balance Thresholds)

    The router dynamically switches between these strategies based on load conditions:
    - Uses load balancing when the system is imbalanced
    - Uses cache-aware routing when the system is balanced

    A system is considered imbalanced if both conditions are met:
    1. (max - min) > abs_threshold
    2. max > rel_threshold * min

Two details there are worth the month. The tree "stores raw text characters instead of token IDs to avoid tokenization overhead" (sgl-model-gateway/src/policies/cache_aware.rs:L21-L23): the router deliberately approximates the radix structure of §2.4 rather than querying it. And it abandons cache affinity entirely when load is imbalanced, on a two-condition test. Both are decisions §9.4 can describe but only an implementation forces you to justify.

Five more, already scoped for you

§13.3 labels each open problem it raises either "engineering work remains" or "no one knows how". The first label is a rung assignment, and five of them are month-sized or smaller: parallelising the NIXL handshake and adding retry-across-replicas when a KV transfer fails (§13.3); threading device_id through backend selection, which §13.3 calls tedious, mechanical, uncontroversial and unwritten (§13.3); chunking the block-hash chain off the hot path (§13.3); and building the joint throughput / SLO-attainment / accuracy-delta harness that nobody has built because it is expensive to run and unflattering to everyone (§13.3). Read that chapter for the evidence; the ladder here is only telling you what each would cost.

Two shapes, and they are less different than they look.

A serious upstream contribution. Past roughly 500 lines, vLLM requires a design document before the code: "For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR" (docs/contributing/README.md:L277-L281). That gate is the honest definition of the rung: a quarter-sized change is one whose design is reviewed, so the writing is a third of the work and the arguing another third. Split the series so each PR is independently useful — landing early is the only lever you have against $R \approx 172$ on a hot directory.

A domain-specific engine for one model family. Not a vLLM competitor: one architecture, one hardware target, with the scheduler, allocator, and attention kernel you now know how to write. It is the best way to find out which of the two engines' complexity is essential and which is generality tax. SGLang's contribution guide points newcomers at a reference implementation of this shape, mini-sglang, "for a quick overview on the structure of sglang" (docs/docs/developer_guide/contribution_guide.mdx:L237-L239). Read it, then write your own rather than extending it — the value is entirely in the decisions you are forced to make.

Scope it by what you are allowed to not support. One model family means one attention shape, so no backend abstraction. One hardware target means no platform layer. Greedy and top-p only means no sampler zoo. Fixed max context means a static KV pool. What is left after those deletions — continuous batching, paged KV, a prefix cache, chunked prefill, CUDA graphs — is the irreducible core this book has been describing, and it is roughly 3,000 lines. §13.1 maps which deletions you will regret first.

The mechanics, read out of the trees rather than remembered. §11.5 and §12.4 cover the per-seam test layout; this is the part common to any change.

The one that catches people is CI job selection, because in both projects a test that is not reachable from your diff never runs. vLLM declares the mapping per job:

- label: ":nvidia: (H200) V1 Spec Decode"
  device: h200_35gb
  key: v1-spec-decode
  timeout_in_minutes: 40
  source_file_dependencies:
    - vllm/config/
    - vllm/distributed/
    - vllm/inputs/
    - vllm/model_executor/
    - vllm/platforms/
    - vllm/sampling_params.py
    - vllm/transformers_utils/
    - vllm/utils/
    - vllm/v1/
    - tests/v1/spec_decode
  commands:
    - export VLLM_WORKER_MULTIPROC_METHOD=spawn
    # TODO: create another `optional` test group for slow tests
    - pytest -v -s -m 'not slow_test' v1/spec_decode

source_file_dependencies is the gate: this 40-minute H200 job runs only if your diff touches one of those paths. Put a spec-decode test somewhere else and it will never execute in CI, and a reviewer will find that out for you. SGLang inverts the direction — the test file declares its own cost and placement, and the runner parses the declaration statically:

## CI Registration

Every CI-discovered test file must call a registration function at module level:

```python
from sglang.test.ci.ci_register import register_cuda_ci

register_cuda_ci(est_time=80, stage="base-b", runner_config="1-gpu-small")
```

Parameters: `est_time` (seconds), `stage` + `runner_config` (target stage and runner pool from `scripts/ci/runner_configs.yml`), `nightly=True` (nightly-only), `disabled="reason"` (temporarily disable).

Keep `est_time`, `stage`, `runner_config` as **literal values** — `run_suite.py` collects them by AST parsing.

Because run_suite.py reads those by AST, they must be literals. The other half of the rule is choosing the cheapest suite that can host your test: base-a-test-cpu runs in stage A, the ~3-minute pre-flight, so a registry test with no GPU dependency belongs there rather than in a nightly (test/README.md:L10, L79-L92).

Two more asymmetries worth knowing before you start. vLLM has a written policy on what it is allowed to break, and the scope is explicit:

Features that fall under this policy include (at a minimum) the following:

- CLI flags
- Environment variables
- Configuration files
- APIs in the OpenAI-compatible API server
- Public Python APIs for the `vllm` library

Read that as a promise and as an exclusion: those five surfaces move through a staged pipeline tied to minor releases. SchedulerInterface, the attention backend protocol, and the KV connector base are not on the list — which is exactly what the "this scheduler interface is not public" warning in §5 was telling you. I could not find a comparable policy document anywhere under SGLang's docs/ or .github/ at 7d89325; the nearest thing is the code-style guidance plus a release doc covering only PyPI mechanics.

vLLM also has an explicit policy on AI-assisted contributions: no "pure agent" PRs, disclose the assistance in the description, attribute with a Co-authored-by: trailer alongside the DCO sign-off (docs/contributing/README.md:L198-L221). Given how this book was assembled, it would be dishonest not to point at it.

Ordered by what they change about how you read code, not chronologically. Where a venue is marked unverified, I could not confirm it from a primary source this session and you should not cite it from here.

If you read three, read Orca, PagedAttention, and FlashAttention-2 — the scheduler, the allocator, and the kernel, which between them account for most of what both engines are.

One widely cited item is deliberately annotated rather than recommended. The Anyscale post How continuous batching enables 23x throughput in LLM inference while reducing p50 latency (Daniel, Shen, Liang, Liaw, June 2023) is where most people first met the term, and it is where the "23×" number comes from. The post's own wording attributes that figure to continuous batching plus PagedAttention against a naive static-batching baseline, in its highest-variance output-length setting. Cite Orca for the mechanism and this post for the terminology, and do not repeat the number without its configuration — §10.3 explains what goes wrong when you do.

Two exclusions are deliberate: benchmark blog posts from either project, for the reasons §10.3 gives; and survey papers on LLM inference, which are accurate and useless, because every claim in them points at one of the papers above and none will tell you where the block table lives.

And the best documentation these projects have is not documentation. vLLM's docs/design/ holds twenty-nine short notes written by the people who did the work, each closer to a paper than to a README. SGLang's equivalent is the PR discussion; the further-reading section below says how to find the load-bearing ones.

Every citation in this book names a file and a line range at a556f3f or 7d89325. Line ranges rot fastest, paths next, concepts slowest. The recipe for re-deriving a claim at a newer commit, in the order to try it:

# 1. does the symbol still exist, and where?
git grep -n 'def annotate_profile'

# 2. when did the line I cited change, and to what?
git log -L 931,940:vllm/v1/worker/gpu_worker.py --oneline

# 3. who introduced or removed this identifier, across renames?
git log -S 'detailed_trace_annotation' --oneline -- vllm/

# 4. did the file move?
git log --follow --oneline -- vllm/v1/core/sched/interface.py

git log -S finds changes in occurrence counts of a string; it can miss same-count rewrites and should not be described as surviving every move or edit. Combine it with git log -G for matching changed lines, git log --follow for a single renamed file and git blame -M -C for moved/copied lines. Inspect the actual diff and PR rationale rather than inferring causality from the search result.

Churn rate is the axis here, and it is not the same as risk. SGLang's base_prefix_cache.py (39) and radix_cache.py (38) move at the same rate, but the first is a contract that other implementations must satisfy and the second is one of those implementations, so a change to the first is far more likely to reach you. Read the tier as "how often do I need to look", and the contract/implementation distinction as "how much does a change hurt".

The pattern generalises past these two repositories. The contracts are stable because changing them breaks every implementation at once: the scheduler's schedule-then-update-from-output cycle, the KV cache's block-table indirection, the connector's two-role split, the prefix cache's insert/match/evict triple. The implementations churn because that is where hardware generations land. Kernels and attention backends are the extreme case — three of the four entries in the hot rim above are kernel dispatch — and any claim in this book about which backend is selected by default, on which SM version, is the first thing to go stale.

The book's own worked example of a claim that will rot is §13.2's version-stamped callout on quantization-format support, which tells you outright to re-check it at your own SHA before it decides anything. Treat every table in Part 13 that way. Per release: skim the two contribution docs for changed CI rules, diff vllm/config/ and python/sglang/srt/server_args.py for added and removed flags, re-run the churn count from §3 on the directory your project depends on. Ten minutes, and it is the difference between a fork that rebases and one that gets rewritten.

The seam that exists but is not wired up. The canonical version is SGLang's attention registry: your factory registers fine and the engine still refuses to start, because --attention-backend's argparse choices is a separate list you also have to extend. The error arrives from argparse, names your backend, and mentions nothing about registration. Generalise the lesson: after registering anything, grep for other module-level constants that mention the same names.

The process you forgot about. Registries are per-process dictionaries, and both engines load plugins independently in the API server, the engine core, and each worker. A registration performed in one process is invisible in the others, and the symptom is a feature that works with multiprocessing disabled and vanishes with it on. vLLM's own scheduler-plugin test sets VLLM_ENABLE_V1_MULTIPROCESSING=0 for exactly this reason (§11.5); that flag is a debugging aid, not a fix.

The per-layer Python predicate. SGLang's style guide is blunt about the cost model you are writing into: "SGLang is a runtime, and most of your code runs on the critical path for every request… Please cache the result as a single boolean value in __init__" (docs/docs/developer_guide/contribution_guide.mdx:L163-L166). The same document asks you to minimise tensor.item() and tensor.cpu(), and the second matters far more: an attribute lookup costs tens of nanoseconds, a device synchronisation tens of microseconds. Against the 4.48 ms Llama-3-8B decode floor of §0.4, one stray .item() per layer is a percent of the step and the predicates are noise. Fix the syncs first.

The benchmark without its configuration. SGLang's quantization guide asks for the exact model, quantization flag, backend flag, hardware, and command in the PR description (docs/docs/developer_guide/quantization_contribution_guide.mdx:L79-L83); a number without them is not evidence.

Thirty minutes, no GPU, and it decides which rung you start on. Clone both repos at the pinned SHAs, then run the churn count for the subsystem you care most about and the smallest test suite that touches it:

# 1. rebase debt for your candidate file set, over the last 6 months
git log --since=2026-02-01 --oneline -- <your path> | wc -l

# 2. vLLM: which CI job would your change trigger?
grep -rn "source_file_dependencies" -A 12 .buildkite/test_areas/ | grep -B 6 "<your path>"

# 3. vLLM: the cheapest suite that proves a seam is reached
pytest -s -v tests/test_attention_backend_registry.py

# 4. SGLang: the CPU pre-flight stage, which gates every PR
python3 test/run_suite.py --hw cpu --suite base-a-test-cpu

If step 1 returns a number above ~50 for a project you were budgeting a month for, pick a different file set or plan to upstream in pieces. If step 3 passes, your environment can host the week-sized attention-backend project without a GPU up to the point where numerics start to matter. Then open the labs — Lab 10 for the weekend rung, Lab 04 and Lab 08 for the projects above.

Record the pinned runtime/model/tokenizer revisions, hardware and maximum experiment budget, baseline, exact workload, quality oracle and target metric. For a performance project, success means an improvement with matched inputs and uncertainty reported, while correctness and the agreed tail-latency or goodput threshold remain satisfied. A negative or inconclusive result is valid when the raw evidence and limitations are preserved.

CPU-only deliverables can validate a ragged index map, stable online-softmax recurrence, request/slot ownership model, grammar state machine or benchmark event parser. They cannot certify CUDA timing, collective execution, graph replay or cross-node KV transport. For GPU work, add a bounded smoke test before a load sweep and a rollback procedure before exposing production traffic.

Use independent negative controls: duplicate allocation, stale request generation, write-after-release, a missing component predicate and a plugin initialization exception. Equal allocation/free totals are insufficient: allocating slot 7 twice and leaking slot 8 can balance counts while ownership is wrong. Check unique live writers, legal shared readers and eventual release by identity. Re-run compatibility tests on every upgrade, even for historically quiet non-public interfaces.

In-tree, and better than most blog posts. From vLLM's docs/design/: paged_attention.md, prefix_caching.md, plugin_system.md, hybrid_kv_cache_manager.md, fused_moe_modular_kernel.md. From SGLang's docs/docs/developer_guide/: contribution_guide.mdx, quantization_contribution_guide.mdx, benchmark_and_profiling.mdx, serve_backend_plugins.mdx — plus test/README.md, the only readable description of the CI system either project has.

Pull requests worth reading in full. SGLang PR 21126 and 26402, named by the quantization guide quoted in §5, are the reference refactors for separating a quantization scheme from its kernel — read them before adding a method. vLLM PR 16899 is pointed at from vllm/v1/spec_decode/llm_base_proposer.py:L1850-L1853, where a comment records that draft-token sampling is still argmax-only "until we find a way to manage the draft prob tensor" — which is a live, well-scoped problem and an unusually clear invitation. vLLM PR 31811 is cited as WIP at vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py:L391-L395 for a generic KV-event implementation across stacked connectors — directly adjacent to the month-sized connector project.

Finding more of them. Both trees annotate hard-won decisions with the PR that made them. git grep -n "vllm-project/vllm/pull" vllm/ and the corresponding git grep -n "sgl-project/sglang/pull" python/ return a few dozen each, every one attached to the line of code it explains. That is a better reading list than any curation, because it is sorted by what actually bit someone.

Job boards. vLLM's contributing guide points at its good-first-issue and new-model queries (docs/contributing/README.md:L15-L22); SGLang points at its good-first-issue and help-wanted labels and at mini-sglang (docs/docs/developer_guide/contribution_guide.mdx:L233-L239). A new model is the highest-value first contribution in either project and the one with the clearest acceptance criterion: it either produces the reference outputs or it does not.

Elsewhere in this book. §13.1 for the design comparison these projects sit inside, §13.2 for picking the engine before picking the project, §13.3 for the problems nobody has solved yet, the labs for the measurements, and SOURCES for every citation in the book with its provenance.