The speculative decoding zoo

Here is the default adaptive schedule SGLang uses when you pass --speculative-adaptive and no config file:

DEFAULT_ADAPTIVE_CONFIG: dict[str, dict] = {
    "1": {
        "candidate_steps": [1, 3, 7],
        "up_hysteresis": 0.0,
        "down_hysteresis": -0.25,
        "ceiling_coeff": 0,
    },
    "8": {
        "candidate_steps": [0, 1, 3],
        # ...
    },
    "32": {
        "candidate_steps": [0, 1],
        # ...
    },
    "64": {
        "candidate_steps": [0],
        # ...
    },
}

The keys are batch sizes. The values are the draft lengths $\kappa$ the controller is permitted to choose. At batch 1 it may draft up to seven tokens. At batch 64 the only legal value is zero — and init_states builds each candidate with speculative_num_draft_tokens=steps + 1 (adaptive_runtime_state.py:L103-L107), so $\kappa = 0$ means one draft token, the bonus, which is plain decode. Somebody profiled this, and shipped a default that says stop speculating above batch 32.

That is §6.2's break-even result, in production, as a constant. §6.2 derived $B_{\text{break-even}} = T^{*}(E - \kappa c)/(\kappa+1) \approx 150$ concurrent sequences for Llama-3-8B with a 1B draft on an H100; SGLang's default is more conservative still, which is what you would expect from a schedule that must be safe across models it has never seen.

So the flag surface is not a menu of speedups. It is a menu of ways to move the two numbers §6.2 named — the acceptance rate $\alpha$ and the draft cost ratio $c = t_D/t_T$ — plus a smaller set of ways to change the verification rule itself. At the pinned SHAs, SGLang's --speculative-algorithm accepts seven builtin spellings plus anything a plugin registers, and vLLM's method resolves to thirteen. Nothing published surveys all of them. This chapter does, and flags what it could not establish.

§6.2's step cost is $t_T(1 + \kappa c)$ and its yield is $E = (1-\alpha^{\kappa+1})/(1-\alpha)$ tokens. Every method in this chapter does exactly one of four things to that expression.

Figure 1 uses cost on the horizontal axis and acceptance on the vertical axis. Above the break-even curve, toward higher acceptance and lower cost, speculation pays in the stated batch-1 model; below it, it does not.

Read the figure horizontally, not vertically. The horizontal axis is the part this chapter can establish from source — how many forward passes a method runs per decode step, which is a fact about the code. The vertical axis is the part nobody has published a fair cross-method measurement of.

Take §6.2's configuration: Llama-3-8B target (4.48 ms per decode step at batch 1), Llama-3.2-1B-class draft, $c = 0.165$, $\kappa = 3$, $\alpha = 0.70$, so $E = 2.533$.

Chain drafting. Three sequential draft passes. Step cost $1 + 3(0.165) = 1.495$ target-steps; speedup $2.533/1.495 = \mathbf{1.69\times}$.

Parallel drafting. One draft pass emits all three tokens. Step cost $1 + 0.165 = 1.165$; speedup $2.533/1.165 = \mathbf{2.17\times}$ at the same $\alpha$. Derived. That is a 28% larger speedup for free — except it is not free, because the three drafted positions never saw each other, and $\alpha$ falls.

Two thresholds answer different questions. Beating no speculation requires $E(\alpha,3)>1.165$, giving $\alpha\approx0.142$. Beating the chain above requires $E(\alpha,3)/1.165>2.533/1.495$, giving $\alpha\approx0.536$. At half the chain's acceptance, $\alpha=0.35$, the parallel speedup is only $1.515375/1.165\approx1.301$, below the chain's 1.694. A lower break-even acceptance does not prove superiority to an already useful chain.

Two honesty checks. Substituting $\kappa c \to c$ into §6.2's $B_{\text{break-even}} = T^{*}(E - \kappa c)/(\kappa+1)$ gives $295(2.533 - 0.165)/4 \approx 175$ sequences, up from 150; and $S_\infty = E/((\kappa{+}1) + c) = 2.533/4.165 = 0.61 < 1$. Parallel drafting widens the window in which speculation pays; it does not make speculation pay at high batch. §6.2's $S_\infty < 1$ theorem survives every method here, because it depends only on $E \le \kappa+1$.

Here is the mask-token block that does the parallel draft, in SGLang's DSpark drafter. Column 0 is the last verified token; every other column is a single repeated mask token id, and the whole $[\,\text{bs},\,\gamma\,]$ block goes through one forward:

        draft_block_ids = torch.full(
            (bs, gamma), int(self._mask_token_id), dtype=torch.long, device=device
        )
        draft_block_ids[:, 0].copy_(draft_input.bonus_tokens.view(-1))
        draft_positions = positions_2d[:, :gamma].reshape(-1)
        draft_cache_loc = verify_cache_loc_2d[:, :gamma].reshape(-1)

DFlash's is the same shape (python/sglang/srt/speculative/dflash_worker_v2.py:L1814-L1815: block_ids.fill_(int(self._mask_token_id)) then block_ids[:, 0].copy_(draft_input.bonus_tokens)), and vLLM states the property outright:

        # Unifying eagle, draft model, and parallel drafting support.
        # DFlash always uses parallel drafting (all tokens in one pass),
        # but has an additional slot for the next_token_id (does not shift like EAGLE)

Every speculative method each engine ships at its pinned SHA. "Trained ckpt" means the method needs weights that do not come with the target. "Own CG runner" means a dedicated CUDA-graph runner class, which is the single best proxy for how much bespoke plumbing a method carries.

One asymmetry that Shape column encodes and vLLM's does not: at these SHAs, tree drafting is SGLang's alone. Setting --speculative-eagle-topk above 1 is genuine tree verification, not a relabelled chain — the draft loop expands top-$k$ children per level (select_top_k_tokens, python/sglang/srt/speculative/spec_utils.py:L343-L355), build_tree_kernel_efficient materialises the ancestry mask in one of three packings (python/sglang/srt/speculative/eagle_utils.py:L151-L214), and verify_tree_greedy_func walks it (:L378-L422). vLLM has no counterpart: the only mention of tree speculation anywhere under vllm/v1/spec_decode/ is an unresolved "# FIXME: when using tree-based specdec, adjust number of forward-passes # according to the depth of the tree." above dummy_run (vllm/v1/spec_decode/llm_base_proposer.py:L1645-L1646), and a config field scopes itself to "non-tree speculation" (vllm/config/speculative.py:L147-L151). Medusa is the mirror-image asymmetry: a medusa method, a MedusaProposer and a model implementation exist in vLLM, while a case-insensitive grep for medusa over the entire SGLang checkout returns nothing. The mask machinery itself belongs to §6.4; what matters here is which engine can build one.

vLLM keeps its own version of this table in the source, and it is the cleanest statement of the chain/parallel split either project has written down:

        ==================== ============= ======== ================
        Algorithm            Method        Parallel Additional slots
        ==================== ============= ======== ================
        EAGLE3               eagle3        No       0
        P-EAGLE              eagle3        Yes      K - 1
        DFlash               dflash        Yes      K
        DSpark               dspark        Yes      K - 1
        MTP                  mtp           No       0
        N-gram               ngram         No       0
        Draft model          draft_model   No       1
        PARD                 draft_model   Yes      K
        ==================== ============= ======== ================

Read the Parallel column as a capability, not a state. It is speculative_config.parallel_drafting, which is False by default (vllm/config/speculative.py:L167-L172, "requires the speculative model be trained to support parallel drafting"). __post_init__ flips it on automatically for exactly two methods — if self.method in ("dflash", "dspark"): self.parallel_drafting = True (vllm/config/speculative.py:L1071-L1072). P-EAGLE and PARD are the same eagle3 and draft_model code paths with the flag set by hand, on a checkpoint that was trained for it; point them at an ordinary checkpoint and you get a drafter reading mask tokens it has never seen.

DFlash — the draft that borrows everything

DFlash's draft model ships without an embedding table and without an LM head. It is a small transformer whose attention context is the target's own intermediate hidden states, injected into its KV cache before the query forward runs:

# Adapted from the DFlash reference implementation (HF) but implemented with
# SGLang primitives (RadixAttention + SGLang KV cache). This model intentionally
# does not include token embeddings or an LM head; DFlash uses the target model's
# embedding/lm_head.

That has a structural consequence SGLang calls out explicitly: because the draft's KV is manufactured from target hidden states rather than grown by the draft's own forwards, there is no draft/draft-extend split to maintain.

    @property
    def draft_worker(self):
        # DFLASH drives the draft model through a plain TpModelWorker: the
        # draft KV is materialized from target hidden states, so there is no
        # EagleDraftWorkerBase draft/draft_extend split to wrap it in.
        return self._draft_worker

Verification is greedy prefix matching over the block — accept while the drafted token equals the target's argmax at the previous position:

    matches = candidates[:, 1:] == target_predict[:, :-1]
    correct_len = matches.to(torch.int32).cumprod(dim=1).sum(dim=1)
    bonus = target_predict[torch.arange(bs, device=target_predict.device), correct_len]

The cumprod is §6.2's "discard the tail" rule expressed as arithmetic: one zero kills everything after it. A non-greedy path exists (python/sglang/srt/speculative/dflash_utils.py:L825-L844), described in its own docstring as "a chain-specialized variant of speculative target-only verification". The two info files are the two halves of the state, not two versions of it: dflash_info.py:L26-L32 is the target-side DFlashVerifyInput, dflash_info_v2.py:L35-L37 the draft-side state carried across overlap iterations — the _v2 is SGLang's overlap-scheduler generation, not a second DFlash.

DSpark — parallel drafting with a sequential refund

DSpark starts from the same mask block, then repairs the mutual blindness that parallel drafting causes. After the single draft forward produces base logits for all $\gamma$ positions, a small "Markov head" walks the block sequentially, biasing each position's logits by the token actually sampled at the previous one. There are no further transformer forwards in that loop:

    sampled_tokens = []
    corrected_logits = []
    prev_tokens = first_prev_tokens.long()
    for step_idx in range(proposal_len):
        step_hidden = None if hidden_states is None else hidden_states[:, step_idx, ...]
        step_logits = head.apply_step_logits(
            base_logits[:, step_idx, :],
            token_ids=prev_tokens,
            hidden_states=step_hidden,
        )
        next_tokens = sampler(step_logits, step_idx)
        sampled_tokens.append(next_tokens)
        corrected_logits.append(step_logits.unsqueeze(1))

Three head shapes ship — ("vanilla", "gated", "rnn"), dspark_components/dspark_config.py:L17 — and a positive markov_rank is mandatory. Default block size is $\gamma = 7$ (dspark_config.py:L16), so the verify window is 8. Note which switch turns the ratio test on: not --speculative-use-rejection-sampling, which raises for DSPARK, but the batch's own sampling parameters. accept_draft_tokens takes the greedy kernel when sampling_info.is_all_greedy, the AcceptSampling kernel when nothing in the batch is greedy, and runs both and merges when the batch is mixed (python/sglang/srt/speculative/dspark_components/dspark_verify.py:L697-L760). The sampling path builds $q$ by softmaxing the corrected logits at each request's temperature — that is the distribution the tokens were actually drawn from, so $q$ in §6.2's ratio test matches the sampler.

DSpark's second head emits a per-drafted-token survival probability. That feeds the adaptive machinery below, and it is why DSpark is the only algorithm for which supports_ragged_verify() returns true (python/sglang/srt/speculative/spec_info.py:L131-L135): different requests in the same batch may verify different numbers of tokens. May, not do. Ragged verify is gated on an environment variable, SGLANG_RAGGED_VERIFY_MODE, whose default is "static" (python/sglang/srt/environ.py:L1137) — and in static mode DSparkPlanner never constructs a HostConfidenceBudgetPlanner at all and build_ragged_layout returns None on every step (python/sglang/srt/speculative/dspark_components/dspark_planner.py:L132-L172, L417-L419). The other two modes, cap-accept and compact (python/sglang/srt/speculative/ragged_verify.py:L13-L16), additionally refuse to start unless the draft checkpoint actually ships a trained confidence head. So a default DSPARK launch runs the parallel mask-block draft and the Markov head, and verifies a uniform block of $\gamma + 1$ positions for every request: whatever confidence head the checkpoint carries goes unconsulted until you set the variable.

Frozen-KV MTP — deleting a forward pass

What is frozen is the target's KV cache, as seen by the draft. An EAGLE draft owns a KV pool: it writes KV for each drafted token, and after verification it must run a draft-extend forward over the accepted tokens to bring that pool up to date. Frozen-KV MTP's assistant owns no pool at all. It reads the target's committed KV read-only, and its rope phase does not advance:

def set_frozen_kv_positions(forward_batch: ForwardBatch, topk: int) -> None:
    """Rope phase = last written target slot, not advanced per draft step."""
    seq_lens = forward_batch.seq_lens
    positions = torch.clamp(seq_lens - 1, min=0).to(torch.int64)

Every draft step therefore attends against the same committed prefix; only the recurrent hidden state moves. The saving falls straight out:

    def _draft_extend_for_decode(self, batch: ScheduleBatch, batch_result) -> None:
        """Frozen 'draft extend': no forward. Pull the last accepted token's
        target hidden from the verify output and stash it as the next-iter seed.

        Replaces verify's `EagleDraftInput` with a `FrozenKVMTPDraftInput` so the
        next draft passes the FROZEN_KV_MTP attn-backend assertions.
        """

Count invocations at a common iteration boundary. In the earlier SGLang EAGLE trace, there are num_steps - 1 recurrent calls and one draft extension; extension also seeds the next iteration. Do not add another extension to a cost that already includes it. Use t_verify + sum(t_recurrent) + t_extend + t_projection and compare frozen-KV's actual calls and acceptance. Omitting draft KV provides a separate memory saving: ($h_{kv}=8$, $d_h=128$, bf16) the draft pool costs $2 \times 8 \times 128 \times 2 = 4$ KiB per token; at 64 sequences of 8k context that is 2.1 GB of HBM that frozen-KV simply does not allocate. Derived.

You do not select frozen-KV MTP directly. It is promoted from EAGLE/NEXTN when the draft checkpoint's architecture is a Gemma4 assistant (python/sglang/srt/arg_groups/speculative_hook.py:L52-L58) — it is not among the seven spellings --speculative-algorithm advertises (python/sglang/srt/server_args.py:L2101-L2105), and it is still being integrated: the predicate every scheduler call site dispatches on carries "FIXME(kpham_sgl): Remove FROZEN_KV_MTP here once we have established support for it in the scheduler.", and answers is_eagle() true for it in the meantime (python/sglang/srt/speculative/spec_info.py:L95-L103). Everything below is what that temporary arrangement does today. vLLM reached the same design for the same models, independently and under a different name — Gemma4Proposer sets:

        # All draft steps predict from the same position (the last
        # target-model position), so positions and seq_lens must not
        # advance between steps.
        self.constant_draft_positions = True

Two codebases, two names, one mechanism. That convergence is evidence the trick belongs to the model family, not to either engine.

Multi-layer EAGLE and STANDALONE — spending more to accept more

Multi-layer EAGLE runs a distinct draft module per step rather than one module recurrently:

    @property
    def draft_runners(self) -> List[ModelRunner]:
        # One runner per draft step (len == speculative_num_steps).
        return self.draft_runner_list

Distinct modules specialize prediction at different draft depths; that is not provably an easier statistical problem. With equal-size modules, resident weights can grow by a factor of the number of modules. For one invocation per step, total traffic is the sum over steps: use sum(c_i), or kappa * c for equal per-call cost, not a second factor of kappa hidden inside c. It is auto-enabled for MiMoV2 and Step3p5 targets (python/sglang/srt/arg_groups/overrides.py:L719-L724, :L1468-L1472) — in both cases only when --speculative-algorithm EAGLE was already asked for; the flag enable_multi_layer_eagle is otherwise False (python/sglang/srt/server_args.py:L2270-L2277). It is one of the few things adaptive speculation explicitly refuses to run with (adaptive_spec_params.py:L72-L76).

STANDALONE is the classical Leviathan setup: a real, independently trained draft LM. What makes it a separate worker rather than a config flag is that it must not share the target's embeddings or LM head:

class StandaloneDraftWorker(EagleDraftWorker):
    """Custom EagleDraftWorker that doesn't share embeddings/lm_head with target model."""
# ...
    def init_lm_head(self):
        """Override to prevent sharing embeddings and lm_head with target model."""
        # For standalone worker, we don't share embeddings and lm_head
        # The draft model uses its own embeddings and lm_head
        pass

It still requires an identical vocabulary, checked twice — by size and by the tokenizers' actual token-to-id maps (standalone_worker_v2.py:L197-L227). Note what STANDALONE is not: it is a standalone model inside the target's own process. The genuinely out-of-process design is decoupled_spec_io.py, which specifies drafter and verifier as separate engines on a ZMQ mesh, selected by --decoupled-spec-role {null,verifier,drafter} (python/sglang/srt/server_args.py:L2312-L2317). Read it as a design document rather than a feature — see the flag below. Its protocol is a two-way stream — DraftSync opens a drafter request, VerifyCommit reports which tokens survived, DraftTailStreamOutput carries speculated tails back — and the verifier is the stated source of truth:

@dataclass
class DraftSync:
    """Open or re-open a drafter request from a verifier-owned prefix.

    The verifier is the source of truth for committed tokens. DraftSync gives
    the drafter the prompt and already committed output prefix that it must
    align to before it can emit draft tail tokens.
    """

Also in the plumbing tier: external_corpus_manager.py lets you push a corpus of token chunks into a running server's n-gram suffix automaton, loaded on a background thread (L23-L31, L41-L54) — drafting from documents the model never saw in training, which belongs to §6.3's family.

§6.2 established the headline: neither engine's shipped default is full probabilistic rejection sampling. Both defaults collapse $q$ to a one-hot distribution, under which $\min(1, p/q) = p(x)$ — still exact, but a different kernel and a different acceptance rate. Confirmed at both SHAs:

    draft_sample_method: DraftSampleMethod = "greedy"
    """How the draft model samples tokens. 'greedy' always picks the argmax
    token, and the draft probabilities are treated as one-hot during rejection
    sampling. 'probabilistic' samples stochastically from the draft
    distribution and uses the full draft logits for the probability ratio test
    during rejection sampling. This comes at the cost of additional GPU memory
    usage."""

SGLang's counterpart is speculative_use_rejection_sampling = False (python/sglang/srt/server_args.py:L2192-L2196), which also requires topk 1 — so on the tree path it is not even available. On top of that sit two multipliers that void the exactness proof outright, both defaulting to the identity (server_args.py:L2182-L2191, covered in §6.2).

vLLM's third axis is rejection_sample_method, which §6.2 handed to this chapter:

    rejection_sample_method: RejectionSampleMethod = "standard"
    """The rejection sampling method to use. 'standard' uses probabilistic
    rejection sampling (with or without cached draft logits, controlled by
    draft_sample_method). 'synthetic' accepts draft tokens with a decaying
    probability calibrated to synthetic_acceptance_rate. 'block' uses block
    verification (Sun et al.), which jointly verifies the draft tokens as a
    block instead of one at a time."""

"block" is Sun et al.'s block verification. Where the standard rule tests one token against one ratio, the block rule accepts a prefix length using the joint probability of the whole prefix and the residual mass at the next position:

            elif USE_BLOCK_VERIFICATION:
                # Block verification (Sun et al., 2024): https://arxiv.org/abs/2403.10444
                prefix_joint_ratio = tl.exp(
                    tl.load(cumulative_log_p_ptr + logit_idx).to(tl.float32)
                )
                next_draft_token = tl.load(
                    draft_sampled_ptr + logit_idx + 2,
                    mask=i < num_draft_tokens - 1,
                    other=-1,
                ).to(tl.int64)
                if next_draft_token >= 0:
                    residual_mass = _compute_global_residual_mass(
                        # ...
                    )
                    denom = residual_mass + 1.0 - prefix_joint_ratio
                    h = tl.where(denom > 0.0, residual_mass / denom, 1.0)
                else:
                    h = prefix_joint_ratio
                accepted_length = tl.where(u <= h, i + 1, accepted_length)

The block rule adds vocabulary-scale residual work. Under the comparison theorem its expected accepted length is at least that of the corresponding standard rule, not strictly larger for every distribution (identical draft and target already accept everything). That acceptance improvement does not guarantee a latency improvement after verifier overhead.

"synthetic" is not a production rule at all — it is an ablation harness. You hand it an acceptance curve and it accepts according to that curve regardless of what the draft actually proposed, so you can sweep "what would a drafter with $\alpha = 0.8$ buy me" without training one. §6.2 already quoted its unconditional_to_conditional_rates; the other half is the shortcut from a single mean acceptance length:

    @staticmethod
    def _acceptance_length_to_rates(length: float, n: int) -> list[float]:
        """Mean acceptance length to unconditional per-position rates, using
        the minimum-variance schedule."""
        num_drafts = length - 1  # expected number of accepted draft tokens
        num_full = int(num_drafts)
        return (
            [1.0] * num_full + [num_drafts - num_full] + [0.0] * (n - num_full - 1)
        )[:n]

The staircase fixes a mean with minimal variance. For fixed draft width kappa and mean accepted-draft count A, expected wasted proposals are exactly kappa - A, independent of variance. Burstiness can affect queueing, adaptive policies, allocation peaks and nonlinear batch costs, so the harness is not a production predictor, but no fixed-width expected-waste penalty follows from variance alone.

The engines organize builtin dispatch differently. SGLang's decorator registry and vLLM's builtin branches expose different extension contracts; vLLM also has the custom proposer entry point discussed below.

SGLang's is a decorator registry with a duck-typing contract:

        """Decorator to register a plugin speculative algorithm. The factory
        takes ``server_args`` and returns the worker class. Pass a
        ``CustomSpecAlgo`` subclass via ``spec_class`` to override any
        ``is_*()`` / ``create_worker`` method.

        Example:
            @SpeculativeAlgorithm.register("MY_SPEC", supports_overlap=False)
            def _factory(server_args):
                return MySpecWorker
        """

The interesting part is the guard. Callers all over the scheduler dispatch on predicates like spec_algorithm.is_eagle(), so a plugin class must answer every predicate the builtin enum answers or it will AttributeError at some unrelated call site. Rather than maintain a second list, registration reflects over the enum:

    from sglang.srt.speculative.spec_info import SpeculativeAlgorithm

    interface = {
        name
        for name in vars(SpeculativeAlgorithm)
        if name.startswith(("is_", "supports_"))
    }
    missing = sorted(interface - set(dir(spec_class)))
    if missing:
        raise TypeError(
            f"{spec_class.__name__} is missing duck-typed methods from "
            f"SpeculativeAlgorithm: {missing}. Add them to {spec_class.__name__} "
            "so plugin-registered algorithms stay dispatchable."
        )

The comment above it names the bug that motivated the guard: "this is how is_some / is_frozen_kv_mtp silently went missing". Builtins are reserved by deriving the name set from the enum (spec_registry.py:L174-L184), so a plugin can never shadow DSPARK.

The extension point is real and, at this SHA, unused: nothing under python/sglang/ calls SpeculativeAlgorithm.register — the only registrations in the tree are the ones test/registered/unit/spec/test_spec_registry.py makes to exercise the guard. So the plugin row in the inventory table is a shape you can fill, not an algorithm that ships.

vLLM's equivalent is a 60-line if/elif in the model runner's constructor, whose declared type is a union that must be edited to add a member:

        if self.speculative_config and get_pp_group().is_last_rank:
            self.drafter: (
                NgramProposer  # noqa: F823
                | NgramProposerGPU
                | SuffixDecodingProposer
                | EagleProposer
                | DFlashProposer
                | DraftModelProposer
                | MedusaProposer
                | ExtractHiddenStatesProposer
                | Gemma4Proposer
                | Step3p5MTPProposer
            )
            if self.speculative_config.method == "custom_class":
                self.drafter = create_custom_proposer(  # type: ignore[assignment]
                    self.vllm_config
                )

Three consequences. Order is load-bearing: use_dflash() and use_gemma4_mtp() must precede use_eagle(), because use_eagle() returns true for mtp, dflash and dspark (vllm/config/speculative.py:L1496) — a new MTP variant inserted below the wrong branch silently becomes an EAGLE. There is no interface: SpecDecodeBaseProposer is a plain class, and MedusaProposer, NgramProposer and SuffixDecodingProposer do not inherit from it and have mutually incompatible propose signatures, so a second if/elif exists at the call site to invoke each correctly. And adding a method means editing at least three places: the Literal, the construction chain, and the invocation chain.

vLLM does ship a user extension point, and its narrowness is instructive:

def create_custom_proposer(vllm_config: VllmConfig):
    """Load and instantiate a user-provided proposer class.

    The class path is read from ``speculative_config.model``
    (e.g., ``"my_module.MyCustomProposer"``).  The class is
    imported, instantiated with *vllm_config*, and returned
    directly so the caller can use it without any wrapper.

    The returned object must expose a callable ``propose`` method.
    """

The entire contract is hasattr(instance, "propose") and callable (L57-L65). The model config field is overloaded as a Python import path, detected by a heuristic that fires on any dotted, slash-free, identifier-legal string (vllm/config/speculative.py:L729-L739) — a local model directory named my.model would be misread as a class path. And the runner calls it with the n-gram-shaped arity, token ids in and token ids out, so you cannot plug in a hidden-state-consuming drafter this way. SGLang's plugin gets a worker with the full draft/verify lifecycle; vLLM's gets a token-list transform.

This is the item that matters most in production, because it is the direct answer to §6.2's break-even result: stop treating $\kappa$ as a constant and let the server turn it down when speculation stops paying.

SGLang's --speculative-adaptive is a real closed loop. Trace one decision.

Step 1 — before drafting, route the batch. EAGLEWorkerV2.forward_batch_generation calls self.activate_step_by_batch(batch.seq_lens.shape[0]) (python/sglang/srt/speculative/eagle_worker_v2.py:L1152). The controller pads the batch size up to the next captured CUDA-graph size, finds the nearest configured slot, and reads that slot's current $\kappa$ (adaptive_spec_params.py:L329-L345). Batch 40, with graphs at {1,2,4,8,16,32,64}, pads to 64 and lands in the "64" slot — whose only candidate is 0. Speculation is off for that step.

Step 2 — verify, then feed the loop from the CPU side. The accept counts are already on the host by the time the batch-result processor runs, so the controller is fed there rather than forcing a sync in the worker:

        # Feed the adaptive controller now that accept_lens is on CPU,
        # instead of doing a synchronous GPU→CPU copy in the worker hot path.
        # BaseSpecWorker provides a no-op default for non-adaptive workers.
        self.model_worker.on_verify_complete_cpu(
            result.num_correct_drafts_per_req_cpu, batch_size=len(batch.reqs)
        )

Step 3 — update the EMA. The measured signal is num_correct_drafts_per_req, the per-request count of accepted draft tokens excluding the bonus — which is exactly $\mathbb{E}[n] = E - 1$ from §6.2. It is smoothed with $\alpha_{\text{ema}} = 0.2$, and updates are gated: 10 warm-up batches, then every 5th (adaptive_spec_params.py:L174-L198).

Step 4 — decide, with hysteresis. The rule is one line of intent, and the code is worth reading for the asymmetry:

        while current_idx > 0:
            prev_step = self.candidate_steps[current_idx - 1]
            # A zero-step candidate disables drafting. Treat zero accepted drafts
            # as low enough to reach it when it is the floor candidate.
            drop_threshold = 0.5 if prev_step == 0 else prev_step - 0.5
            drop_threshold += self.down_hysteresis
            if self.ema_accept_len <= drop_threshold:
                current_idx -= 1
            else:
                break

        moved_down = current_idx < old_idx
        if not moved_down:
            while current_idx < len(self.candidate_steps) - 1:
                current_step = self.candidate_steps[current_idx]
                rise_threshold = current_step - 0.5 + self.up_hysteresis
                if self.ema_accept_len > rise_threshold:
                    current_idx += 1
                else:
                    break

The class states the intent: "if drafts are consistently accepted, try more steps; if drafts are consistently rejected early, reduce steps to avoid waste... Probes one step beyond observed acceptance" (L140-L151). Downward moves are evaluated first and, once taken, block the upward loop in the same tick — no oscillation within a decision. At batch 1 the shipped down_hysteresis = -0.25 makes it 0.25 tokens harder to drop than to rise, biasing the low-batch case toward speculating.

Step 5 — swap the runtime state. Changing $\kappa$ changes every shape in the step, so a "state" is six objects swapped together:

@dataclass
class SpecRuntimeState:
    """A complete set of runtime resources bound to a specific speculative
    decoding configuration.

    Each decode round runs three stages — draft, verify, extend — and every
    stage has shape-dependent resources (attention backends and CUDA graphs)
    that must match the current configuration.  Switching adaptive steps
    means swapping the entire state atomically.
    """

At runtime the swap is six pointer assignments (eagle_worker_v2.py:L1386-L1425) — genuinely free. The whole cost is paid at startup: init_states builds and captures a full set of attention backends and CUDA graphs for every candidate step (adaptive_runtime_state.py:L94-L111). Three candidates means roughly three times the graph capture time and graph memory. That is the honest price of adaptive speculation, and it is why cuda_graph_bs_for_step exists to prune which batch sizes each step must capture.

The finer loop: DSpark's confidence budget

SGLang's controller picks one $\kappa$ for the whole batch, on a five-batch cadence. DSpark's adaptive verification picks a $\kappa$ per request, every step, from the drafter's own confidence. vLLM implements the same idea and its version is the clearer read. Every (request, step) slot is scored by survival probability, and a global top-$k$ admits the best ones under a budget:

    """Admit the globally best `draft_budget` draft slots, in place.

    Every (request, step) slot is scored by its survival probability, the running
    product of that request's per-position confidences, and the highest scores win.
    Survival only decreases along a request, so a global top-k always admits
    continuously along steps with a request.
    """
    survival = confidence_probs[idx_mapping].cumprod(dim=1)
    steps = torch.arange(num_steps, device=survival.device)
    # Out-of-range slots score -inf so they never outrank a real draft.
    survival = survival.masked_fill(out_of_range, -float("inf"))
    flat = survival.flatten()
    winners = flat.topk(draft_budget).indices

Survival products are nonincreasing when finite confidence values lie in [0,1]. That alone does not guarantee a prefix under arbitrary top-k ties: scores [0.5,0.5] can select the second position without the first. Use a deterministic earlier-position-first tie rule, or closure repair before converting selection to capacities. Probability-one edges, zeros and floating-point underflow make this an operational case, not only a mathematical corner.

And the budget itself is chosen by maximising predicted tokens per second against a profiled cost table:

        num_tokens_to_estimated_accepted_tokens = np.concatenate(
            ([num_sampling_requests], num_sampling_requests + np.cumsum(scores))
        )
        costs = (
            draft_cost_ms[len(req_ids)]
            + verify_cost_ms[
                num_non_draft_tokens_total : num_non_draft_tokens_total
                + max_draft_budget
                + 1
            ]
        )
        # ...
        draft_budget = int(np.argmax(num_tokens_to_estimated_accepted_tokens / costs))

That argmax of estimated-accepted-tokens over milliseconds is §6.2's speedup formula $S = E/(1+\kappa c)$, evaluated numerically at every candidate budget instead of solved symbolically at one $\alpha$. SGLang's planner reaches the identical expression under a different name — theta = tau_star / step_time then argmax (python/sglang/srt/speculative/dspark_components/dspark_planner.py:L965-L972). The cost table is built from real profiled step timings and is deliberately a step function below the CUDA-graph capture limit, "because execution pads to the next captured size, so cost is a step function of the padded size: smooth interpolation would invent marginal per-token costs that don't exist within a pad bucket" (adaptive_verification.py:L86-L90). Speculation budgeting that ignores CUDA graph padding buckets will systematically over-draft.

Speculation across a prefill/decode boundary

Under P/D disaggregation (§1.6) the decode instance receives a request whose prefill ran somewhere else. Plain decode needs the KV cache transferred. Speculation needs the draft seed state transferred too, and how much that is depends entirely on the method. SGLang encodes the distinction in one predicate:

    def carries_draft_hidden_states(self) -> bool:
        """Whether the disagg prefill->decode transfer carries draft hidden
        states (EAGLE-family only; STANDALONE's vanilla draft ignores them)."""
        return self.is_eagle()

For EAGLE the seed is genuinely heavy — per request, a hidden-state vector plus the top-$k$ probabilities and indices, all stacked on the decode side from fields the prefill node had to ship:

    spec_info = EagleDraftInput(
        topk_p=topk_p,
        topk_index=topk_index,
        hidden_states=hidden_states,
        bonus_tokens=last_tokens_tensor,
        dsa_topk_indices=dsa_topk_indices,
    )
    spec_info.capture_hidden_mode = CaptureHiddenMode.LAST

For Llama-3-8B that hidden state is 4096 bf16 values, 8 KB per request — small against the KV transfer, but a different tensor on a different schedule, and it must arrive before the first draft. Under multi-layer EAGLE the top-$k$ payload multiplies by the step count (eagle_disaggregation.py:L27-L31).

DFlash and DSpark ship nothing extra. Their disaggregation helpers are 32 lines each and synthesise the seed locally from the last token and the sequence lengths — make_draft_input_v2(bonus_tokens=last_tokens_tensor, new_seq_lens=batch.seq_lens) (dflash_disaggregation.py:L21-L24). DSpark instead calls make_next_draft_input with the same two arguments (dspark_disaggregation.py:L21-L24). The parallel-drafting family is cheaper to disaggregate, because it drafts from target hidden states recomputed locally rather than from a recurrent draft state that has to be carried across the wire. The file-size difference — 109 lines against 32 — is the whole argument.

Speculation plus LoRA

The pinned SGLang HEAD is exactly this commit: "[Spec][LoRA] Support multi-adapter LoRA with EAGLE/NEXTN/DFLASH/DSPARK speculative decoding (#34337)". Before it, the combination was rejected outright. Three reasons it is hard, all in the diff.

1. The draft shares the target's LM head, and LoRA wraps it. EAGLE-family and DFlash drafts borrow lm_head and embed_tokens; once an adapter wraps those modules, the draft silently starts consuming adapter deltas through a path never designed for them. The fix is to peel the wrapper — unwrap_lora_layer(getattr(model, "lm_head", None)) at eagle_worker_v2.py, dflash_worker_v2.py:L484 and :L1753-L1754.

2. The verify batch has a different token width than a decode batch, and LoRA's segmented-GEMM layout assumes it knows that width:

def get_batch_token_counts(forward_batch: ForwardBatch) -> Tuple[int, int]:
    """(total tokens, max tokens per request) for LoRA segment math."""
    mode = forward_batch.forward_mode
    if mode.is_decode():
        return forward_batch.batch_size, 1
    if mode.is_target_verify():
        num_tokens_per_req = forward_batch.spec_info.draft_token_num
        return forward_batch.batch_size * num_tokens_per_req, num_tokens_per_req

Hence the constant that gates the whole feature — the supported set is exactly the algorithms whose verify forward presents a uniform per-request width:

# Speculative algorithms whose verify forward presents a uniform per-request
# token width, which is what the LoRA segment layout assumes.
_LORA_SPEC_ALGORITHMS = ("EAGLE", "EAGLE3", "DFLASH", "DSPARK")

Which is why the two features that make $\kappa$ vary are the two LoRA's static layout cannot tolerate — though they are refused on different terms, and the difference matters at the command line. DSPARK itself is on the allow-list: _check_lora_speculative_compatibility only raises for it when SGLANG_RAGGED_VERIFY_MODE != "static", and static is the default (python/sglang/srt/environ.py:L1137), so LoRA plus a default DSPARK launch is a supported combination — you lose the per-request budget, not the adapter. Adaptive speculation is refused unconditionally. Both refusals are startup ValueErrors raised while validating server args, not silent misbehaviour at decode time:

            (
                self.speculative_algorithm == "DSPARK" and ragged_mode != "static",
                f"does not support SGLANG_RAGGED_VERIFY_MODE={ragged_mode!r}: "
                "the per-request verify lengths it schedules break the "
                "uniform-width LoRA segment layout",
            ),
            (
                self.speculative_adaptive,
                "does not support --speculative-adaptive: the draft is built "
                "from a static ServerArgs snapshot, and the runtime-state "
                "swap does not rebuild LoRA cuda-graph metadata",
            ),

3. The adapter changes the target but not the draft — which costs throughput, never correctness. This is §6.2's theorem earning its keep, and SGLang says it in a log line:

    logger.warning(
        "LoRA adapter '%s' targets embedding modules (%s) while EAGLE-family "
        "speculative decoding is enabled. The shared draft consumes their "
        "base weights, so those deltas do not influence drafting and may "
        "reduce the accept rate. Outputs are unaffected.",
        lora_name,
        ", ".join(modules),
    )

An adapter changes the target distribution while the shared draft can remain unchanged. TV distance can rise, fall or stay equal; an imperfect draft might happen to be closer to the adapted target. Measure per-adapter acceptance and latency. Exact verification preserves the adapted target distribution only when its probabilities and proposal probabilities are correctly accounted for. See §7.5.

Three more sharp edges

• mlp_speculator is dead in vLLM. It is a valid method literal and is auto-detected at vllm/config/speculative.py:L977, but has no branch in the dispatch chain, so it falls through to raise ValueError("Unknown speculative decoding method: ...").
• num_draft_tokens_per_pos is written and never read in vllm/v1/spec_decode/metrics.py:L48-L49. The four Prometheus counters that do exist are vllm:spec_decode_num_drafts, ..._num_draft_tokens, ..._num_accepted_tokens, ..._num_accepted_tokens_per_pos; everything else in the log line is derived at print time.
• DSpark's cost table degenerates without a profile. --speculative-dspark-sps-table-path is optional, and its help text says what omitting it costs: "Omit for an uninitialized flat constant-SPS table: the budget degenerates to verify-all (zero throughput gain by itself)" (python/sglang/srt/server_args.py:L2152-L2160). You get the confidence head's costs with none of its benefits. That is the second way to get nothing out of the confidence head; the first is leaving SGLANG_RAGGED_VERIFY_MODE at its default static, in which the budget planner is never constructed and the table is never consulted.

Use the same target-step baseline and iteration boundary. The following bisection solves the acceptance needed by a parallel proposal to beat the stated chain, not merely to beat no speculation. Confidence-derived expected yield is a model: validation must compare calibrated predictions to actual accepted counts and profiled total cost, including extension even at a zero draft budget when that path still runs.

def emitted(alpha, depth=3):
    return sum(alpha ** i for i in range(depth + 1))

c = 0.165
chain = emitted(0.7) / (1 + 3*c)
parallel_half_acceptance = emitted(0.35) / (1 + c)
assert parallel_half_acceptance < chain
low, high = 0.0, 1.0
for _ in range(60):
    middle = (low + high) / 2
    if emitted(middle) / (1+c) < chain:
        low = middle
    else:
        high = middle
assert 0.53 < high < 0.54
assert abs(emitted(high) / (1+c) - chain) < 1e-12
print("Chain speedup:", chain, "parallel tie acceptance:", high)

Print the full flag surface first — it moves weekly, and this is the version-stamped way to check what your build actually accepts:

grep -n "speculative" python/sglang/srt/server_args.py | sed -n '1,80p'
python3 -c "from sglang.srt.speculative.spec_info import SpeculativeAlgorithm as A; print([a.name for a in A])"

Then run the experiment this chapter argues for. Hold everything fixed, enable the adaptive controller, and watch it walk $\kappa$ down as you raise concurrency — the log line "Adaptive spec params updated: steps {old} -> {new} (ema_accept_len=...)" (adaptive_spec_params.py:L253-L257) is the observable. The algorithm has to be EAGLE or EAGLE3 at topk 1, or the controller disables itself before the first step and you will watch for a line that never prints:

python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path <an EAGLE3 head for this target> \
    --speculative-eagle-topk 1 \
    --speculative-adaptive

# Contrast: the same flag on a parallel drafter is accepted and then thrown away.
# Expect "speculative_adaptive disabled: speculative_algorithm=DFLASH
# (only EAGLE/EAGLE3 are supported). Falling back to static speculative params."
python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat \
    --speculative-adaptive

Sweep concurrency 1 → 8 → 32 → 64 and record the step the controller settles on at each. Then set --speculative-adaptive-config to a file whose "64" slot allows [0, 1, 3] and measure whether the controller's choice actually maximises throughput, or whether the shipped floor of [0] is leaving tokens on the table for your workload. That is the only honest way to find your own $B_{\text{break-even}}$, and it is Lab 08's territory. Nothing in this chapter was measured.

• Sun et al., SpecTr / block verification (arXiv:2403.10444) — the URL vLLM's kernel cites at vllm/v1/worker/gpu/spec_decode/rejection_sampler_utils.py:L596. Read it against the USE_BLOCK_VERIFICATION branch; the denom = residual_mass + 1 - prefix_joint_ratio line is the paper's acceptance function.
• sgl-project/sglang#30261 — "Add DSpark: confidence-scheduled speculative decoding". The introducing PR, and the only place either repo explains what DSpark is for. Follow-ups #31434, #31985, #32160.
• sgl-project/sglang#34337 — "[Spec][LoRA] Support multi-adapter LoRA with EAGLE/NEXTN/DFLASH/DSPARK speculative decoding". The pinned HEAD. The _LORA_SPEC_ALGORITHMS tuple and the four-entry unsupported list are the most compact statement anywhere of what speculation assumes about batch geometry.
• LMSYS, "The next generation of speculative decoding: DFlash and Spec V2" is linked by the pinned tree. Absence of a paper link in that checkout is not evidence that no paper exists.
• SGLang's own Adaptive Speculative Decoding doc page states this chapter's central economics in one sentence: "At high batch sizes, the cost of each wasted draft step is multiplied across all sequences in the batch, so the optimal step count is often lower than at low batch sizes." It also documents the EAGLE/EAGLE3 and topk-1 restrictions that adaptive_unsupported_reason enforces in code.
• Source to read next, in order: python/sglang/srt/speculative/spec_info.py (the enum and every predicate the scheduler dispatches on), spec_registry.py (the extension point), adaptive_spec_params.py (the control loop), then vllm/v1/worker/gpu/spec_decode/adaptive_verification.py (the finer loop). Roughly 1,200 lines total, and they contain the entire design space.
• The algorithm and its proof are §6.2; where drafts come from is §6.3; EAGLE, Medusa, MTP and tree verification are §6.4; the disaggregation this chapter leans on is §1.6; LoRA serving is §7.5.