Frontend DSL, grammar backends, router, extension points

You wrote a replacement for SGLang's radix tree and pointed the server at it. It comes back with this:

def create_tree_cache(ctx: TreeCacheBuildContext) -> BasePrefixCache:
    """Route to the matching factory to construct Radix Cache."""
    name = ctx.server_args.radix_cache_backend
    if name:
        factory = get_radix_cache_factory(name)
        if factory is None:
            raise ValueError(
                f"--radix-cache-backend={name!r} is not registered. "
                f"Registered backends: {registered_radix_cache_backends()}. "
                "External backends must call register_radix_cache_backend(...) at import time."
            )
        cache = factory(ctx)

That error names the function, says when to call it, and prints what is already registered. It is the best-behaved seam in the tree. Three directories away, register_grammar_backend(name, init_func) is a bare dict assignment with no validation at all, and an in-tree test that asserts the second registration silently wins. Both are extension points. They disagree about almost everything else.

§11.5 ran this audit on vLLM and found the same kind of disagreement — quantization and attention overwrite silently, KVConnectorFactory raises, linear kernels append at lowest priority. This is the SGLang half, so that §13.1 gets symmetric material. But the audit is only half the story: SGLang's surface contains three things with no vLLM counterpart at all — a language frontend, a grammar-backend registry where vLLM has a hard-coded if/elif chain, and a Rust gateway that is a separately versioned product.

The four topics are not peers; they sit at three distances from the GPU. The frontend language runs in your process and talks HTTP. The router runs in its own process — a different language, release cadence, and version number. The grammar backends and the registries run inside the engine processes §12.1 laid out. Nothing in the frontend can see a KV block; nothing in the router can either.

The seam between the frontend and the engine is worth staring at, because it is the whole design. The language never sends a KV handle, a session id, or a cache key. It sends text. The sharing that makes a forked program cheap is not arranged by the frontend at all — it falls out of RadixAttention matching two prompts that happen to start the same way (§2.4). The language's only job was to make sure they do.

State the current status first, from the repo's own code-structure doc, so nothing below misleads:

- `benchmark/`: Benchmark implementations and dataset utilities.
- `cli/`: Command-line interface commands and entrypoints.
- `kernels/`: Kernel interfaces, implementations, selection, and debugging utilities shared by the runtimes.
- `lang/`: Deprecated language frontend that is no longer actively maintained.
- `multimodal_gen/`: Core runtime for image, video, and audio generation models, most of which are diffusion models.
- `srt/`: Core runtime for autoregressive language models. (SRT = SGLang Runtime.)
- `test/`: Shared test and evaluation utilities.

Fourteen files, 4,644 lines, one line of documentation saying do not build on it. The interface today is the OpenAI-compatible server (§9.1). Read this section the way §2.2 reads the deleted PagedAttention kernel and §6.5 reads dead jump-forward decoding: a retired design, explicitly labelled, taught because understanding it explains something that is still live.

Here, what is still live is RadixAttention, and the language is where its motivation comes from.

The bottleneck the language was answering

A structured LM program — chain of thought with $n$ branches, a self-consistency vote, a tool-choice tree, a rubric scored on five axes — issues many generations that share a long preamble. Naively, each branch is an independent request and each pays a full prefill of the shared preamble.

Price it. The KV cell size from FORMULAS is $k = 2 \cdot L \cdot h_{kv} \cdot d_h \cdot b$. For Llama-3-8B in bf16 ($L=32$, $h_{kv}=8$, $d_h=128$, $b=2$) that is $2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072$ bytes, i.e. 128 KiB per token. A program that forks 8 ways after a 2,048-token shared prompt holds:

1.75 GiB of H100 HBM recovered, and seven eighths of the prefill FLOPs skipped — arithmetic from the formula and published Llama-3-8B shapes, not a measurement. That is the entire economic argument for both the language and the cache. They are one insight approached from two directions: the cache makes shared prefixes cheap if requests have them; a program with a fork has one by construction. Only one of those two directions turned out to be necessary, which is why one of them is deprecated and the other is the first bullet in the project README.

What a program looked like

A program is a Python function whose first parameter is the state s, decorated with @sgl.function; the decorator asserts the calling convention at definition time (assert argspec.args[0] == "s", python/sglang/lang/ir.py:L148). Everything is +=: a literal string, or an SglExpr — a gen, a select, a role marker. Control flow is ordinary Python, which is the interesting part: if s["tool"] == "calculator": reads a variable the model just produced, so the interpreter must block on that generation before the branch is evaluable. Here is the fork idiom as the in-tree program tests exercise it:

        # Generate detailed tips
        forks = s.fork(fork_size)
        for i in range(fork_size):
            forks[
                i
            ] += f"Now, I expand tip {i+1} into a detailed paragraph:\nTip {i+1}:"
            forks[i] += sgl.gen("detailed_tip", max_tokens, stop=["\n\n"])
        forks.join()

        # Concatenate tips and summarize
        s += "Here are these tips with detailed explanation:\n"
        for i in range(fork_size):
            s += f"Tip {i+1}:" + forks[i]["detailed_tip"] + "\n"

        s += "\nIn summary," + sgl.gen("summary", max_tokens=512)

The prefix-tree shape is visible in the source. One preamble, fork_size branches, a join() that collects every result — a program that cannot be written without a shared prefix.

Worked trace: what a fork actually did

StreamExecutor.fork() is the load-bearing 30 lines, and the mechanism is not what the name suggests:

    def fork(
        self,
        size: int = 1,
        position_ids_offset: Optional[List[int]] = None,
    ):
        if size > 1 and str(self.text_):
            self.submit(SglCommitLazy())

        self.sync()
        size = int(size)

        exes = [
            StreamExecutor(
                self.backend,
                self.arguments,
                self.default_sampling_para,
                self.chat_template,
                self.stream,
            )
            for _ in range(size)
        ]
        for i in range(size):
            exes[i].variables = dict(self.variables)
            exes[i].text_ = str(self.text_)
            exes[i].messages_ = list(self.messages_)
            exes[i].cur_role = self.cur_role
            exes[i].cur_role_begin_pos = self.cur_role_begin_pos
            exes[i].fork_start_text_pos = len(self.text_)
            exes[i].images_ = list(self.images_)

            # TODO(ying): handle API speculative execution

Each child gets str(self.text_) — a full copy of the parent's string. No KV handle, no request id, no reference to a tree node. Each child then POSTs that whole string as its prompt (python/sglang/lang/backend/runtime_endpoint.py:L166-L167 sets "text": s.text_). The one optimisation the frontend performs is the SglCommitLazy at the top of fork(), which fires a zero-token generation whose only purpose is to plant the shared prefix in the server's radix tree before the branches race for it:

    def commit_lazy_operations(self, s: StreamExecutor):
        data = {"text": s.text_, "sampling_params": {"max_new_tokens": 0}}
        self._add_images(s, data)
        res = http_request(
            self.base_url + "/generate",
            json=data,
            api_key=self.api_key,
            verify=self.verify,
        )
        self._assert_success(res)

The full path, in order: SglFunction.run() → run_program() → StreamExecutor.__init__ starts a worker thread → ProgramState.fork(n) → StreamExecutor.fork() submits SglCommitLazy → _execute_commit_lazy_operations → RuntimeEndpoint.commit_lazy_operations posts max_new_tokens: 0 → each child StreamExecutor runs _execute_gen → RuntimeEndpoint.generate posts the full text → the server's match_prefix finds the committed nodes (§12.3) → every branch prefills only its own suffix → ProgramStateGroup.join() gathers the child variables back into the parent (python/sglang/lang/interpreter.py:L1052-L1066).

Two observations explain how to migrate, without establishing the maintainers' reasons for deprecation. The shown DSL uses HTTP requests and a ThreadPoolExecutor, so an ordinary client can reproduce prompt construction and concurrency. Equivalence also requires matching commit/warm order, branching, cancellation, joins and failures; identical prompts alone do not reproduce program semantics. Server-side continuous batching and prefix caching benefit HTTP clients independently of adopting the historical language.

The README marks the language deprecated, and the cited activity counts show much less recent work than srt or the gateway. Manual tests and five API backends remain in the snapshot, but their presence does not prove present-day compatibility or successful execution. Treat the old frontend as historical unless a pinned end-to-end smoke test succeeds.

§6.5 owns the mechanism — how a grammar compiles to a token-level FSM, how the bitmask lands on logits, where the host-side cost goes, and why jump-forward decoding is dead code at this SHA. What belongs here is the seam: what a backend must implement, how one is selected, and how the shape of that interface differs from vLLM's.

The contract has two halves. A BaseGrammarObject is per-request state: accept_token(token), rollback(k), allocate_vocab_mask, fill_vocab_mask(mask, idx), and the three static methods reset_vocab_mask, move_vocab_mask, apply_vocab_mask (python/sglang/srt/constrained/base_grammar_backend.py:L58-L107). A BaseGrammarBackend is the per-server compiler, and its abstract surface is four dispatch methods with permissive defaults:

    def dispatch_fallback(self, key_type: str, key_string: str) -> BaseGrammarObject:
        """
        This function should not be reached in any case.
        """
        raise ValueError(f"Invalid key_type: {key_type}={key_string}")

    def dispatch_json(self, key_string: str) -> BaseGrammarObject:
        return self._not_supported("json", key_string)

    def dispatch_regex(self, key_string: str) -> BaseGrammarObject:
        return self._not_supported("regex", key_string)

    def dispatch_ebnf(self, key_string: str) -> BaseGrammarObject:
        return self._not_supported("ebnf", key_string)

    def dispatch_structural_tag(self, key_string: str) -> BaseGrammarObject:
        return self._not_supported("structural_tag", key_string)

Every default returns an InvalidGrammarObject after logging "Skip unsupported {key_type=}, {key_string=}", so a backend that only understands JSON schemas overrides one method and degrades gracefully on the other three. vLLM takes the opposite stance — one abstract compile_grammar(request_type, grammar_spec, stop_token_ids) that must handle every member of the StructuredOutputOptions enum:

class StructuredOutputBackend(ABC):
    """Engine-level backend for structured output requests."""

    vllm_config: VllmConfig
    tokenizer: TokenizerLike
    vocab_size: int

    @abstractmethod
    def compile_grammar(
        self,
        request_type: StructuredOutputOptions,
        grammar_spec: str,
        stop_token_ids: set[int] | None = None,
    ) -> StructuredOutputGrammar:
        """
        Compiles a grammar specification into a structured output grammar.

        Args:
            request_type (StructuredOutputOptions): The type of structured
                output request.

Both engines converged on the same three-verb runtime contract — accept a token, fill a bitmask, ask whether the grammar is terminated. The divergence is in selection. SGLang has a registry, checked first:

def register_grammar_backend(name, init_func):
    GRAMMAR_BACKEND_REGISTRY[name] = init_func


def create_grammar_backend(
    server_args: ServerArgs,
    tokenizer,
    vocab_size: int,
    eos_token_ids: Optional[set] = None,
    think_end_ids: Optional[List[int]] = None,
) -> Optional[BaseGrammarBackend]:
    name = get_exec().kernel.grammar_backend

    # Custom grammar backend has the highest priority
    if name in GRAMMAR_BACKEND_REGISTRY:
        return GRAMMAR_BACKEND_REGISTRY[name](
            server_args, tokenizer, vocab_size, eos_token_ids
        )

    # Default grammar backends

vLLM has a hard-coded chain in the manager instead, ending in raise ValueError(f"Unsupported structured output backend: {backend}"):

        if self.backend is None:
            assert request.sampling_params is not None
            backend = request.sampling_params.structured_outputs._backend
            vocab_size = self.vllm_config.model_config.get_vocab_size()
            if backend == "xgrammar":
                self.backend = XgrammarBackend(
                    self.vllm_config,
                    tokenizer=self.tokenizer,
                    vocab_size=vocab_size,
                )
            elif backend == "guidance":
                self.backend = GuidanceBackend(
                    self.vllm_config,
                    tokenizer=self.tokenizer,
                    vocab_size=vocab_size,
                )

A fifth structured-output backend in vLLM is a core-file PR. In SGLang it is a register_grammar_backend call from a plugin. Two caveats before you celebrate. "Custom grammar backend has the highest priority" means a plugin registering under the name "xgrammar" shadows the built-in with no warning; and the overwrite semantics are pinned by a test, so they are intentional, not an oversight:

    def test_overwrite_registration(self):
        register_grammar_backend("dup", lambda *a: "first")
        register_grammar_backend("dup", lambda *a: "second")
        self.assertEqual(
            GRAMMAR_BACKEND_REGISTRY["dup"](None, None, None, None), "second"
        )

ReasonerGrammarBackend demonstrates wrapper composition, but the custom-registry early return in the shown factory may bypass the later wrapper. A third-party backend does not automatically inherit reasoning support merely because it implements BaseGrammarBackend. Verify the pinned factory path or supply equivalent wrapping explicitly; test entry/exit of reasoning blocks, per-request grammar isolation and fail-closed errors.

Consumption is via GrammarManager, which owns the backend and a grammar_queue of requests whose grammars are still compiling. Compilation runs on a ThreadPoolExecutor and is cached by (key_type, key_string):

    def get_cached_or_future_value(
        self, key: Tuple[str, str], require_reasoning: bool
    ) -> Tuple[BaseGrammarObject | Future[BaseGrammarObject], bool]:
        value = self.cache.get(key)
        if value:
            copied_value = value.copy()
            copied_value.maybe_init_reasoning(require_reasoning)
            return copied_value, True
        value = self.executor.submit(self._init_value_dispatch, key, require_reasoning)
        return value, False

    def set_cache(self, key: Tuple[str, str], value: BaseGrammarObject):
        self.cache[key] = value

On a miss, GrammarManager.process_req_with_grammar parks the request; get_ready_grammar_requests() later releases it after an all-gather across the DP/TP group so every rank agrees which grammars finished (python/sglang/srt/constrained/grammar_manager.py:L184-L197). Your backend's copy() must therefore be cheap and must not share mutable FSM state — the default BaseGrammarObject.copy() returns self (base_grammar_backend.py:L109-L110), correct only for a stateless object.

§9.4 owns routing policy — cache-aware routing, the approximate cache model the gateway keeps, the Rust/Python max_tree_size default disagreement, and the fact that nothing tells the gateway when the engine evicts. This section is about the router as a thing you deploy: what it is built from, how it ships, how it finds workers.

Start with the manifest, because it answers the shipping question outright:

[workspace]
members = ["bindings/python"]
exclude = ["bindings/golang", "examples"]

[package]
name = "sgl-model-gateway"
version = "0.3.2"
edition = "2021"

[features]
default = ["grpc-client"]
grpc-client = []
grpc-server = []

vendored-openssl = ["openssl/vendored"]

[lints.rust]
unused_qualifications = "warn"

[lib]
name = "smg"
crate-type = ["rlib"]

[[bin]]
name = "sgl-model-gateway"
path = "src/main.rs"

[[bin]]
name = "smg"
path = "src/main.rs"

[[bin]]

It is both. The crate builds three identically-sourced binaries (sgl-model-gateway, smg, amg, all from src/main.rs), and the workspace member bindings/python wraps the same library into a wheel with maturin:

[project.scripts]
smg = "sglang_router.cli:main"
amg = "sglang_router.cli:main"
sglang-router = "sglang_router.cli:main"


[tool.maturin]
python-source = "src"
module-name = "sglang_router.sglang_router_rs"

Note the version: the wheel is sglang-router 0.3.2, matching the crate but unrelated to the SGLang engine's version — a separate product on a separate release cadence, also on Docker Hub as lmsysorg/sgl-model-gateway (sgl-model-gateway/README.md:L47-L52). The Python side is a thin PyO3 shim: sglang_router/router.py imports Router and PolicyType from the compiled sglang_router.sglang_router_rs module and maps CLI strings onto Rust enums (sgl-model-gateway/bindings/python/src/sglang_router/router.py:L7-L34). Reading Rust is not a prerequisite for operating it.

Worker discovery has three modes, none of which the engine participates in:

#[derive(Debug, Clone)]
pub struct ServiceDiscoveryConfig {
    pub enabled: bool,
    pub selector: HashMap<String, String>,
    pub check_interval: Duration,
    pub port: u16,
    pub namespace: Option<String>,
    // PD mode specific configuration
    pub pd_mode: bool,
    pub prefill_selector: HashMap<String, String>,
    pub decode_selector: HashMap<String, String>,
    // Bootstrap port annotation specific to mooncake implementation
    pub bootstrap_port_annotation: String,
    // Router node discovery for mesh
    pub router_selector: HashMap<String, String>,
    pub router_mesh_port_annotation: String,
    // When true (IGW mode), also discover selector pods as Regular workers alongside PD workers
    pub igw_mode: bool,
}

Two details an operator needs. The default poll is check_interval: Duration::from_secs(60) (service_discovery.rs:L57), so a dead pod is caught by the health checker long before the discovery loop notices. And the prefill bootstrap port is read from a pod annotation defaulting to "sglang.ai/bootstrap-port" (service_discovery.rs:L63) — the contract between your Kubernetes manifests and the gateway's PD handshake, and an overridable default rather than a constant.

The dashed edge is the honest one, and §9.4 owns why it matters: the gateway's cache-aware policy maintains an approximation of each worker's radix tree, and no engine-side eviction ever reaches it.

The artefact to read if you do not read Rust is mini_lb.py, 462 lines of FastAPI implementing the same PD-dispatch shape. It announces its limits in the constructor:

    def _validate_router_args(self, router_args: RouterArgs):
        logger.warning(
            "\x1b[33mMiniLB is only for debugging purposes, it only supports random policy!\033[0m"
        )

        # NOTE: too many arguments unsupported, just validate some important ones
        if router_args.policy != "random":
            logger.warning("[MiniLB] Overriding policy to random")

Its selection is two random.randint calls (mini_lb.py:L103-L112) and its dispatch is the dual-POST that §9.4 traces — the same request goes to a prefill worker and a decode worker concurrently, and only the decode response reaches the client:

        async with aiohttp.ClientSession(
            timeout=aiohttp.ClientTimeout(
                total=self.timeout
            )  # Add timeout for request reliability
        ) as session:

            tasks = [
                session.post(f"{prefill_server}/{endpoint}", json=prefill_req),
                session.post(f"{decode_server}/{endpoint}", json=decode_req),
            ]

            # Wait for both responses to complete. Prefill should end first.
            prefill_response, decode_response = await asyncio.gather(*tasks)

A hypothetical 1 ms route decision is about 0.09% of 256 times the illustrative 4.48 ms step floor. That compares only selection with generation, not total proxy cost. A gateway can process every SSE frame and every output byte, so socket backpressure, copying, parsing, TLS and bandwidth can affect throughput as well as tail latency. Measure routing selection and streaming relay separately; the model is not a benchmark of this router.

Now the systematic pass, the same one §11.5 ran over vLLM. For each seam: what you implement, how you register, what happens on a name collision.

Five of these deserve a closer look.

Model registration is by convention, not by call

There is no register_model function. import_model_classes walks a package with pkgutil.iter_modules and picks up any module defining EntryClass:

                logger.warning(f"Ignore import error when loading {name}: {e}")
                continue
            if hasattr(module, "EntryClass"):
                entry = module.EntryClass
                if isinstance(
                    entry, list
                ):  # To support multiple model classes in one module
                    for tmp in entry:
                        assert (
                            tmp.__name__ not in model_arch_name_to_cls
                        ), f"Duplicated model implementation for {tmp.__name__}"
                        model_arch_name_to_cls[tmp.__name__] = tmp
                else:
                    assert (
                        entry.__name__ not in model_arch_name_to_cls
                    ), f"Duplicated model implementation for {entry.__name__}"
                    model_arch_name_to_cls[entry.__name__] = entry

The dict key is entry.__name__, the Python class name, which must therefore equal the HF config.json architecture string because resolve_model_cls looks up architectures[0] directly. Out-of-tree, you point one environment variable at your package — the only path in the tree that opts into overwriting:

ModelRegistry = _ModelRegistry()
ModelRegistry.register("sglang.srt.models")

if external_pkg := envs.SGLANG_EXTERNAL_MODEL_PACKAGE.get():
    ModelRegistry.register(external_pkg, overwrite=True)

Note also the logger.warning(f"Ignore import error when loading {name}") two lines above: a model module that fails to import is skipped, not fatal, unless strict=True. The symptom is "Model architectures [...] are not supported for now" with a long list that omits yours. Grep the log for that warning first. And import_model_classes is @lru_cache()d on the package name, so a second registration of the same package in one process is a no-op.

The attention registry is two registrations, not one

ATTENTION_BACKENDS = {}


def register_attention_backend(name):
    def decorator(fn):
        ATTENTION_BACKENDS[name] = fn
        return fn

    return decorator

The registered value is a factory taking the model runner, not a class, which is why in-tree entries can branch on runner state — create_flashinfer_backend returns FlashInferAttnBackend or FlashInferMLAAttnBackend depending on runner.use_mla_backend, and allocates a CUDA stream when EAGLE is on (attention_registry.py:L41-L66). Lookup raises ValueError(f"Invalid attention backend: {backend_str}") (python/sglang/srt/model_executor/model_runner_components/attention_backend_setup.py:L254-L257).

But --attention-backend's argparse choices list is a separate module constant, ATTENTION_BACKEND_CHOICES at server_args.py:L181, consumed at three call sites (L1717, L1726, L1735). Registering the factory without also calling add_attention_backend_choices (server_args.py:L405-L406) gets you rejected by argparse before the engine starts. Three further list-extenders sit beside it (server_args.py:L401-L419), and the comment above the chunked-prefix list states the contract: "Out-of-tree platforms may extend this list … before ServerArgs construction" (server_args.py:L211-L214). This is a related registration-order hazard, but vLLM quantization rejects unknown methods later in configuration validation rather than through argparse choices. The broader ordering issue §11.5 found in vLLM's quantization path, reproduced in a different registry.

The speculative registry is the strictest thing in either engine

def register_algorithm(
    name: str,
    *,
    supports_overlap: bool = False,
    validate_server_args: Optional[ServerArgsValidator] = None,
    spec_class: Type[CustomSpecAlgo] = CustomSpecAlgo,
) -> Callable[[WorkerFactory], WorkerFactory]:
    """Return a decorator that registers a plugin algorithm under ``name``.

    Pass a ``spec_class`` subclass of ``CustomSpecAlgo`` to override any
    ``is_*()`` / ``supports_*()`` / ``create_worker`` method.
    """
    upper = name.upper()
    if upper in _reserved_names():
        raise ValueError(
            f"'{upper}' is a reserved speculative algorithm name; cannot be re-registered."
        )
    if upper in _REGISTRY:
        raise ValueError(f"Speculative algorithm '{upper}' already registered.")
    _assert_custom_spec_algo_conforms(spec_class)

    def decorator(factory: WorkerFactory) -> WorkerFactory:

Three guards in order. _reserved_names() is derived lazily from the SpeculativeAlgorithm enum plus {"NEXTN"}, so a new built-in becomes reserved without anyone editing a second list (spec_registry.py:L168-L180). Duplicates raise. _assert_custom_spec_algo_conforms reflects over vars(SpeculativeAlgorithm) for every is_* / supports_* method and raises TypeError if your class is missing one — with a comment recording that this is exactly how is_some and is_frozen_kv_mtp silently went missing before the guard existed (spec_registry.py:L184-L201). The reason: CustomSpecAlgo duck-types the enum so the dozens of if spec_algorithm.is_eagle(): branches through the scheduler and model runner keep working for a plugin algorithm. §6.6 covers the resulting zoo; this is why adding to it does not break those branches.

Quantization has no registry at all

QUANTIZATION_METHODS is a module dict built from in-tree imports (python/sglang/srt/layers/quantization/__init__.py:L143), and get_quantization_config raises ValueError(f"Invalid quantization method: {quantization}. Available methods: ...") on an unknown name. The only out-of-tree route is a platform plugin:

    if current_platform.is_out_of_tree():
        config = current_platform.get_quantization_config(quantization)

        # If the platform has a quantization config, use it else use the default
        if config is not None:
            return config

    return QUANTIZATION_METHODS[quantization]

vLLM has @register_quantization_config and SGLang does not; SGLang has register_grammar_backend and vLLM does not. Neither project set out to build a coherent plugin architecture — both grew seams where somebody needed one.

The config surface: arg_groups declares, it does not mutate

python/sglang/srt/arg_groups/ is the newest and most disciplined of the seams. Its module docstring states the rule: model-identity adjustments are "DECLARED here and materialized onto server_args at the end of __post_init__ (gate order, last writer wins) — model code never mutates ServerArgs fields imperatively" (arg_groups/overrides.py:L15-L20). Two declaration forms exist: a constant dict MODEL_OVERRIDES, and a decorator for derived cases:

def register_model_override(architecture: str):
    """Register a derived-override provider for ``architecture``.

    The decorated callable receives ``(server_args, hf_config)``, must not
    mutate either, and returns a ``{field: resolved_value}`` dict (possibly
    empty when nothing applies). Providers needing derived model data beyond
    the HF config go through ``server_args.get_model_config()`` (cached,
    read-only) — never anything mutating.
    """

    def decorator(fn: Callable[..., dict]) -> Callable[..., dict]:
        _MODEL_OVERRIDE_FNS.setdefault(architecture, []).append(fn)
        return fn

    return decorator

The collision policy is a third kind. collect_model_override_declarations pins the order exactly: the constant entry first, then exact-keyed callables in registration order, then matching predicate-keyed callables, "last writer wins downstream in the gate" (overrides.py:L354-L360). Everyone runs. If your provider and an in-tree family provider both set attention_backend, whichever registered later wins, and the source is recorded by fn.__qualname__ so the log names it. This is the one SGLang seam where a collision is designed rather than tolerated.

The escape hatch: hooks on dotted paths

SGLang's general plugin mechanism is not a set of registries at all. It is a monkey-patch framework with four hook types, keyed on any fully-qualified name in the codebase:

class HookType(Enum):
    """Types of hooks that can be applied to functions or classes."""

    BEFORE = "before"  # Execute before original; can modify args
    AFTER = "after"  # Execute after original; can modify return value
    AROUND = "around"  # Wrap original; full control over execution
    REPLACE = "replace"  # Replace the original function or class entirely

The module docstring's own example targets "sglang.srt.managers.scheduler.Scheduler.schedule" (hook_registry.py:L15-L21). Two entry-point groups feed it — PLATFORM_PLUGINS_GROUP = "sglang.srt.platforms" and GENERAL_PLUGINS_GROUP = "sglang.srt.plugins" (python/sglang/srt/plugins/__init__.py:L27-L29) — and load_plugins() runs each plugin then calls HookRegistry.apply_hooks(). Registering a second REPLACE on one target logs a warning naming both plugins and their distributions, and keeps the last (hook_registry.py:L120-L136). The timing discipline is the same one vLLM enforces — load before config is built:

        # Ensure plugins are loaded before ServerArgs construction,
        # so hooks on ServerArgs.__post_init__ fire correctly.
        load_plugins()

It is called again at the top of run_scheduler_process — "Load plugins so hooks can override Scheduler and its dependencies" (python/sglang/srt/managers/scheduler.py:L5124-L5125) — and defensively in _launch_subprocesses (engine.py:L1086-L1088), once per process, guarded by a module-level _plugins_loaded flag, exactly as vLLM's load_general_plugins() is. Same lesson as §11.5: the registry dicts are per-process, so a plugin must be re-entrant and side-effect free.

The hook registry makes the naive answer "everything is extensible, you can patch any dotted path". That is true and it is not a plan. Here is what has no versioned seam.

That last card mirrors vLLM, where exactly one interface — ModelRegistry.register_model — carries a published guarantee. SGLang guarantees a different one:

            raise TypeError(
                f"Serve backend {name!r} factory returned {type(backend).__name__}; "
                "expected sglang.cli.serve_backends.ServeBackend."
            )
        if backend.api_version != SERVE_BACKEND_API_VERSION:
            raise RuntimeError(
                f"Serve backend {name!r} uses API version {backend.api_version}; "
                f"this SGLang release requires version {SERVE_BACKEND_API_VERSION}."
            )

SERVE_BACKEND_API_VERSION = 1 at serve_backends.py:L24, and the docs say why you must hard-code your side: "Declare the API version implemented by your extension as a literal. Do not copy SGLang's current version constant at runtime; a fixed value lets a future SGLang release detect an older plugin contract" (developer_guide/serve_backend_plugins.mdx:L149). This is the seam for plugging a different runtime into sglang serve MODEL --model-type my_runtime, and the only place in either project where a version number does compatibility work.

The router's equivalent of a plugin is not a Rust trait — it is a WebAssembly module uploaded at runtime over HTTP into a middleware layer:

        .route("/wasm", post(add_wasm_module))
        .route("/wasm/{module_uuid}", delete(remove_wasm_module))
        .route("/wasm", get(list_wasm_modules))

Admin routes behind auth, running on wasmtime (sgl-model-gateway/Cargo.toml:L131). It is request-level middleware, not routing policy: you can rewrite, reject, or annotate a request without a Rust rebuild, but you still cannot choose a worker.

Tests live in two trees with different meanings. test/registered/ is CI-discovered: every file there calls a registration function at module level, and run_suite.py collects est_time, stage, and runner_config by AST parsing, so those arguments must be literals (test/README.md:L62-L73). Unit tests mirror the source tree — srt/mem_cache/radix_cache.py maps to unit/mem_cache/test_radix_cache_unit.py (developer_guide/contribution_guide.mdx:L44-L50). The grammar-registry test quoted earlier registers itself with register_cpu_ci(2.0, "base-a-test-cpu"): two seconds, CPU only, the earliest stage. A registry test needs no GPU, and putting it in the CPU stage means it gates every PR rather than a nightly. That is the pattern to copy for any seam you add.

test/manual/ is explicitly outside CI, which is where the frontend language's tests live.

CI runs three sequential stages — A (pre-flight, ~3 min), B (basic, ~30 min), C (advanced, ~30 min) (test/README.md:L10). The gate that surprises newcomers is that CI does not run on a PR at all unless it carries the run-ci label, and only users in .github/CI_PERMISSIONS.json can apply it. Authors can always comment /rerun-failed-ci on their own PR; /tag-and-rerun-ci is the command for a fresh one, because /tag-run-ci-label alone only affects future commits (developer_guide/contribution_guide.mdx:L120-L128). Selective reruns skip building a PR-local kernel wheel, so a change under python/sglang/kernels/aot/ needs the full workflow.

Local hygiene is pre-commit: pip3 install pre-commit; pre-commit install; pre-commit run --all-files, re-run once if the first pass fixes things (contribution_guide.mdx:L27-L35). Docs links are checked with Mintlify — cd docs && mint broken-links --check-anchors --check-redirects. And the code-style section is unusually specific about the cost model you are writing into: "SGLang is a runtime, and most of your code runs on the critical path for every request … A common pattern is some runtime checks in the model forward pass … Please cache the result as a single boolean value in __init__" (contribution_guide.mdx:L163-L166). Against the 4.48 ms Llama-3-8B decode floor, a per-layer Python predicate evaluated 32 times per step is real money.

• Registering an attention backend and forgetting argparse. The factory dict and the CLI choices list are different module-level objects in different files. You get argparse: error: argument --attention-backend: invalid choice and no hint that your registration succeeded. Call add_attention_backend_choices([...]) from the same plugin, before ServerArgs is constructed.
• Shadowing a built-in grammar backend. create_grammar_backend checks GRAMMAR_BACKEND_REGISTRY before the if name == "outlines" chain, and the registry silently accepts "xgrammar". Two plugins that both pick a friendly name will silently resolve to whichever imported last. Namespace your name.
• Assuming fork() shares KV directly. It copies a Python string. If the tree evicted the prefix between the commit and the branches, both branches re-prefill and the program is slower than one request. SglCommitLazy is a hint, not a lock — §12.3 owns the refcount rules that decide whether the node survives.
• A model module that fails to import disappears silently. import_model_classes logs "Ignore import error when loading {name}" at WARNING and continues, so a typo'd import in your architecture file surfaces as "Model architectures [...] are not supported for now" with a long list that omits yours. Set strict=True or grep for that warning.
• Two model packages, one env var. SGLANG_EXTERNAL_MODEL_PACKAGE takes a single package name; use an aggregate package that exposes/registers both vendors' classes, resolve name collisions explicitly, and ensure every worker imports the aggregate. Cached discovery does not prohibit this composition.
• MiniLB in production. It forces --policy random and raises ValueError("MiniLB only supports PD disaggregation mode") outside PD mode (mini_lb.py:L64-L65). It exists to make PD dispatch readable, not to serve traffic.
• Building on sglang.lang in 2026. The package README says deprecated and no CI covers it. Migration is mechanical: the shared prefix is the payload, so send it as the prompt of $n$ independent requests and let RadixAttention do what the interpreter was arranging.

# 1. Enumerate every registration function in the tree, in one pass.
grep -rn "^def register_\|^    def register(" python/sglang/srt/ python/sglang/cli/ | grep -v test

# 2. Print the collision policy of each, side by side. This is the chapter's table, from source.
sed -n '55,69p'   python/sglang/srt/mem_cache/registry.py        # raises
sed -n '222,243p' python/sglang/srt/speculative/spec_registry.py # raises, twice, plus conformance
sed -n '34,39p'   python/sglang/srt/layers/attention/attention_registry.py  # overwrites
sed -n '347,348p' python/sglang/srt/constrained/base_grammar_backend.py     # overwrites
sed -n '96,100p'  python/sglang/srt/arg_groups/overrides.py                 # appends

# 3. Run the grammar-registry unit test. No GPU, no weights, about two seconds.
python3 test/registered/unit/constrained/test_base_grammar_backend.py

# 4. Prove the frontend has no privileged channel: same two prompts, no DSL.
python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-0.5B-Instruct --port 30000 &
P="Here are two tips for staying healthy: 1. Balanced Diet. 2. Regular Exercise."
curl -s localhost:30000/generate -d "{\"text\": \"$P\", \"sampling_params\": {\"max_new_tokens\": 0}}" > /dev/null
curl -s localhost:30000/generate -d "{\"text\": \"$P Expand tip 1:\", \"sampling_params\": {\"max_new_tokens\": 32}}"
curl -s localhost:30000/generate -d "{\"text\": \"$P Expand tip 2:\", \"sampling_params\": {\"max_new_tokens\": 32}}"

# 5. Confirm the shared prefix was reused rather than re-prefilled.
curl -s localhost:30000/server_info | python3 -m json.tool | grep -i cache

# 6. See the router's two shipping forms.
cd sgl-model-gateway && grep -A3 '\[\[bin\]\]' Cargo.toml
grep -A4 'tool.maturin' bindings/python/pyproject.toml

Step 4 is the one that teaches. Those three curls are what a two-way fork compiles down to: a zero-token commit followed by two full-text prompts. Compare the cache hit rate against the same two prompts sent without the commit and you have measured the frontend language's entire contribution.

• Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs — the paper the frontend comes from. Read it for the motivation, then python/sglang/lang/interpreter.py for what actually shipped; the interpreter's coordination is thinner than the paper's framing suggests, and python/sglang/README.md:L8 is where that story ends.
• developer_guide/serve_backend_plugins.mdx — the only versioned extension contract in the project, end to end: entry point, ServeBackend, ServeBackendDetection, and the automatic-routing rules for --model-type auto.
• developer_guide/contribution_guide.mdx and test/README.md — CI stages, the run-ci label, register_*_ci, and why est_time must be a literal.
• sgl-model-gateway/README.md — 49 KB of operator documentation: build modes, IGW, PD flags, the worker registration API, the Prometheus metric list. Its vLLM counterpart is rust/README.md, stating the same thesis from a project that still has a Python frontend.
• references/frontend/frontend_tutorial.mdx and choices_methods.mdx — the frontend as the project still documents it, including the three select scoring methods (token_length_normalized, greedy_token_selection, unconditional_likelihood_normalized) that python/sglang/lang/choices.py implements.
• §6.5 for the grammar mechanism and the jump-forward story, §9.4 for routing policy, §12.3 for the cache the frontend was aiming at, and §11.5 for the vLLM half of this audit. §13.1 puts the two side by side.