18 min read7 sections

Lab 09 · for chapter 06-05

Structured decoding overhead

Price the grammar mask: throughput with and without a JSON schema, and how it scales with batch size.

Download lab script · CPU reference checks · Environment preflight. Install the schema-validation requirements before a live sweep; the saved result separates parseability, schema validity and truncation.

Constrained decoding is CPU work bolted onto a GPU pipeline. This lab puts a number on it: throughput with and without a JSON schema, how the gap moves with batch size, and how it moves with schema complexity — then separates the one-off compile cost from the per-token mask cost, because they have different fixes and people conflate them constantly.

Hardware

One GPU that holds an 8B model — 24 GB — and, unusually for this book, a CPU you should describe in your results. The grammar mask is filled on the host, one row per constrained request per step, and vLLM's parallel fill path caps its worker count at half your core count or eight, whichever is smaller. A 64-core server and an 8-core workstation will give you different answers to the same question at high batch, and the difference is not the GPU. Print nproc alongside your numbers.

Not executed here

run.py was written against the backend names, request schemas, and mask code cited below, and its argument handling and arithmetic were exercised, but it has not been run against a live engine — no GPU was available while writing. There is no measured throughput number anywhere on this page. Predictions are labelled as arithmetic; everything else is cited to source.

§1

What you measure

The overhead has three components, and they show up in different metrics:

once per schema

Compilation

Turning a JSON Schema into an FSM or pushdown automaton. Hits TTFT of the first request carrying that schema, then is cached. Measured in milliseconds to seconds, depending on how nasty the schema is.

every token

Mask fill

Asking the automaton which of ~128k tokens are legal right now, and packing the answer into a bitmask. Host-side, per request, per step. This is the throughput tax.

every token

Copy and apply

H2D copy of the bitmask and one elementwise kernel that sets illegal logits to −∞. Small, and the part everyone assumes is the expensive one.

Size the third one first, because it turns out to be the cheap one and knowing that redirects your attention. The bitmask packs 32 vocabulary entries per int32 — you can read the packing straight out of SGLang's Triton apply kernel, which shifts a loaded word right by 0…31 and tests the low bit:

python/sglang/kernels/ops/grammar/bitmask_ops.py:L65-L77 SGLang

        batch_id = row_id if indices_ptr is None else tl.load(indices_ptr + row_id)
        offsets = block_offset + tl.arange(0, BLOCK_SIZE)
        bitmask_offsets = block_offset // 32 + tl.arange(0, BLOCK_SIZE // 32)
        vocab_mask = offsets < vocab_size
        packed_bitmask_mask = bitmask_offsets < bitmask_strides
        packed_bitmask = tl.load(
            bitmask_ptr + batch_id * bitmask_strides + bitmask_offsets,
            packed_bitmask_mask,
        )
        bitmask = ((packed_bitmask[:, None] >> (tl.arange(0, 32)[None, :])) & 1) == 0
        bitmask = bitmask.reshape(BLOCK_SIZE)

        tl.store(

So one row is $\lceil V/32 \rceil$ words $= V/8$ bytes. For Llama-3's 128,256-token vocabulary that is 4,008 int32 = 16,032 bytes per constrained request per step. At 256 concurrent constrained requests: 4.1 MB copied host-to-device per decode step. If a decode step takes 20 ms, that is 205 MB/s over a link that does tens of GB/s. Derived; arithmetic. The copy is not your problem. The 256 host-side fill calls that produced those bytes might be.

§2

The backends that actually exist

Four names circulate; the two engines accept overlapping but different subsets, and one commonly cited name (llguidance) is a valid value on exactly one of them. vLLM:

vllm/config/structured_outputs.py:L12-L24 vLLM

StructuredOutputsBackend = Literal[
    "auto", "xgrammar", "guidance", "outlines", "lm-format-enforcer"
]


@config
class StructuredOutputsConfig:
    """Dataclass which contains structured outputs config for the engine."""

    backend: StructuredOutputsBackend = "auto"
    """Which engine will be used for structured outputs (e.g. JSON schema,
    regex, etc) by default. With "auto", we will make opinionated choices
    based on request contents and what the backend libraries currently support,

Note that vLLM spells the llguidance backend guidance, and that there is no none. It is selected through a nested config object rather than a flat flag:

vllm/engine/arg_utils.py:L1672-L1674 vLLM

        vllm_group.add_argument(
            "--structured-outputs-config", **vllm_kwargs["structured_outputs_config"]
        )

which the flexible parser lets you write either as JSON or dotted, and the two are exactly equivalent:

vllm/utils/argparse_utils.py:L117-L125 vLLM

    _json_tip: str = (
        "When passing JSON CLI arguments, the following sets of arguments "
        "are equivalent:\n"
        '   --json-arg \'{"key1": "value1", "key2": {"key3": "value2"}}\'\n'
        "   --json-arg.key1 value1 --json-arg.key2.key3 value2\n\n"
        "Additionally, list elements can be passed individually using +:\n"
        '   --json-arg \'{"key4": ["value3", "value4", "value5"]}\'\n'
        "   --json-arg.key4+ value3 --json-arg.key4+='value4,value5'\n\n"
    )

SGLang's is a flat flag with a closed list, and it does have a none:

python/sglang/srt/server_args.py:L245, L1749-L1756 SGLang

GRAMMAR_BACKEND_CHOICES = ["xgrammar", "outlines", "llguidance", "none"]
# ...
    grammar_backend: A[
        Optional[str],
        Arg(
            help="Choose the backend for grammar-guided decoding.",
            choices=GRAMMAR_BACKEND_CHOICES,
        ),
        NS("exec.kernel"),
    ] = None

none does not mean "no overhead on constrained requests" — it means there is no backend at all, and structured output is unavailable:

python/sglang/srt/constrained/base_grammar_backend.py:L415-L424 SGLang

    elif name == "none":
        if get_serving().enable_strict_thinking:
            raise ValueError(
                "--enable-strict-thinking requires a grammar backend that supports "
                "token filtering, but grammar_backend='none' was specified. Use "
                "--grammar-backend xgrammar or another backend that supports token "
                "filtering."
            )
        return None

So the "off" arm of this lab is the same server with no schema on the request, not a server with the backend disabled. Changing the server between arms changes more than one thing.

The name mismatch is not cosmetic: vLLM's guidance backend is llguidance, imported under that name at the top of the module that implements it.

vllm/v1/structured_output/backend_guidance.py:L25-L32 vLLM

if TYPE_CHECKING:
    import llguidance
    import llguidance.hf as llguidance_hf
    import llguidance.torch as llguidance_torch
else:
    llguidance = LazyLoader("llguidance", globals(), "llguidance")
    llguidance_hf = LazyLoader("llguidance.hf", globals(), "llguidance.hf")
    llguidance_torch = LazyLoader("llguidance.torch", globals(), "llguidance.torch")

Backend names at the pinned SHAs. Read this before writing a flag from memory.

Library	vLLM spelling	SGLang spelling
XGrammar	`xgrammar`	`xgrammar`
llguidance / Guidance	`guidance`	`llguidance`
Outlines	`outlines`	`outlines`
LM Format Enforcer	`lm-format-enforcer`	not present
pick for me	`auto` (the default)	unset (the default)
no backend at all	not present	`none`

§3

Where the cost actually lands

vLLM overlaps the mask fill with the forward pass

This is the single most important structural fact for interpreting your results, and it is four lines of the engine core. The model is launched non-blocking, the bitmask is built while the GPU is busy, and only then does the step join:

vllm/v1/engine/core.py:L593-L603 vLLM

        scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())
        future = self.model_executor.execute_model(scheduler_output, non_block=True)
        grammar_output = self.scheduler.get_grammar_bitmask(scheduler_output)
        with (
            self.capture_iteration_details(scheduler_output) as iteration_details,
            self.log_error_detail(scheduler_output),
        ):
            model_output = future.result()
            if model_output is None:
                model_output = self.model_executor.sample_tokens(grammar_output)

The consequence is a prediction you can test: at small batch, where the forward pass is long relative to a handful of mask fills, structured decoding should cost close to nothing. The overhead only becomes visible when total fill time exceeds forward time — which happens as batch grows, because fill cost is linear in the number of constrained requests while forward time is nearly flat in the memory-bound regime.

vLLM parallelises the fill, but only above 128

The thread pool for mask filling is constructed only when max_num_seqs exceeds a hard-coded threshold of 128:

vllm/v1/structured_output/__init__.py:L61-L69 vLLM

        max_batch_size = self.vllm_config.scheduler_config.max_num_seqs
        self.fill_bitmask_parallel_threshold = 128
        if self.fill_bitmask_parallel_threshold < max_batch_size:
            self.fill_bitmask_parallel_batch_size = 16
            # Use:
            # - at least 1 CPU
            # - at most half the number of CPUs or 8, whichever is less
            max_workers = max(1, min(multiprocessing.cpu_count() // 2, 8))
            self.executor_for_fillmask = ThreadPoolExecutor(max_workers=max_workers)

and it is used only when this step has more than 128 constrained requests and speculative decoding is off:

vllm/v1/structured_output/__init__.py:L250-L256, L280-L282 vLLM

        # Optimized parallel filling of bitmasks for
        # non-spec, large-batch-size cases
        if (
            len(structured_output_request_ids) > self.fill_bitmask_parallel_threshold
            and max_num_spec_tokens == 0
        ):
            promises = []
# ... chunking into groups of fill_bitmask_parallel_batch_size ...
        else:
            # Fallback to serial filling of bitmasks for small-batch-size cases
            for req_id in structured_output_request_ids:

Three testable predictions fall out, and none of them is guessable from documentation:

Sweeping concurrency past 128 should produce a visible knee — per-step fill cost stops growing linearly and drops by up to the worker count.
Running with --max-num-seqs 128 exactly, the parallel path can never fire, because the executor is not even constructed (128 < 128 is false). Set it to 129 and the same workload behaves differently.
Turning on speculative decoding disables the parallel path at every batch size. If you are measuring spec decode and structured output together, you are on the serial path no matter how many CPUs you bought.

SGLang fills serially, except on llguidance

SGLang's batched fill is a plain Python loop in the base class, and the XGrammar object does not override it:

python/sglang/srt/constrained/base_grammar_backend.py:L88-L94 SGLang

    @staticmethod
    def fill_vocab_mask_batched(
        entries: List[GrammarRow], vocab_mask: torch.Tensor
    ) -> None:
        """Fill listed rows, leaving unlisted rows untouched."""
        for entry in entries:
            entry.grammar.fill_vocab_mask(vocab_mask, entry.row)

The llguidance backend does override it, with a native batched call:

python/sglang/srt/constrained/llguidance_backend.py:L151-L159 SGLang

    @staticmethod
    def fill_vocab_mask_batched(
        entries: List[GrammarRow], vocab_mask: torch.Tensor
    ) -> None:
        """Use the native fill when every entry is a plain llguidance grammar."""
        if all(isinstance(entry.grammar, GuidanceGrammar) for entry in entries):
            fill_token_bitmask_batched(entries, vocab_mask)
            return
        BaseGrammarObject.fill_vocab_mask_batched(entries, vocab_mask)

That gives you a clean A/B inside one engine: --grammar-backend xgrammar versus --grammar-backend llguidance, same model, same schema, same workload. Any divergence that grows with batch size is the batched fill earning its keep. The call site skips finished and terminated rows, so the row count is the number of live constrained requests, not the batch size:

python/sglang/srt/sampling/sampling_batch_info.py:L253-L264 SGLang

        # Rows omitted here (finished / terminated / non-grammar requests) retain
        # the freshly allocated buffer's unconstrained value.
        entries = [
            GrammarRow(row=row, grammar=grammar)
            for row, grammar in enumerate(self.grammars)
            if grammar and not grammar.finished and not grammar.is_terminated()
        ]
        first_grammar.fill_vocab_mask_batched(entries, vocab_mask)

        # Move the mask to the device if needed
        vocab_mask = first_grammar.move_vocab_mask(vocab_mask, self.device)
        self.grammar_mask = GrammarMask(first_grammar, vocab_mask)

Compilation is a queue, and it is cached

SGLang parks a request in a grammar queue until its automaton is built, polling on a 5 ms interval:

python/sglang/srt/constrained/grammar_manager.py:L204-L225 SGLang

            # Poll for ready requests
            start_time = time.perf_counter()
            while time.perf_counter() - start_time < self.SGLANG_GRAMMAR_POLL_INTERVAL:
                for i, req in enumerate(self.grammar_queue):
                    if i in ready_req_idxs:
                        continue

                    if (
                        req.finished() or req.grammar is None
                    ):  # It is aborted by AbortReq
                        ready_req_idxs.add(i)
                        continue

                    assert isinstance(req.grammar, futures.Future), f"{req=}"
                    if req.grammar.done():
                        ready_req_idxs.add(i)

                if len(ready_req_idxs) == len(self.grammar_queue):
                    break

                # Sleep a bit to avoid busy waiting
                time.sleep(self.SGLANG_GRAMMAR_POLL_INTERVAL / 10)

python/sglang/srt/environ.py:L374-L376 SGLang

    SGLANG_GRAMMAR_POLL_INTERVAL = EnvFloat(0.005)
    SGLANG_GRAMMAR_MAX_POLL_ITERATIONS = EnvInt(10000)
    SGLANG_DISABLE_OUTLINES_DISK_CACHE = EnvBool(False)

The queue depth is exposed as a metric, so you can watch it fill during a cold burst:

python/sglang/srt/observability/metrics_collector.py:L280-L285 SGLang

        self.num_grammar_queue_reqs = Gauge(
            name="sglang:num_grammar_queue_reqs",
            documentation="The number of requests in the grammar waiting queue.",
            labelnames=labels.keys(),
            multiprocess_mode="mostrecent",
        )

vLLM compiles on a thread pool too, and hands XGrammar a compiler cache whose size is an environment variable:

vllm/v1/structured_output/backend_xgrammar.py:L66-L71 vLLM

        self.compiler = xgr.GrammarCompiler(
            tokenizer_info,
            max_threads=8,
            cache_enabled=True,
            cache_limit_bytes=vllm.envs.VLLM_XGRAMMAR_CACHE_MB * 1024 * 1024,
        )

That cache is why a naive "cost of structured decoding" benchmark reports almost nothing. Send the same schema a thousand times and you compile once. vLLM's own structured-output benchmark has a dataset mode built specifically to defeat it, and the comment says so out loud:

benchmarks/benchmark_serving_structured_output.py:L161-L169 vLLM

        if args.dataset == "json-unique":
            json_schemas = [copy.deepcopy(schema) for _ in range(args.num_prompts)]
            for i in range(len(json_schemas)):
                if "properties" not in json_schemas[i]:
                    json_schemas[i]["properties"] = {}
                json_schemas[i]["properties"][f"__optional_field_{uuid.uuid4()}"] = {
                    "type": "string",
                    "description": "An unique optional field to avoid cached schemas",
                }

Run json and json-unique as separate arms. The difference between them is the compile cost, isolated.

§4

Running it

The engines' benchmark, which is vLLM-shaped

vLLM ships a purpose-built structured-output serving benchmark with exactly the knob this lab needs — the fraction of requests carrying a schema:

benchmarks/benchmark_serving_structured_output.py:L1072-L1077 vLLM

    parser.add_argument(
        "--structured-output-ratio",
        type=float,
        default=1.0,
        help="Ratio of Structured Outputs requests",
    )

benchmarks/benchmark_serving_structured_output.py:L912-L916 vLLM

    parser.add_argument(
        "--dataset",
        default="json",
        choices=["json", "json-unique", "grammar", "regex", "choice", "xgrammar_bench"],
    )

It will not measure SGLang

This benchmark accepts --backend sglang, but it attaches the schema using vLLM's own request field, not the portable OpenAI one. Against an SGLang server that field is not a constraint — you would be benchmarking unconstrained generation and calling it structured. This is why run.py exists and sends response_format instead.

benchmarks/benchmark_serving_structured_output.py:L484-L489 vLLM

    def prepare_extra_body(request) -> dict:
        extra_body = {}
        # Add the schema to the extra_body
        extra_body["structured_outputs"] = {}
        extra_body["structured_outputs"][request.structure_type] = request.schema
        return extra_body

Both engines do accept the OpenAI-standard response_format with a json_schema, and both funnel it to the same place internally. vLLM rewrites it into its own json constraint:

vllm/entrypoints/openai/engine/protocol.py:L218-L227 vLLM

    if response_format is None or response_format.type == "text":
        return structured_outputs

    overrides: dict[str, Any]
    if response_format.type == "json_object":
        overrides = {"json_object": True}
    elif response_format.type == "json_schema":
        json_schema = response_format.json_schema
        assert json_schema is not None
        overrides = {"json": json_schema.json_schema}

and SGLang normalises a bare schema into the nested json_schema shape before doing the same:

python/sglang/srt/entrypoints/openai/protocol.py:L1044-L1055 SGLang

        response_format = values.get("response_format")
        if not response_format:
            return values

        if response_format.get("type") != "json_schema":
            return values

        schema = response_format.pop("schema", None)
        json_schema = response_format.get("json_schema")

        if json_schema:
            return values

Commands

shell — servers shell

$ vllm serve meta-llama/Meta-Llama-3-8B-Instruct --port 8000 \
    --structured-outputs-config.backend xgrammar --max-num-seqs 256

$ vllm serve meta-llama/Meta-Llama-3-8B-Instruct --port 8000 \
    --structured-outputs-config '{"backend": "guidance"}' --max-num-seqs 256

$ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct \
    --port 30000 --grammar-backend xgrammar --enable-metrics

$ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct \
    --port 30000 --grammar-backend llguidance --enable-metrics

shell — labs/09-structured-decoding-overhead shell

$ python3 run.py schemas                 # print the four complexity tiers and their sizes
$ python3 run.py --help                  # every knob

# the core measurement: same server, schema off then on, at several batch sizes
$ python3 run.py sweep --engine vllm --base-url http://127.0.0.1:8000 \
    --concurrency 1 8 32 128 129 256 --schema none flat --requests 2048

# schema complexity at fixed batch
$ python3 run.py sweep --engine sglang --base-url http://127.0.0.1:30000 \
    --concurrency 64 --schema none flat nested deep --requests 256

# isolate compile cost: same schema every time vs a fresh one every time
$ python3 run.py sweep --engine vllm --base-url http://127.0.0.1:8000 \
    --concurrency 1 --schema nested --requests 64 --unique-schemas

shell — vLLM's own benchmark, for the vLLM-only arms shell

$ python3 benchmarks/benchmark_serving_structured_output.py --backend vllm \
    --model meta-llama/Meta-Llama-3-8B-Instruct --dataset json \
    --structured-output-ratio 0.0 --num-prompts 500 --max-concurrency 128

$ python3 benchmarks/benchmark_serving_structured_output.py --backend vllm \
    --model meta-llama/Meta-Llama-3-8B-Instruct --dataset json \
    --structured-output-ratio 1.0 --num-prompts 500 --max-concurrency 128

$ python3 benchmarks/benchmark_serving_structured_output.py --backend vllm \
    --model meta-llama/Meta-Llama-3-8B-Instruct --dataset json-unique \
    --structured-output-ratio 1.0 --num-prompts 500 --max-concurrency 128

§5

What to expect

Predicted shape of each sweep, from the code above. These are directions and mechanisms, not numbers — the magnitudes are what you are measuring.

Sweep	Predicted shape	Mechanism
Batch 1, schema on vs off	Nearly identical throughput	One fill hides entirely inside the forward pass vLLM launched non-blocking.
Batch 1 → 128, vLLM	Gap widens roughly linearly	Serial fill: cost is $O(\text{batch})$ against a forward time that barely moves.
128 → 129, vLLM	Discontinuity — the gap narrows	Parallel path engages; up to `min(cores/2, 8)` threads in chunks of 16.
Same, with `--max-num-seqs 128`	No discontinuity at all	The executor is never constructed, so the parallel branch is unreachable.
Same, with speculative decoding on	No discontinuity at all	The branch also requires zero speculative tokens.
xgrammar vs llguidance, SGLang	Divergence grows with batch	Only llguidance overrides the serial per-row loop with a native batched fill.
`json` vs `json-unique`	TTFT jumps, throughput mostly does not	Compile cost is per distinct schema and cached; the mask cost is per token regardless.
flat → deeply nested schema	Compile cost grows sharply, fill cost grows mildly	Automaton construction has to walk the whole schema; a mask fill is one automaton query whatever built it, and the bitmask is the same size either way.

Traps that will invalidate a run, in roughly the order they catch people:

You compared two servers instead of two request shapes. Restarting with a different backend changes CUDA graph capture, memory layout, and warmup state. The schema-off arm must be the same process with no response_format on the request.
You reused one schema and concluded compilation is free. It is free the 999 times after the first. Run json-unique or --unique-schemas before saying anything about compile cost.
Your batch never reached the concurrency you asked for. A grammar-constrained response can hit its stop condition sooner than a free-form one, so requests retire faster and the running batch is smaller than you asked for. Check the engine's own running-request count, not your client's in-flight count.
You benchmarked SGLang with vLLM's structured benchmark. The schema silently did nothing. Confirm by diffing outputs: check both JSON parseability and actual JSON Schema validation; invalid output can also result from truncation or unsupported constraints.
Your CPU was the variable. Two machines with the same GPU and different core counts will disagree above batch 128 by design. Report nproc.

§6

Exercises

Compute the per-step host-to-device bitmask traffic for a 256-request batch on a model with a 256,000-token vocabulary. Then compute what fraction of a 100 GB/s PCIe link that uses at a 20 ms step time. Does the copy explain any throughput loss you can measure?
Predict what happens to the vLLM overhead curve when you serve with --max-num-seqs 128 versus --max-num-seqs 129, at an offered concurrency of 200 in both cases. Then run it. Explain the result by quoting two lines of source.
Run the same schema-on workload against SGLang with --grammar-backend xgrammar and with --grammar-backend llguidance, at concurrency 8 and 256. Which arm's advantage grows, and does the growth match the code you read?
Take a schema with one deeply nested $ref cycle and time the first request's TTFT against the tenth's. Then restart the server and repeat with VLLM_XGRAMMAR_CACHE_MB=0. Predict the difference before running.
Read vllm/v1/engine/core.py around the step function and answer: if grammar mask construction took longer than the model forward pass, what would you see in the engine's throughput metrics, and how would you distinguish it from the GPU simply being slow?

Answers

256,000/8 = 32,000 bytes per row; ×256 rows = 8.19 MB per step. At 20 ms per step that is 410 MB/s, or 0.41% of a 100 GB/s link. No: the copy cannot explain a measurable throughput loss, and if you see one you should be looking at the 256 host-side fill calls that produced the buffer, not the transfer. (Derived; arithmetic.)
At --max-num-seqs 128 the overhead grows linearly all the way up and there is no knee, because self.fill_bitmask_parallel_threshold < max_batch_size is 128 < 128, false, so executor_for_fillmask is never created. At 129 the executor exists and the runtime branch len(structured_output_request_ids) > 128 can fire, so the curve bends once your steps actually carry more than 128 constrained requests. The two lines are the constructor guard and the runtime condition, quoted above.
llguidance's advantage grows, because it is the only backend that replaces the per-row Python loop with a native batched fill. At concurrency 8 the loop runs eight times and the difference is noise; at 256 it runs 256 times and the interpreter overhead per row starts to matter. If your measurement shows a constant-factor gap instead of a growing one, suspect that your batch never actually reached 256 — check the engine's running-request count.
Uncached, every request pays compilation, so TTFT is high and flat across all ten. Cached, the first pays and the rest do not, so you see one outlier and nine fast requests. The delta between the first request in each configuration should be small — both compiled — while the delta on requests two through ten is the entire cache benefit. That is the cleanest isolation of compile cost available without a profiler.
You would see decode throughput fall while GPU utilisation also falls, which is the signature that distinguishes it from a slow GPU: a slow GPU is busy, a blocked host is not. The step launches the forward non-blocking and then builds the mask, so if the mask wins the race the GPU finishes and idles waiting for future.result() to be reached. Confirm with a profile — lab 10 — and look for a gap between the end of one forward and the start of the next that scales with the number of constrained requests.

§7

Key takeaways

The overhead is three separate costs: one-off compilation, per-token host-side mask fill, and a per-token copy plus kernel. Any can affect throughput or latency when it reaches the critical path; measure compilation queues, transfer synchronization, fill and mask application separately.
The bitmask is $V/8$ bytes per constrained request per step — 16 KB on Llama-3, a few MB per step at high batch. Low average byte volume does not rule out transfer latency, synchronization or missed overlap as a bottleneck.
vLLM overlaps mask construction with the model forward, so structured decoding is close to free at low batch and stops being free exactly when fill time exceeds forward time.
vLLM's parallel fill has a hard threshold of 128 in two places: the executor is only built when max_num_seqs > 128, and it is only used when a step carries more than 128 constrained requests and speculative decoding is off. Both are testable in an afternoon.
SGLang fills serially on every backend except llguidance, which is the only one that overrides the per-row loop. That is the cleanest single-engine A/B in this lab.
The backend name is not portable: llguidance is guidance on vLLM and llguidance on SGLang, and lm-format-enforcer exists on only one of them.
A schema-cost benchmark that reuses one schema measures nothing about compilation. Defeat the cache deliberately, or say plainly that you did not.