The OpenAI-compatible API surface
vllm/entrypoints/launchers/api_server/routers.pyvllm/entrypoints/openai/chat_completion/serving.pypython/sglang/srt/entrypoints/http_server.pypython/sglang/srt/entrypoints/openai/
a556f3f · sglang 7d89325Nobody chose the OpenAI API. It won because the first generation of LLM tooling was written against it, and by the time open-weight serving mattered, every SDK, every agent framework and every gateway already spoke it. So both vLLM and SGLang implement a specification neither of them wrote, for a model neither of them trained — and the interesting engineering is entirely in the gap between accepting a field and implementing its semantics.
The problem
Here is one JSON body. Nothing exotic:
{"model": "meta-llama/Llama-3-8B-Instruct",
"messages": [{"role": "user", "content": "<7000 tokens of transcript>"}],
"max_tokens": 4096,
"temperature": 0}Send it to vLLM: HTTP 200. The answer stops mid-sentence at 1,192 tokens with finish_reason: "length", because vLLM took the minimum of what you asked for and what fits:
if max_model_len < input_length:
raise ValueError(
f"Input length ({input_length}) exceeds model's maximum "
f"context length ({max_model_len})."
)
model_max_tokens = max_model_len - input_length
platform_max_tokens = current_platform.get_max_output_tokens(input_length)
fallback_max_tokens = (
max_tokens
if max_tokens is not None
else default_sampling_params.get("max_tokens")
)
return min(
val
for val in (
model_max_tokens,
fallback_max_tokens,
override_max_tokens,
platform_max_tokens,
)
if val is not None
)Send the same body to SGLang: HTTP 400, with a message that spells out exactly why:
total_tokens = max_new_tokens + input_token_num
error_msg = (
f"Requested token count exceeds the model's maximum context length "
f"of {self.context_len} tokens. You requested a total of {total_tokens} "
f"tokens: {input_token_num} tokens from the input messages and "
f"{max_new_tokens} tokens for the completion. Please reduce the number "
f"of tokens in the input messages or the completion to fit within the limit."
)
raise ValueError(error_msg)First reduce the preceding request to a valid context budget, for example a 7000-token prompt with max_tokens=512. Then add "n": 4. vLLM returns 400 — n must be 1 when using greedy sampling. SGLang returns 200 with four choices that are byte-identical, because at temperature=0 it rewrites the request to top_k=1 and samples four times from a one-element distribution.
Both servers are "OpenAI-compatible". Both are honest about it. The contract simply does not say what happens here, and two reasonable teams read the silence two different ways. This chapter is about where that silence lives.
Mental model
An OpenAI-compatible server is not one contract. It is three, stacked, and each is owned by a different party:
Routes and status codes
Which paths exist, what a 400 versus a 404 versus a 503 means, how streaming is framed. Owned by the engine. Fully under its control, and the part it gets most right.
Request schema and semantics
Which knobs exist and what they do to sampling. Owned jointly: the field names come from OpenAI, the semantics from the engine's sampler. This is where compatibility is thinnest.
messages to token ids
A Jinja template that ships with the checkpoint, not with the engine. Neither server authored it; both apply it; they resolve it differently.
The third one is the surprise. When you POST a messages array, the string the model actually sees is produced by a template file downloaded from the model repo, evaluated by Jinja, with variables the engine supplies. Change the checkpoint's chat_template.jinja and your server's behaviour changes with no code deployed anywhere. That is a supply chain, and it runs straight through your API surface.
Start with tier one: what is actually mounted.
Figure 1 — the mounted endpoint surface of both engines at the pinned SHAs. vLLM registers routes conditionally on the model's supported tasks; SGLang mounts every route unconditionally at import time. Ctrl/Cmd + wheel to zoom · drag to pan · double-click to fit
The dotted edge is the load-bearing one. In vLLM the generation routes and the pooling routes are registered by separate conditionals on the same task tuple:
if "generate" in supported_tasks:
from vllm.entrypoints.generate.api_router import (
register_generate_api_routers,
)
register_generate_api_routers(app)
# ...
if any(task in POOLING_TASKS for task in supported_tasks):
from vllm.entrypoints.pooling.factories import register_pooling_api_routers
register_pooling_api_routers(app, supported_tasks, model_config)Because pooling and generation cannot share a batch (§7.5), one engine process serves one of those two families. Ask a generation server for /v1/embeddings and the route is not merely unimplemented — it is not mounted, so FastAPI answers 404 before any handler runs. SGLang mounts both paths regardless and fails deeper, with a message that tells you what to do:
# Validate embedding requests
if isinstance(obj, EmbeddingReqInput) and self.is_generation:
raise ValueError(
"This model does not appear to be an embedding model by default. "
"Please add `--is-embedding` when launching the server or try another model."
)Same architectural constraint, two error surfaces: a 404 from the router versus a 400 from the tokenizer manager. If you are writing a gateway that probes capabilities, you need both branches.
What compatible actually means
Every field in an OpenAI request lands in exactly one of four tiers. Naming them is most of the work, because clients cannot tell them apart from the outside — all four return 200.
| Tier | What happens | Verified example |
|---|---|---|
| honoured | Field maps 1:1 onto a sampler parameter | top_p, frequency_penalty, stop — all appear verbatim in both engines' to_sampling_params |
| clamped | Accepted, then silently rewritten | max_tokens in vLLM (min against remaining context); temperature in (0, 0.01) raised to 0.01 |
| ignored | Parsed, validated, never read | user in both engines; suffix on vLLM's /v1/completions |
| extension | No OpenAI field exists; arrives via extra_body | top_k, min_p, repetition_penalty, ignore_eos, priority, cache_salt |
The ignored tier is documented — but only in a code comment nobody reads:
thinking_token_budget: ThinkingTokenBudget = None
include_reasoning: bool = True
parallel_tool_calls: bool | None = True
# NOTE this will be ignored by vLLM
user: str | None = None """Completion API similar to OpenAI's API.
See https://platform.openai.com/docs/api-reference/completions/create
for the API specification. This API mimics the OpenAI Completion API.
NOTE: Currently we do not support the following feature:
- suffix (the language models we currently support do not support
suffix)
"""There is a fifth, nastier case: fields implemented somewhere other than where you'd expect. parallel_tool_calls: false reads like a constraint on generation. In vLLM it is a post-hoc list truncation applied after the model has already emitted every call:
def maybe_filter_parallel_tool_calls(
choice: _ChatCompletionResponseChoiceT, request: ChatCompletionRequest
) -> _ChatCompletionResponseChoiceT:
"""Filter to first tool call only when parallel_tool_calls is explicitly False."""
if request.parallel_tool_calls is not False:
return choice
if isinstance(choice, ChatCompletionResponseChoice) and choice.message.tool_calls:
choice.message.tool_calls = choice.message.tool_calls[:1]
elif (
isinstance(choice, ChatCompletionResponseStreamChoice)
and choice.delta.tool_calls
):
choice.delta.tool_calls = [
tool_call for tool_call in choice.delta.tool_calls if tool_call.index == 0
]
return choiceYou are billed for the tokens of the calls that get thrown away, and your latency includes generating them. The field is honoured; the cost model behind it is not what a client would assume.
Where the fields land
Both engines funnel the request object through one method that produces engine-native sampling parameters. In vLLM it returns a typed SamplingParams; in SGLang, a plain dict. The mapping is where the names change:
sampling_params = {
"temperature": get_param("temperature"),
"max_new_tokens": self.max_completion_tokens or self.max_tokens,
"min_new_tokens": self.min_tokens,
"stop": stop,
"stop_token_ids": self.stop_token_ids,
"stop_regex": self.stop_regex,
"top_p": get_param("top_p"),
"top_k": get_param("top_k"),
"min_p": get_param("min_p"),
"presence_penalty": self.presence_penalty,
"frequency_penalty": self.frequency_penalty,
"repetition_penalty": get_param("repetition_penalty"),
"regex": self.regex,
"ebnf": self.ebnf,
"n": self.n,
"no_stop_trim": self.no_stop_trim,
"ignore_eos": self.ignore_eos,
"skip_special_tokens": self.skip_special_tokens,
"logit_bias": self.logit_bias,
"custom_params": self.custom_params,
"sampling_seed": self.seed,
"spaces_between_special_tokens": spaces_between_special_tokens,
}Note what is not in that dict: user. Note the extensions that have no OpenAI field at all: stop_regex, ebnf, no_stop_trim, min_new_tokens. And note "max_new_tokens": self.max_completion_tokens or self.max_tokens — the deprecated field is the fallback, so sending both silently prefers max_completion_tokens. vLLM resolves the same precedence explicitly and remembers which name won, so its error messages can name the field you actually used (vllm/entrypoints/openai/chat_completion/protocol.py:L607-L625).
One field, two memory footprints
The same accepted field can be implemented with wildly different costs. Take logit_bias on Llama-3-8B, vocabulary 128,256. SGLang materialises a dense bias matrix for the entire batch as soon as any one request carries the field:
logit_bias = None
if any(r.sampling_params.logit_bias is not None for r in reqs):
logit_bias = torch.zeros(len(reqs), vocab_size, device=device)
for i, r in enumerate(reqs):
if r.sampling_params.logit_bias is not None:
for key, value in r.sampling_params.logit_bias.items():
logit_bias[i, int(key)] = valueDerived (arithmetic, not measured): torch.zeros defaults to fp32, so at batch 256 that allocation is $256 \times 128{,}256 \times 4\ \text{B} = 131.3\ \text{MB}$ of HBM, rebuilt whenever the batch composition changes — triggered by a single client sending one biased token. vLLM instead keeps three parallel flat lists and advanced-indexes them into the logits, so its cost is proportional to the number of biased entries, not to $B \times V$:
self.bias_tensor = self._device_tensor(biases, torch.float32)
self.logits_slice = (
self._device_tensor(reqs, torch.int32),
self._device_tensor(tok_ids, torch.int32),
)
def _device_tensor(self, data: list, dtype: torch.dtype) -> torch.Tensor:
return async_tensor_h2d(data, device=self.device, dtype=dtype)
def apply(self, logits: torch.Tensor) -> torch.Tensor:
if self.biases:
logits[self.logits_slice] += self.bias_tensor
return logitsBut vLLM pays for that elsewhere. Turn on speculative decoding and the field stops working, with a startup log line as the only notice:
if vllm_config.speculative_config:
if custom_logitsprocs:
raise ValueError(STR_SPEC_DEC_REJECTS_LOGITSPROCS)
logger.warning(
"min_p and logit_bias parameters won't work with speculative decoding."
)
return LogitsProcessors(
[MinTokensLogitsProcessor(vllm_config, device, is_pin_memory)]
)Which tier is logit_bias in, then? Honoured or ignored — depending on a server flag the client cannot see. That is the sharpest statement of the chapter's thesis: tier membership is a property of the deployment, not of the API.
Chat templates: the prompt contract
An OpenAI request carries structured messages. The model consumes a flat token sequence. The function between them is a Jinja template that neither engine wrote, and how each engine finds that template differs materially.
vLLM's resolution is a four-level priority chain, and it is explicitly ordered:
# 1st priority: The given chat template
if chat_template is not None:
# Resolve template names (e.g. "tool_use") to actual Jinja content
# so that downstream kwargs detection can parse template variables.
return tokenizer.get_chat_template(chat_template, tools=tools)
# 2nd priority: AutoProcessor chat template, unless tool calling is enabled
if tools is None:
chat_template = _try_get_processor_chat_template(
tokenizer,
revision=model_config.revision,
code_revision=model_config.code_revision,
trust_remote_code=model_config.trust_remote_code,
)
if chat_template is not None:
return chat_template
# 3rd priority: AutoTokenizer chat template
try:
return tokenizer.get_chat_template(chat_template, tools=tools)Read the second branch carefully: adding a tools block to your request can change which template file is selected, because the AutoProcessor template is skipped entirely when tools are present. Two requests to the same server, same model, can render through two different templates.
SGLang has a different first move. Before it ever looks at the checkpoint's own template, it regex-matches the model path string against a registry of built-in conversation templates:
else:
# Guess chat template from model path
self.guess_chat_template_from_model_path(model_path)
# If no pre-defined template was found, fallback to HuggingFace template
if self._chat_template_name is None:
# Try HuggingFace template first
hf_template = self._resolve_hf_chat_template(tokenizer_manager)
if hf_template:
# override the chat template
if tokenizer_manager.tokenizer:
tokenizer_manager.tokenizer.chat_template = hf_template@register_conv_template_matching_function
def match_vicuna(model_path: str):
if re.search(r"vicuna|llava-v1\.5|llava-next-video-7b", model_path, re.IGNORECASE):
return "vicuna_v1.1"Name your local checkout /models/vicuna-style-llama3-sft and SGLang will render every request with the Vicuna conversation template, ignoring the Jinja template inside the checkpoint. vLLM, given the identical directory, uses the checkpoint's template. Neither is wrong; the API surface just does not tell the client which happened.
vLLM filters chat_template_kwargs down to the variables the template actually declares, and drops the rest without error (vllm/renderers/hf.py:L644-L672). Sending {"enable_thinking": false} to a model whose template names the variable thinking is a silent no-op — you get reasoning tokens you paid to suppress, and a 200 either way.
Templates also insert text you did not send. The Llama-3.1 tool template shipped in vLLM's examples fabricates a system message whenever tools are present and you supplied none:
{%- if messages[0]['role'] == 'system' %}
{%- if messages[0]['content'] is string %}
{%- set system_message = messages[0]['content']|trim %}
{%- else %}
{%- set system_message = messages[0]['content'][0]['text']|trim %}
{%- endif %}
{%- set messages = messages[1:] %}
{%- else %}
{%- if tools is not none %}
{%- set system_message = "You are a helpful assistant with tool calling capabilities. Only reply with a tool call if the function exists in the library provided by the user. If it doesn't exist, just reply directly in natural language. When you receive a tool call response, use the output to format an answer to the original user question." %}
{%- else %}
{%- set system_message = "" %}
{%- endif %}
{%- endif %}Two further consequences fall straight out of that snippet. First, only messages[0] is examined — a system message placed later in the array is not extracted and renders as an ordinary turn. Second, the whole system block is conditioned on tools, so the prefix-cache key of an otherwise identical conversation changes the moment a client attaches a tool list.
And the last line of the template is the one every debugging session eventually reaches:
{%- if add_generation_prompt %}
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
{%- endif %}vLLM exposes add_generation_prompt as a request field defaulting to true, plus continue_final_message for prefill-the-assistant workflows, and rejects the combination outright: "Cannot set both `continue_final_message` and `add_generation_prompt` to True." (vllm/entrypoints/openai/chat_completion/protocol.py:L985-L995). Set add_generation_prompt: false by accident and the model continues the user's turn — grammatical, plausible, and completely wrong.
Figure 2 — one chat request's full transformation, with the real Llama-3.1 template variables named at each hop. Token counts are illustrative; the vocabulary size 128,256 is exact for Llama-3-8B. Ctrl/Cmd + wheel to zoom · drag to pan · double-click to fit
Tool calls and the streaming contract
The tool-calling request contract is symmetric across both engines: a tools array of JSON Schemas, a tool_choice of "none" | "auto" | "required" or a named function. vLLM validates it hard — empty tools: [] is a 400 ("`tools` must not be an empty array."), a named choice that matches no tool is a 400, and specifying tool_choice with no tools is a 400 (vllm/entrypoints/openai/chat_completion/protocol.py:L899-L983). SGLang runs equivalent checks in _validate_request (python/sglang/srt/entrypoints/openai/serving_chat.py:L829-L868).
The response side has no symmetry at all, because models do not emit tool calls in a common format. They emit whatever their fine-tune taught them, and something has to reverse it. Hence a per-model parser registry in both projects. Compare two vLLM parsers:
class Hermes2ProToolParser(ToolParser):
structural_tag_model = "hermes"
tool_call_start_token: str = "<tool_call>"
tool_call_end_token: str = "</tool_call>"
tool_call_regex = re.compile(
r"<tool_call>(.*?)</tool_call>|<tool_call>(.*)", re.DOTALL
) bot_token: str = "<|python_tag|>"
structural_tag_model = "llama"
# Simple regex to find opening braces - we'll use JSON decoder for parsing
# This handles arbitrary nesting depth correctly
tool_call_start_regex: re.Pattern = re.compile(r"\{")
json_decoder: json.JSONDecoder = json.JSONDecoder()Hermes brackets calls in XML-ish tags. Llama-3 emits a special token then bare JSON — so its parser has to scan for { and incrementally raw_decode. Neither strategy works on the other's output. At these SHAs vllm/tool_parsers/ holds 45 files matching *_tool_parser.py, and SGLang's ToolCallParserEnum maps 34 parser names onto 30 detector modules in python/sglang/srt/function_call/:
def __init__(self, tools: List[Tool], tool_call_parser: str, tokenizer=None):
detector_class = self.ToolCallParserEnum.get(tool_call_parser)
if detector_class:
kwargs = {}
if tokenizer is not None:
sig = inspect.signature(detector_class)
if "tokenizer" in sig.parameters:
kwargs["tokenizer"] = tokenizer
detector = detector_class(**kwargs)
else:
raise ValueError(f"Unsupported tool_call_parser: {tool_call_parser}")Both registries are keyed by a server startup flag (--tool-call-parser), not by the request. Pick the wrong one and the model's tool calls arrive as ordinary content — a 200, well-formed, with tool_calls: null and your function call sitting in the text field.
The SSE contract
Streaming is where the contract is most precisely specified and most quietly divergent. Both engines emit text/event-stream, one JSON object per data: line, terminated by a literal sentinel:
except GenerationError as e:
yield f"data: {self._convert_generation_error_to_streaming_response(e)}\n\n"
except Exception as e:
logger.exception("Error in chat completion stream generator.")
data = self.create_streaming_error_response(e)
yield f"data: {data}\n\n"
# Send the final done message after all response.n are finished
yield "data: [DONE]\n\n" except ValueError as e:
if not stream_started:
raise
error = self.create_streaming_error_response(str(e))
yield f"data: {error}\n\n"
yield "data: [DONE]\n\n"The if not stream_started: raise is the whole error contract for streaming in one line. Before the first frame, an exception can still become a real HTTP status. After it, the 200 header is already on the wire and the only channel left is a data: frame carrying an error object — followed, dutifully, by [DONE]. A client that only checks response.status_code will treat a mid-stream engine failure as success.
Usage accounting diverges too. vLLM requires include_usage before it will honour continuous_usage_stats; SGLang treats them independently:
def should_include_usage(
stream_options: StreamOptions | None, enable_force_include_usage: bool
) -> tuple[bool, bool]:
if enable_force_include_usage:
return True, True
if stream_options:
include_usage = bool(stream_options.include_usage)
include_continuous_usage = include_usage and bool(
stream_options.continuous_usage_stats
)
else:
include_usage, include_continuous_usage = False, False
return include_usage, include_continuous_usagedef should_include_usage(
stream_options: StreamOptions | None, stream_response_default_include_usage: bool
) -> tuple[bool, bool]:
# When stream_options are specified in the request
if stream_options:
include_usage = (
stream_options.include_usage or stream_response_default_include_usage
)
continuous_usage_stats = bool(stream_options.continuous_usage_stats)
else:
include_usage, continuous_usage_stats = (
stream_response_default_include_usage,
False,
)
return include_usage, continuous_usage_statsvLLM also refuses the combination stream_options without stream=true at validation time — "Stream options can only be defined when `stream=True`." SGLang accepts and ignores it.
Figure 3 — the exact frame sequence a client receives from vLLM for one streaming chat request with stream_options.include_usage set. The usage frame is optional; the terminal frame is not. Ctrl/Cmd + wheel to zoom · drag to pan · double-click to fit
That terminal frame is not decoration. §1.2 found that the two projects' own benchmark harnesses disagree about whether it counts toward end-to-end latency — SGLang's stops the clock after it, vLLM's before. Identical wire protocol, two readings, a measurable delta in every published comparison. The mechanism by which frames are produced belongs to §9.3; what the client sees is this figure.
The error contract
vLLM maps exception types onto HTTP status codes in one place, which makes the contract auditable:
if isinstance(exc, VLLMValidationError):
err_type = "BadRequestError"
status_code = HTTPStatus.BAD_REQUEST
param = exc.parameter
elif isinstance(exc, VLLMUnprocessableEntityError):
err_type = "UnprocessableEntityError"
status_code = HTTPStatus.UNPROCESSABLE_ENTITY
param = exc.parameter
elif isinstance(exc, VLLMNotFoundError):
err_type = "NotFoundError"
status_code = HTTPStatus.NOT_FOUND
param = None
elif isinstance(exc, VLLMClientError):
# Any other client-caused error defaults to 400.
err_type = "BadRequestError"
status_code = HTTPStatus.BAD_REQUEST
param = None
elif isinstance(exc, GenerationError):
err_type = "InternalServerError"
status_code = exc.status_code
param = NoneSGLang's mapping is coarser — three buckets in one try: HTTPException passes its own code through, ValueError becomes 400, everything else becomes 500 (python/sglang/srt/entrypoints/openai/serving_base.py:L110-L132). In practice that is enough, because nearly every user-facing validation failure in SGLang raises ValueError.
The sharpest divergence is the unknown-model case. vLLM checks it on every generation request:
if self._is_model_supported(request.model):
return None
# if _check_model has been called earlier, this will be unreachable
raise VLLMNotFoundError(f"The model `{request.model}` does not exist.")SGLang returns a correctly-shaped 404 from GET /v1/models/{model} (python/sglang/srt/entrypoints/http_server.py:L1875-L1892) but performs no model check inside /v1/chat/completions. A colon in the field is interpreted as a LoRA selector instead:
def _parse_model_parameter(self, model: str) -> Tuple[str, Optional[str]]:
"""Parse 'base-model:adapter-name' syntax to extract LoRA adapter.
Returns (base_model, adapter_name) or (model, None) if no colon present.
"""
if ":" not in model:
return model, None
# Split on first colon only to handle model paths with multiple colons
parts = model.split(":", 1)
base_model = parts[0].strip()
adapter_name = parts[1].strip() or NoneSo a client that typos the model name gets a 404 from vLLM and a perfectly good answer from the wrong-named model on SGLang. A client that sends a model name containing a colon — a registry-style tag like myorg/llama3:v2 — gets a LoRA lookup on SGLang.
Malformed JSON inside a response_format schema is its own category. vLLM rejects the obvious shape errors at validation time ("When response_format type is 'json_schema', the 'json_schema' field must be provided."), but a schema that is structurally valid yet uses an unsupported keyword fails later, inside the grammar compiler. Which compiler that is depends on the request:
class StructuredOutputsConfig:
"""Dataclass which contains structured outputs config for the engine."""
backend: StructuredOutputsBackend = "auto"
"""Which engine will be used for structured outputs (e.g. JSON schema,
regex, etc) by default. With "auto", we will make opinionated choices
based on request contents and what the backend libraries currently support,
so the behavior is subject to change in each release."""
disable_any_whitespace: bool = False
"""If `True`, json output will always be compact without any whitespace.
If `False`, the model may generate whitespace between JSON fields,
which is still valid JSON. This is only supported for xgrammar
and guidance backends."""§6.5 established what that means operationally: "auto" is a per-request fallback chain, not a fixed default. Two requests to the same server can compile on two different grammar engines. Read the second docstring above and the API consequence is immediate — disable_any_whitespace is honoured only on xgrammar and guidance, so under "auto" the whitespace in your JSON output is a function of which backend accepted your particular schema. Byte-for-byte reproducibility across requests is not something this surface promises.
Pitfalls when porting from OpenAI
Collected, with the source of each already cited above.
| Field | vLLM | SGLang |
|---|---|---|
max_tokens overflow | silently clamped to remaining context, 200 + finish_reason: length | 400 with a message naming both counts |
temperature: 0 | greedy path; forces top_p=1, top_k=0, min_p=0 | rewritten to temperature=1.0, top_k=1 |
temperature in (0, 0.01) | raised to 0.01 with a warning | left as-is (eps is 1e-6) |
n > 1 with temperature: 0 | 400 — n must be 1 when using greedy sampling | 200, n identical choices |
logprobs | chat: bool + top_logprobs; completions: int. top_logprobs without logprobs: true is a 400 | same split; top_logprobs is Optional[int] defaulting to None |
max_completion_tokens | wins over max_tokens; the winning name is remembered for error text | max_completion_tokens or max_tokens |
unknown model | 404 The model `X` does not exist. | accepted; a colon is parsed as a LoRA selector |
user | accepted, ignored (comment says so) | accepted, absent from the sampling dict |
suffix (completions) | accepted, ignored (docstring says so) | not present on CompletionRequest |
stream_options without stream | 400 | accepted, ignored |
seed | to SamplingParams.seed; -1 normalised to None | to sampling_seed |
On seed: neither engine promises that the same seed yields the same tokens across two runs, because the batch a request lands in changes the floating-point reduction order. That is §10.4's subject; the API-surface consequence is that seed is a reproducibility hint, not a guarantee, and no field in the OpenAI schema expresses the difference.
On stop: both trim the stop string from the output by default. vLLM's include_stop_str_in_output and SGLang's no_stop_trim are inverted spellings of the same switch, and both are extensions. SGLang additionally offers stop_regex, which OpenAI has no field for at all.
vLLM's /health calls client.check_health() and returns 503 on EngineDeadError (vllm/entrypoints/serve/instrumentator/health.py:L22-L33). SGLang's /health_generate pushes a real one-token request through the engine (python/sglang/srt/entrypoints/http_server.py:L654-L680) — a genuinely deeper probe, but one that queues behind live traffic. Its plain /health short-circuits to 200 unless SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION is set. Point your Kubernetes liveness probe at the wrong one and you either never detect a wedged engine or you restart a merely-busy one.
One more asymmetry worth knowing: vLLM always mounts /metrics; SGLang mounts Prometheus middleware only under --enable-metrics (python/sglang/srt/entrypoints/http_server.py:L285-L287). A scrape config that assumes the endpoint exists will silently collect nothing. §9.5 owns what is in those series.
Hands-on
These commands require running servers. Schema-only validation can be tested without executing a model, but successful completions and generation-health probes require a loaded model and supported runtime. Record model, tokenizer, device, and launch arguments.
# 1. Enumerate what is actually mounted. FastAPI publishes its own route table.
curl -s localhost:8000/openapi.json | python3 -c \
'import json,sys; [print(m.upper(), p) for p,v in sorted(json.load(sys.stdin)["paths"].items()) for m in v]'
# 2. The ignored tier: identical outputs, one with a bogus user field.
curl -s localhost:8000/v1/completions -H 'content-type: application/json' \
-d '{"model":"M","prompt":"hi","max_tokens":5,"seed":1,"user":"alice","suffix":"XYZ"}' | jq .choices[0].text
# 3. The clamped tier: ask for more than fits and read finish_reason.
curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' \
-d '{"model":"M","messages":[{"role":"user","content":"count to 500"}],"max_tokens":999999}' \
| jq '.choices[0].finish_reason, .usage'
# 4. The n>1 greedy split. 400 on vLLM, 200 with duplicate choices on SGLang.
curl -s -o /dev/null -w '%{http_code}\n' localhost:8000/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"model":"M","messages":[{"role":"user","content":"hi"}],"n":4,"temperature":0}'
# 5. See the rendered prompt without generating (vLLM only).
curl -s localhost:8000/v1/chat/completions/render -H 'content-type: application/json' \
-d '{"model":"M","messages":[{"role":"user","content":"hi"}],"return_prompt_text":true}'
# 6. Watch the SSE frames raw, terminal frame included.
curl -sN localhost:8000/v1/chat/completions -H 'content-type: application/json' \
-d '{"model":"M","messages":[{"role":"user","content":"hi"}],"stream":true,
"stream_options":{"include_usage":true}}'Step 5 is the single most useful debugging tool on this surface. vLLM's /v1/chat/completions/render and /v1/completions/render (registered when generate or render is a supported task, vllm/entrypoints/scale_out/render/api_router.py:L26-L53) return the templated prompt string without running the model — so you can see the synthetic system message, confirm add_generation_prompt fired, and diff two checkpoints' templates in one command. SGLang has no equivalent render route; the nearest tool is POST /v1/tokenize on the rendered string.
Exercises
- Read and answer. Open
vllm/entrypoints/launchers/api_server/routers.py. A model whosesupported_tasksis("embed",)is served. List every route that is mounted and every one that is not. Then explain why/tokenizeis in the first list. - Predict, then verify. A client sends
chat_template_kwargs: {"enable_thinking": false}to a model whose template declares the variable asthinking. What status code comes back and what does the model do? Verify againstvllm/renderers/hf.py:L644-L672. - Derive. A server runs Llama-3-8B (vocab 128,256) with a steady-state batch of 192 on SGLang. One tenant starts sending
logit_biaswith a single biased token on 1% of requests. How much extra HBM is allocated per batch rebuild, and how does that change if the tenant biases 5,000 tokens instead of one? Then answer the same for vLLM. - Predict, then verify. You POST to
/v1/chat/completionswithstream: true. The engine dies after 40 tokens. What HTTP status did the client already receive, what arrives on the wire next, and what does a naive SDK report? Cite the two lines that decide it. - Read and answer. Find one field in
python/sglang/srt/entrypoints/openai/protocol.py'sChatCompletionRequestthat is in the OpenAI schema but absent from the dict returned byto_sampling_params, other thanuser. Explain where, if anywhere, it is consumed.
Answers
1. Mounted: /health, /load, /version, /metrics, /v1/models, the LoRA routes, /start_profile, /stop_profile, /tokenize, /detokenize, /ping, /invocations, plus the pooling family (/v1/embeddings, /v2/embed, /pooling, /score, /rerank, ...). Not mounted: everything under register_generate_api_routers — /v1/chat/completions, /v1/completions, /v1/responses, /v1/messages, /cohere/v2/chat — and the elastic-EP routes. /tokenize is in register_vllm_serve_api_routers, which runs unconditionally: tokenization needs only the tokenizer, not a generation-capable engine.
2. HTTP 200, and the model reasons normally. resolve_chat_template_kwargs computes accept_vars from the template's undeclared Jinja variables plus the tokenizer's accepted kwargs, then returns {k: v for k, v in chat_template_kwargs.items() if k in accept_vars}. An unrecognised key is dropped with no log line. The only kwargs that raise are the two in unexpected_vars: chat_template and tokenize.
3. SGLang's allocation does not depend on how many tokens are biased — torch.zeros(len(reqs), vocab_size) is 192 × 128,256 × 4 B = 98.5 MB either way, and it is charged to the whole batch, not the one tenant. Biasing 5,000 tokens changes nothing. vLLM allocates three flat tensors sized by total biased entries: with 1% of 192 requests biasing one token that is ~2 entries (~24 bytes); at 5,000 tokens each it is ~9,600 entries (~115 KB). Both derived arithmetic, not measured.
4. The client already has HTTP 200 with Content-Type: text/event-stream — the header went out with the first frame. Next it receives one data: frame containing an error object, then data: [DONE]. A naive SDK that checks only the status code reports success with a truncated completion. The deciding lines are vllm/entrypoints/openai/chat_completion/serving.py:L907-L912 and, on the SGLang side, python/sglang/srt/entrypoints/openai/serving_chat.py:L1764-L1770 — where if not stream_started: raise is the only path back to a real status code.
5. logprobs and top_logprobs. Neither reaches to_sampling_params; they are handled on the response-construction path, because logprob depth is a decision about what the engine returns rather than how it samples. parallel_tool_calls is a second valid answer — it is consumed only by the tool-call assembly at serving_chat.py:L1112 and L1133.
Key takeaways
- A field's tier is a property of the deployment, not the API.
logit_biasis honoured on vLLM until you enable speculative decoding, at which point it becomes a startup warning and a silent no-op. No response field tells the client which world it is in. - The most consequential divergence is context overflow. vLLM clamps
max_tokensand returns 200 with a truncated answer; SGLang rejects with 400 and an explicit token accounting. Any retry logic written against one is wrong against the other. - The prompt contract is owned by the checkpoint, not the engine. A Jinja file from the model repo decides what the model sees — including system messages you never sent. vLLM resolves it through a four-level chain that a
toolsblock can redirect; SGLang regex-matches the model path first and may never read the checkpoint's template at all. - Per-model tool parsers are unavoidable and are selected at server startup. Hermes emits XML tags, Llama-3 emits a special token then bare JSON; no single parser reverses both. Choose wrong and tool calls arrive as ordinary
contentwith a 200. - Once an SSE stream starts, the status code is frozen. Engine failures after the first frame arrive as a
data:error object followed by[DONE]. Streaming clients must parse frames, not statuses. - Endpoint availability encodes an architectural constraint. vLLM registers generation and pooling routes under separate task conditionals because the two cannot share a batch — so a 404 on
/v1/embeddingsis not a missing feature, it is the batching model showing through the API.
Further reading
- OpenAI Chat Completions reference — the spec both engines track. The field ordering in
vllm/entrypoints/openai/chat_completion/protocol.pymirrors this page deliberately, with a comment saying so. - vLLM: OpenAI-Compatible Server — the extra-parameters sections are generated from the
--8<-- [start:chat-completion-extra-params]markers you can see in the protocol source. - SGLang: OpenAI APIs — companion pages cover the native
/generateendpoint, which exposes fields the OpenAI schema has no room for. - HuggingFace chat templating — the upstream definition of
add_generation_prompt,{% generation %}, and thetoolsvariable that both engines feed. - vLLM PR #5649 — OpenAI tools API with named function calling, and PR #8343 —
--enable-auto-tool-choiceand the tool parser plugin interface. The discussions are the best account of why per-model parsers exist. - SGLang PR #2544 — function-call parser framework, which introduced the
BaseFormatDetectorabstraction behindpython/sglang/srt/function_call/. - §9.2 for what happens to the rendered string; §9.3 for the plumbing behind Figure 3; §6.1 for what the mapped sampling parameters cost on device.