Streaming and the end-to-end request lifecycle

Here is a log line SGLang emits in production:

        all other fields (meta_info, etc.) are taken from the last chunk.
        """
        if len(out_list) >= 20:
            logger.warning(
                "Streaming backlog: rid=%s, coalescing %d queued chunks into one. "
                "This may inflate P99 ITL for affected requests.",
                rid,
                len(out_list),
            )

Twenty tokens were generated, detokenised, and handed to the HTTP layer for one request, and the HTTP layer had not picked up a single one of them. At Llama-3-8B's derived decode floor of 4.48 ms per step (§0.4), twenty steps is 90 ms of engine time in which the frontend never ran. The client then receives one SSE frame carrying twenty tokens, and the §1.2 identity between TPOT and mean ITL quietly breaks: nineteen of those inter-token gaps are zero and one is 90 ms.

Nothing in that sentence is about attention kernels, block tables, or sampling. It is about plumbing: which process holds which queue, who wakes whom, and what happens when one side runs faster than the other. That plumbing is also where the resource leaks live — a request whose client vanished and whose KV blocks nobody freed costs you capacity for as long as the generation would have taken.

So: one streaming chat request, end to end, in both engines, every hop named. Then the three questions only the end-to-end view can answer — where the latency goes, what a disconnect does, and what happens when requests arrive faster than they retire.

The engine step loop and the HTTP connection run on independent clocks and neither can block the other. That single sentence explains most of the architecture.

The step loop is a while True that schedules a batch, launches a forward pass, and processes the result. It has no idea HTTP exists, and its period is set by the GPU: 4.48 ms for Llama-3-8B on an H100 at batch 1, and still roughly 4.48 ms at batch 32 because that is inside the weight-bandwidth-bound regime (§4.4 puts the crossover near batch 118). The HTTP side is an asyncio event loop whose period is set by clients, TCP windows, and JSON serialisation. Between them sits a queue, and everything interesting is a property of that queue: how deep it gets, who drains it, what happens if nobody does.

Both engines put the boundary in the same place — between the part that talks HTTP and the part that owns the GPU — because a Python process busy serialising JSON for 200 SSE streams cannot also issue CUDA launches on a 4.48 ms deadline. §5.5 established that topology; this chapter walks a request through it.

One forward pass produces one token for every request in the batch. The frontend must turn each of those into its own SSE frame on its own socket. That gives a hard budget nobody writes down but everybody hits.

Let $T_{\text{step}}$ be the engine step period, $B$ the running batch size, and $c$ the frontend CPU cost of turning one sampled token id into bytes on a socket — detokenise, build the delta object, serialise to JSON, write. The frontend keeps up only if

Instantiate it. Llama-3-8B (L=32, d=4096, h=32, h_kv=8, d_h=128) in bf16 on an H100 SXM: 15.01 GB of weights over 3.35 TB/s gives $T_{\text{step}} = 4.48$ ms, and that number barely moves up to batch ~118. So:

These are arithmetic from $T_{\text{step}}$ and $B$, not measurements. The shape is the point: the budget shrinks linearly in batch size while $c$ does not shrink at all, so every serving system eventually becomes frontend-bound, and it happens at a batch size, not at a token rate. Both engines respond the same way — not by shrinking $c$, but by making the frontend do fewer, bigger units of work when it falls behind. That is what the backlog warning is: coalescing, not dropping.

Two consequences worth holding onto before the traces:

• The engine never waits for the frontend. In both engines the step loop pushes into a queue and immediately schedules the next batch. A slow client costs that client's ITL distribution; it does not cost throughput until memory runs out.
• Coalescing is lossless for tokens and lossy for timing. When the queue drains late, the tokens all arrive — concatenated. What is destroyed is the per-token timestamp, which is exactly the thing §1.2's ITL is computed from.

Hop 1 — the route. Routers are registered per feature. register_api_routers (vllm/entrypoints/launchers/api_server/routers.py:L12-L20) calls into register_generate_api_routers (vllm/entrypoints/generate/api_router.py:L21-L26), which attaches the chat router. The endpoint itself:

@router.post(
    "/v1/chat/completions",
    dependencies=[Depends(validate_json_request)],
    responses={
        HTTPStatus.OK.value: {"content": {"text/event-stream": {}}},
        HTTPStatus.BAD_REQUEST.value: {"model": ErrorResponse},
        HTTPStatus.NOT_FOUND.value: {"model": ErrorResponse},
        HTTPStatus.INTERNAL_SERVER_ERROR.value: {"model": ErrorResponse},
        HTTPStatus.NOT_IMPLEMENTED.value: {"model": ErrorResponse},
    },
)
@with_cancellation
@load_aware_call
async def create_chat_completion(request: ChatCompletionRequest, raw_request: Request):

FastAPI has already validated and coerced the JSON body into a ChatCompletionRequest pydantic model — that is the entire validation stage, and §9.1 owns what it does and does not accept. Note @with_cancellation; we come back to it in §8.

Hop 2 — chat template and tokenise. OpenAIServingChat.render_chat_request delegates to OnlineRenderer.render_chat (vllm/renderers/online_renderer.py:L117-L218), which validates tool-choice, then calls preprocess_chat → Renderer.render_chat_async:

        if tok_params is None:
            tok_params = self.default_chat_tok_params

        rendered = [
            self.render_messages_async(conversation, chat_params)
            for conversation in conversations
        ]

        out_conversations = list[list["ConversationMessage"]]()
        dict_prompts = list[DictPrompt]()
        for conv, prompt in await asyncio.gather(*rendered):
            out_conversations.append(conv)
            dict_prompts.append(prompt)

        tok_prompts = await self.tokenize_prompts_async(dict_prompts, tok_params)

Both the Jinja render and the tokenizer call are blocking work, so they go into a thread pool built in the renderer constructor (vllm/renderers/base.py:L85-L99), sized by renderer_num_workers, default 1 (vllm/config/model.py:L367). First asynchrony boundary, first pitfall: with one worker, tokenising a 100k-token prompt serialises against every other request's tokenisation, though not against the event loop itself.

Hop 3 — sampling params and the generator. _create_chat_completion converts the request to a SamplingParams (§6.1 owns what those mean on the GPU). One field matters here:

            output_kind=(
                RequestOutputKind.DELTA if self.stream else RequestOutputKind.FINAL_ONLY
            ),

DELTA is what makes the per-request collector coalesce rather than overwrite. Then:

                generator = self.engine_client.generate(
                    engine_input,
                    sampling_params,
                    sub_request_id,
                    lora_request=lora_request,
                    trace_headers=trace_headers,
                    priority=self._get_priority(request, raw_request),
                    data_parallel_rank=data_parallel_rank,
                    session_id=session_id,
# ...
        (result_generator,) = generators

        if request.stream:
            return self.chat_completion_stream_generator(
                request,
                result_generator,
                request_id,

Hop 4 — registration, in two places at once. AsyncLLM.add_request creates a RequestOutputCollector and then does exactly two things:

    async def _add_request(
        self,
        request: EngineCoreRequest,
        prompt: str | None,
        parent_req: ParentRequest | None,
        index: int,
        queue: RequestOutputCollector,
    ):
        # Add the request to OutputProcessor (this process).
        self.output_processor.add_request(request, prompt, parent_req, index, queue)

        # Add the EngineCoreRequest to EngineCore (separate process).
        await self.engine_core.add_request_async(request)

Read the two comments. The request now exists in two places, and the rest of the chapter — especially the abort path — is about keeping those two copies in agreement.

Hop 5 — the process boundary. The EngineCoreRequest is a msgspec.Struct with array_like=True and gc=False (vllm/v1/engine/__init__.py:L107-L122), so it encodes to a msgpack array rather than a map. The send:

    def _send_input(
        self,
        request_type: EngineCoreRequestType,
        request: Any,
        engine: EngineIdentity | None = None,
    ) -> Awaitable[Any]:
        if engine is None:
            engine = self.core_engine

        message = (request_type.value, *self.encoder.encode(request))
        return self._send_input_message(message, engine)
# ...
        return self.input_socket.send_multipart((engine,) + message, copy=False)
# ...
    async def add_request_async(self, request: EngineCoreRequest) -> None:
        request.client_index = self.client_index
        await self._send_input(EngineCoreRequestType.ADD, request)

The frontend socket is a zmq.ROUTER and the reply socket a zmq.PULL (vllm/v1/engine/core_client.py:L554-L563); the engine side connects a zmq.DEALER and pushes back on a zmq.PUSH (vllm/v1/engine/core.py:L1690-L1697, L1795-L1800). Every one of these sockets is created with high-water marks explicitly disabled:

    if socket_type in (zmq.PULL, zmq.DEALER, zmq.ROUTER):
        socket.setsockopt(zmq.RCVHWM, 0)
        socket.setsockopt(zmq.RCVBUF, buf_size)

    if socket_type in (zmq.PUSH, zmq.DEALER, zmq.ROUTER):
        socket.setsockopt(zmq.SNDHWM, 0)
        socket.setsockopt(zmq.SNDBUF, buf_size)

Remember that when you get to §9: there is deliberately no backpressure at the transport.

Hop 6 — into the engine process. A dedicated IO thread polls the DEALER, decodes, and pushes onto a plain queue.Queue (vllm/v1/engine/core.py:L1674-L1775, ending at self.input_queue.put_nowait((request_type, request))). The busy loop is the consumer:

    def run_busy_loop(self):
        """Core busy loop of the EngineCore."""
        while self._handle_shutdown():
            # 1) Poll the input queue until there is work to do.
            self._process_input_queue()
            # Publish request counts before and after GPU step to ensure freshness.
            self._maybe_publish_request_counts()
            # 2) Step the engine core and return the outputs.
            self._process_engine_step()
            self._maybe_publish_request_counts()

        raise SystemExit

_process_input_queue (L1417-L1446) drains the queue through _handle_client_request (L1520-L1553), which for an ADD calls EngineCore.add_request → self.scheduler.add_request(request) (vllm/v1/engine/core.py:L478). The request is now in the scheduler's waiting queue. Note the loop shape: input is drained only between steps, never during one.

Hop 7 — scheduled, allocated, executed. One step:

        # Check for any requests remaining in the scheduler - unfinished,
        # or finished and not yet removed from the batch.
        if not self.scheduler.has_requests():
            return {}, False
        scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())
        future = self.model_executor.execute_model(scheduler_output, non_block=True)
        grammar_output = self.scheduler.get_grammar_bitmask(scheduler_output)
        with (
            self.capture_iteration_details(scheduler_output) as iteration_details,
            self.log_error_detail(scheduler_output),
        ):
            model_output = future.result()
            if model_output is None:
                model_output = self.model_executor.sample_tokens(grammar_output)

        # Before processing the model output, process any aborts that happened
        # during the model execution.
        self._process_aborts_queue()
        engine_core_outputs = self.scheduler.update_from_output(
            scheduler_output, model_output
        )

Inside schedule(), admission and the token budget belong to §1.4; the kv_cache_manager.allocate_slots call at vllm/v1/core/sched/scheduler.py:L639 belongs to §2.2; the attention metadata built inside execute_model belongs to §3.4; sampling belongs to §6.1. What this chapter owns is the shape: a single synchronous call chain from one schedule() to one batch of sampled token ids, with no per-request branching anywhere in it.

Hop 8 — the demultiplex begins. update_from_output loops over the batch and appends one struct per request, keyed by the client that submitted it:

            if should_emit_output:
                # Add EngineCoreOutput for this Request.
                outputs[request.client_index].append(
                    EngineCoreOutput(
                        request_id=req_id,
                        new_token_ids=new_token_ids,
                        finish_reason=finish_reason,
                        new_logprobs=new_logprobs,
                        new_sampling_mask=new_sampling_mask,
                        new_prompt_logprobs_tensors=prompt_logprobs_tensors,
                        pooling_output=pooler_output,

Finish detection happened just before this, in check_stop (vllm/v1/core/sched/utils.py:L94-L130): EOS token, a member of stop_token_ids, max_model_len, or max_tokens. Notice what is not there — stop strings. Those need text, which the engine process does not have. §9.2 owns that split; its consequence appears in hop 11.

Hop 9 — back across the boundary. _process_engine_step puts each (client_index, EngineCoreOutputs) pair on an output queue drained by another IO thread (vllm/v1/engine/core.py:L1812-L1826) that msgpack-encodes and PUSHes. On the frontend, an asyncio task pulls and decodes:

        async def process_outputs_socket():
            try:
                while True:
                    frames = await output_socket.recv_multipart(copy=False)
                    resources.validate_alive(frames)
                    outputs: EngineCoreOutputs = decoder.decode(frames)

Hop 10 — the fan-out. A single background task in the API server process owns the whole demux:

        async def output_handler():
            try:
                while True:
                    # 1) Pull EngineCoreOutputs from the EngineCore.
                    outputs = await engine_core.get_output_async()
                    num_outputs = len(outputs.outputs)
# ...
                    # Split outputs into chunks of at most
                    # VLLM_V1_OUTPUT_PROC_CHUNK_SIZE, so that we don't block the
                    # event loop for too long.
                    engine_core_outputs = outputs.outputs
                    for start in range(0, num_outputs, chunk_size):
                        end = start + chunk_size
                        outputs_slice = engine_core_outputs[start:end]
                        # 2) Process EngineCoreOutputs.
                        processed_outputs = output_processor.process_outputs(
                            outputs_slice, outputs.timestamp, iteration_stats
                        )
                        # NOTE: RequestOutputs are pushed to their queues.
                        assert not processed_outputs.request_outputs
# ...
                        # Allow other asyncio tasks to run between chunks
                        if end < num_outputs:
                            await asyncio.sleep(0)

                        # 3) Abort any reqs that finished due to stop strings.
                        if processed_outputs.reqs_to_abort:
                            await engine_core.abort_requests_async(
                                processed_outputs.reqs_to_abort
                            )

VLLM_V1_OUTPUT_PROC_CHUNK_SIZE defaults to 128 (vllm/envs.py:L169). At batch 256 the handler yields once mid-batch so SSE writers can run; at batch 32 it does not yield at all, and the whole demux is one uninterruptible block of the event loop.

Hop 11 — detokenise and hand off. Inside process_outputs, per request:

                # 2) Detokenize the token ids into text and perform stop checks.
                stop_string = req_state.detokenizer.update(
                    new_token_ids, finish_reason == FinishReason.STOP
                )
                if stop_string:
                    finish_reason = FinishReason.STOP
                    stop_reason = stop_string
# ...
            # 4) Create and handle RequestOutput objects.
            if request_output := req_state.make_request_output(
                new_token_ids,
                pooling_output,
                finish_reason,
                stop_reason,
                kv_transfer_params,
                ec_transfer_params,
            ):
                if req_state.streaming_input:
                    request_output.finished = False

                if req_state.queue is not None:
                    # AsyncLLM: put into queue for handling by generate().
                    req_state.queue.put(request_output)

This is the exact point where one batched engine output becomes N independent per-request streams. vllm/v1/engine/detokenizer.py is §9.2's territory; here, note only that it runs in the frontend event loop and that a stop string discovered here has to be sent back across the process boundary as an abort, which is what step 3 of the output handler does.

Hop 12 — the per-request queue. The "queue" is a one-slot mailbox that merges:

class RequestOutputCollector:
    """
    Collects streamed RequestOutputs per individual request,
    for hand-off to the consuming asyncio generate task.

    When streaming deltas, RequestOutputs are merged if the
    producer gets ahead of the consumer.
    """

    def __init__(self, output_kind: RequestOutputKind, request_id: str):
        self.aggregate = output_kind == RequestOutputKind.DELTA
        self.request_id = request_id
        self.output: RequestOutput | PoolingRequestOutput | Exception | None = None
        self.ready = asyncio.Event()
# ...
    def put(self, output: RequestOutput | PoolingRequestOutput | Exception) -> None:
        """Non-blocking put operation."""
        if self.output is None or isinstance(output, Exception):
            self.output = output
            self.ready.set()
        elif isinstance(self.output, RequestOutput) and isinstance(
            output, RequestOutput
        ):
            # This ensures that request outputs with different request indexes
            # (if n > 1) do not override each other.
            self.output.add(output, aggregate=self.aggregate)

put merges into one queue slot, but that is not a byte-memory bound. Merged token arrays and text can still grow with unconsumed output. Bound output length, buffered bytes, consumer lifetime, and concurrency separately. This is the structural twin of SGLang's backlog warning, minus the warning.

Hop 13 — out of the engine API. AsyncLLM.generate is the consumer:

            # The output_handler task pushes items into the queue.
            # This task pulls from the queue and yields to caller.
            finished = False
            while not finished:
                # Note: drain queue without await if possible (avoids
                # task switching under load which helps performance).
                out = q.get_nowait() or await q.get()

                # Note: both OutputProcessor and EngineCore handle their
                # own request cleanup based on finished.
                assert isinstance(out, RequestOutput)
                finished = out.finished
                if out is not STREAM_FINISHED:
                    yield out

Hop 14 — the SSE frame. chat_completion_stream_generator consumes that async generator, builds the delta, and yields raw SSE:

                    delta_text = output.text

                    if (
                        not delta_text
                        and not output.token_ids
                        and not previous_num_tokens[i]
                    ):
                        # Chunked prefill case, don't return empty chunks
                        continue
# ...
                    # handle streaming just a content delta (no parsers)
                    else:
                        delta_message = DeltaMessage(content=delta_text)
# ...
                    data = chunk.model_dump_json(exclude_unset=True)
                    yield f"data: {data}\n\n"

The generator is handed to Starlette as StreamingResponse(content=generator, media_type="text/event-stream") (vllm/entrypoints/openai/chat_completion/api_router.py:L74), and uvicorn awaits each send(). That await is where TCP backpressure enters the system. On finish, chat_completion_stream_generator emits the terminal chunk with finish_reason, optionally a usage-only chunk, then yield "data: [DONE]\n\n" (serving.py:L912).

Hop 1 — the route. One flat module holds every endpoint:

@app.post("/v1/chat/completions", dependencies=[Depends(validate_json_request)])
async def openai_v1_chat_completions(
    request: ChatCompletionRequest, raw_request: Request
):
    """OpenAI-compatible chat completion endpoint."""
    return await raw_request.app.state.openai_serving_chat.handle_request(
        request, raw_request
    )

Hop 2 — validate, template, tokenise. OpenAIServingBase.handle_request (python/sglang/srt/entrypoints/openai/serving_base.py:L73-L109) stamps received_time = monotonic_time() as its very first statement — that is where SGLang's TTFT clock starts — then validates and converts. The conversion applies the chat template and tokenises, in the HTTP worker process:

            try:
                rendered_prompt = self.tokenizer_manager.tokenizer.apply_chat_template(
                    openai_compatible_messages,
                    tokenize=False,
                    add_generation_prompt=True,
                    tools=tools,
                    return_dict=False,
                    **extra_template_kwargs,
                )
                prompt_ids = self.tokenizer_manager.tokenizer.encode(
                    rendered_prompt, **encode_kwargs
                )

Note the difference from vLLM: this is a plain synchronous call on the event loop, not a thread-pool offload. _convert_to_internal_request then packs prompt_ids into GenerateReqInput(input_ids=...), so the later _tokenize_one_request short-circuits — the /v1/chat/completions path never re-tokenises.

Hop 3 — kick the generator before the 200. A detail worth stealing:

        generator = self._generate_chat_stream(adapted_request, request, raw_request)

        # Kick-start the generator to trigger validation before HTTP 200 is sent.
        # If validation fails (e.g., context length exceeded), we can still return
        # a proper HTTP 400 error response instead of streaming it as SSE payload.
        try:
            first_chunk = await generator.__anext__()
        except ValueError as e:
            return self.create_error_response(str(e))

        async def prepend_first_chunk():
            yield first_chunk
            async for chunk in generator:
                yield chunk

        return StreamingResponse(
            prepend_first_chunk(),
            media_type="text/event-stream",
            background=self.tokenizer_manager.create_abort_task(adapted_request),
        )

SGLang holds the response open until the first chunk exists, so a context-length rejection becomes a real HTTP 400 instead of an SSE error frame after a 200. vLLM instead checks engine_client.errored up front and streams errors as SSE (vllm/entrypoints/openai/chat_completion/serving.py:L236-L240, L905-L910). Also note the background= task — that is the disconnect handler, and we return to it in §8.

Hop 4 — register and dispatch. TokenizerManager.generate_request creates the per-request state, tokenises (a no-op here), sends, and then awaits:

                # Tokenize the request and send it to the scheduler
                if obj.is_single:
                    tokenized_obj = await self._tokenize_one_request(obj)
                    state = self.rid_to_state[obj.rid]
                    if obj.return_prompt_token_ids:
                        state.prompt_token_ids = list(tokenized_obj.input_ids)
                    self._send_one_request(tokenized_obj)
                    async for response in self._wait_one_response(obj, request):
                        yield response

Hop 5 — the process boundary. _send_one_request stamps a dispatch timestamp and calls _dispatch_to_scheduler → sock_send:

def sock_send(socket: zmq.Socket, obj: Any, flags: int = 0) -> None:
    if _USE_PICKLE_IPC:
        socket.send_pyobj(obj, flags=flags, protocol=pickle.HIGHEST_PROTOCOL)
        return

    socket.send(msgpack_encode(obj), flags=flags)

Read the branch order: pickle is the default. _USE_PICKLE_IPC = envs.SGLANG_USE_PICKLE_IPC.get() (python/sglang/srt/managers/io_struct.py:L2381) and that env var defaults to True (python/sglang/srt/environ.py:L337), so a stock SGLang server sends send_pyobj at pickle.HIGHEST_PROTOCOL and never reaches msgpack_encode. The structs are still msgspec.Structs — they are just pickled rather than msgpack-encoded, and PickleWrapper, the shim that lets opaque payloads ride inside a msgpack frame, is dead on this path (io_struct.py:L106-L116). Set SGLANG_USE_PICKLE_IPC=0 to get the msgpack wire format vLLM always uses. The socket is a zmq.PUSH to the scheduler's PULL (python/sglang/srt/managers/tokenizer_manager.py:L553-L555), and SGLang also disables high-water marks (python/sglang/srt/utils/network.py:L248-L253).

Hop 6 — the scheduler loop. Unlike vLLM's blocking-queue-plus-IO-thread, SGLang polls non-blocking once per iteration:

                while True:
                    try:
                        if self.recv_limit_reached(len(recv_reqs)):
                            break
                        recv_req = sock_recv(self.recv_from_tokenizer, zmq.NOBLOCK)
                    except zmq.ZMQError:
                        break
                    recv_reqs.append(recv_req)

Only attention-TP rank 0 pulls; _broadcast_reqs_across_ranks then replicates to the other ranks, which is the Model B topology §5.5 described. Then:

    def event_loop_overlap(self):
        """A scheduler loop that overlaps the CPU processing and GPU computation."""
        self.result_queue: Deque[
            Tuple[ScheduleBatch, Union[GenerationBatchResult, EmbeddingBatchResult]]
        ] = deque()

        def pop_and_process():
            # Process the results of the last batch
            tmp_batch, tmp_result = self.result_queue.popleft()
            self.process_batch_result(tmp_batch, tmp_result)

        while True:
            if self.gracefully_exit:
                break

            # Receive requests
            recv_reqs = self.request_receiver.recv_requests()
            self.process_input_requests(recv_reqs)

process_input_requests dispatches by type (scheduler.py:L1923) into handle_generate_request, which constructs the Req (scheduler.py:L2435-L2470) and enqueues it:

    def _add_request_to_queue(self, req: Req, is_retracted: bool = False):
        if not self._set_or_validate_priority(req):
            return
        if self.disaggregation_mode == DisaggregationMode.NULL:
            if self._abort_on_queued_limit(req):
                return
            self._prefetch_kvcache(req)
            self.waiting_queue.append(req)
            req.time_stats.set_wait_queue_entry_time()

Hop 7 — batch, run, finish-check. get_next_batch_to_run (§1.4) picks the batch; run_batch executes it; process_batch_result_decode walks the batch and updates each request:

            req.output_ids.extend(next_token_id)
            new_accept_len = len(next_token_id)

            self._maybe_update_reasoning_tokens(req, next_token_id)
            req.time_stats.set_last_decode_finish_time()
            req.update_finish_state(new_accept_len)

            self._handle_finish_state_updated_req(req, batch, result, i, logits_output)

Req.update_finish_state (python/sglang/srt/managers/schedule_batch.py:L1632-L1672) is SGLang's check_stop — and it checks stop strings too, because SGLang keeps a running decoded tail on the Req. That is a real architectural difference from vLLM: SGLang can finish a request in the scheduler process on a stop string; vLLM must round-trip an abort.

Hop 8 — the stream interval gate. Not every step emits:

    def accept(self, *, req: Req) -> None:
        if req.finished():
            assert not req.finished_output
            req.finished_output = True
            if req.finished_len is None:
                req.finished_len = len(req.output_ids)
            should_output = True
        else:
            if req.stream:
                stream_interval = (
                    req.sampling_params.stream_interval or self.default_stream_interval
                )

                # origin stream_interval logic
                should_output = (
                    len(req.output_ids) % stream_interval == 1
                    if stream_interval > 1
                    else len(req.output_ids) % stream_interval == 0
                )

                if should_output:
                    # check_match_stop_str_prefix if  tail_str's suffix match stop_str prefix
                    should_output &= not req.check_match_stop_str_prefix()

Default stream_interval is 1 (python/sglang/srt/server_args.py:L1480-L1484), so every token is emitted; raise it and you trade ITL smoothness for frontend CPU. vLLM has the identical knob, applied instead in the frontend at vllm/v1/engine/output_processor.py:L299-L313. The suffix check is the stop-string-prefix hold that §9.2 owns.

Hop 9 — to the detokeniser process. The accumulator produces one BatchTokenIDOutput covering the whole batch and pushes it (output_streamer.py:L173-L182). The DetokenizerManager is a bare loop:

    def event_loop(self):
        """The event loop that handles requests"""
        while True:
            with self.soft_watchdog.disable():
                recv_obj = sock_recv(self.recv_from_scheduler)
            output = self._request_dispatcher(recv_obj)
            if output is not None:
                sock_send(self.send_to_tokenizer, output)
            self.soft_watchdog.feed()

handle_batch_token_id_out turns BatchTokenIDOutput into BatchStrOutput and pushes it on to the HTTP worker. This is the structural difference from vLLM: detokenisation is a full extra process hop, which costs one more serialise/deserialise round per step and buys immunity from both the GPU loop and the socket writers.

Hop 10 — the fan-out. Back in the HTTP worker, handle_loop (tokenizer_manager.py:L2151-L2164) receives and _handle_batch_output demultiplexes by rid:

            if out_dict is not None:
                state.out_list.append(out_dict)
                pending_notify[rid] = state

                if len(pending_notify) >= batch_notify_size:
                    for s in pending_notify.values():
                        s.event.set()
                    pending_notify = {}
                    await asyncio.sleep(0)

SGLang's per-request mailbox is a list plus an asyncio.Event, and it batches the wakeups: batch_notify_size defaults to 16 (python/sglang/srt/server_args.py:L1485-L1489), so at batch 32 the loop wakes 16 request tasks, yields once, then wakes the rest. Compare vLLM: one Event.set() per request, no yield inside the fan-out at all below 128.

Hop 11 — the per-request await. _wait_one_response drains the list atomically:

            # Drain all pending outputs atomically.
            out_list = state.out_list
            state.out_list = []
            finished = state.finished
            state.event.clear()

            # With incremental streaming, each chunk is a delta — coalesce
            # multiple queued chunks to avoid dropping token ids.
            incremental_stream = is_stream and self.incremental_streaming_output
            if incremental_stream and len(out_list) > 1:
                out = self._coalesce_streaming_chunks(
                    out_list,
                    obj.rid,
                    state.customized_info_accumulated.keys(),
                )

There is the backlog warning from §1, in context. Where vLLM merges on put and can therefore never accumulate, SGLang appends on put and merges on get — which is why it can see the depth and log it. Same outcome for the client, better observability.

Hop 12 — the SSE frame. _generate_chat_stream iterates tokenizer_manager.generate_request (serving_chat.py:L1544-L1546), reads content["meta_info"]["finish_reason"], builds the delta and yields f"data: {chunk.model_dump_json()}\n\n", closing with yield "data: [DONE]\n\n" (serving_chat.py:L1770).

Now walk the trace a second time with §1.2's stopwatch. TTFT is $t_1 - t_a$ and each $\mathrm{ITL}_k$ is $t_k - t_{k-1}$, where $t_a$ is arrival and $t_k$ is when the client sees token $k$.

Where the clock starts. Neither engine's $t_a$ is the TCP SYN. SGLang stamps received_time = monotonic_time() as the first line of handle_request (serving_base.py:L79), after FastAPI has parsed and pydantic-validated the body; vLLM stamps arrival_time = time.time() at the top of render_chat_async (vllm/renderers/base.py:L1075), later still. Both under-report TTFT relative to what the client measures, by the cost of parse plus validation — small for a 500-token prompt, not small for a request carrying 200 tool definitions.

Where the clock stops. $t_1$ as the engine reports it is when the frame is yielded, not when it is on the wire. SGLang stamps set_response_sent_to_client_time() just before the yield (tokenizer_manager.py:L1742-L1747). The socket write is asynchronous after that point. Under a slow client the engine's TTFT and the client's TTFT diverge and only the client's is real.

The column that matters is the fourth. Only the first three rows are on the engine's critical path. Everything after the ZMQ hop is on the client's ITL but not on the engine's step period, which is why a frontend that falls behind degrades tail ITL long before it degrades tokens per second — and why you can have a P99 ITL alarm firing while the throughput dashboard looks perfect.

Two structural sources of ITL jitter fall out of the traces. The batch-notify quantum: SGLang wakes request tasks 16 at a time, so request 17 waits on an asyncio.sleep(0) that request 1 did not. The output-processing chunk: vLLM processes up to 128 requests without yielding, so at batch 128 the last SSE writer starts only after all 128 detokenisations have run. Both are deliberate throughput trades, and both show up as ITL variance rather than in the mean.

A browser tab closes mid-generation. The GPU does not know. Follow the signal.

vLLM. Two mechanisms, and they hand over to each other. Before the response object exists, @with_cancellation races the handler against a task that awaits http.disconnect:

async def listen_for_disconnect(request: Request) -> None:
    """Returns if a disconnect message is received"""
    while True:
        message = await request.receive()
        if message["type"] == "http.disconnect":
# ...
            break
# ...
        handler_task = asyncio.create_task(handler_func(*args, **kwargs))
        cancellation_task = asyncio.create_task(listen_for_disconnect(request))

        done, pending = await asyncio.wait(
            [handler_task, cancellation_task], return_when=asyncio.FIRST_COMPLETED
        )
        for task in pending:
            task.cancel()

Its own docstring notes that once a StreamingResponse is returned, the response object takes over disconnect listening. Starlette then cancels the generator, which raises GeneratorExit inside AsyncLLM.generate, and that is caught:

        # If the request is disconnected by the client, generate()
        # is cancelled or the generator is garbage collected. So,
        # we abort the request if we end up here.
        except (asyncio.CancelledError, GeneratorExit):
            if q is not None:
                await self.abort(q.request_id, internal=True)
            if self.log_requests:
                logger.info("Request %s aborted.", request_id)
            raise
# ...
    async def abort(
        self, request_id: str | Iterable[str], internal: bool = False
    ) -> None:
        """Abort RequestId in OutputProcessor and EngineCore."""

        request_ids = (
            (request_id,) if isinstance(request_id, str) else as_list(request_id)
        )
        all_request_ids = self.output_processor.abort_requests(request_ids, internal)
        await self.engine_core.abort_requests_async(all_request_ids)

Both copies of the request die: output_processor.abort_requests pops the frontend state (output_processor.py:L508-L513), and abort_requests_async sends an ABORT frame. On the engine side that frame goes onto two queues:

                        if request_type == EngineCoreRequestType.ABORT:
                            # Aborts are added to *both* queues, allows us to eagerly
                            # process aborts while also ensuring ordering in the input
                            # queue to avoid leaking requests. This is ok because
                            # aborting in the scheduler is idempotent.
                            self.aborts_queue.put_nowait(request)

                    # Push to input queue for core busy loop.
                    self.input_queue.put_nowait((request_type, request))

The extra aborts_queue is drained by _process_aborts_queue() mid-step, between future.result() and update_from_output (core.py:L607). That is why the comment exists: without it, an abort arriving during a long forward pass would wait a full step. The scheduler then does the freeing:

        # Remove all requests from queues at once for better efficiency
        if running_requests_to_remove:
            self.running = remove_all(self.running, running_requests_to_remove)
        if waiting_requests_to_remove:
            self.waiting.remove_requests(waiting_requests_to_remove)
            self.skipped_waiting.remove_requests(waiting_requests_to_remove)

        # Second pass: set status and free requests
        for request in valid_requests:
            delay_free_blocks = False
# ...
            request.status = finished_status
            self._free_request(request, delay_free_blocks=delay_free_blocks)
# ...
    def _free_blocks(self, request: Request):
        assert request.is_finished()
        self._free_request_blocks(request)
        del self.requests[request.request_id]

SGLang. Three paths, and the code names them. The disconnect trigger is a Starlette background task — it runs after the response finishes, for any reason including a client vanishing:

    def create_abort_task(self, obj: GenerateReqInput):
        # Abort the request if the client is disconnected.
        async def abort_request():
            await asyncio.sleep(2)
            if obj.is_single:
                self.abort_request(obj.rid)
            else:
                for rid in obj.rid:
                    self.abort_request(rid)

The two-second sleep is a grace window; abort_request is a no-op if the rid already left rid_to_state (tokenizer_manager.py:L1948-L1953). A second trigger covers requests still queued: _wait_one_response wakes every SGLANG_REQUEST_STATE_WAIT_TIMEOUT seconds (default 4, python/sglang/srt/environ.py:L1217) and calls await request.is_disconnected(). In the scheduler, abort_request then picks one of three methods by where the request is:

            # Abort method 1: directly pop from the queue
            # This only works for requests that have not started anything.
            # We still need to send something back to TokenizerManager to clean up the state.
            req = self.waiting_queue.pop(i)
# ...
        # Delete the requests in the grammar queue
        # Abort method 2: call `set_finish_with_abort`
        # The request will still run one prefill forward pass.
# ...
                # Abort method 3: set `to_finish`
                # The request will still run one decode forward pass.
                # Then we reuse all existing code to clean up the KV cache allocation.
                logger.debug(f"Abort running request. {req.rid=}")
                req.to_finish = FINISH_ABORT()

Method 3 is the interesting one. Rather than reach into the running batch and free pages while a forward pass may still be writing them, SGLang marks the request and lets it exit through the normal path one decode step later, where release_kv_cache → tree_cache.cache_finished_req (python/sglang/srt/mem_cache/common.py:L214-L219, python/sglang/srt/mem_cache/radix_cache.py:L459-L461) does the freeing. vLLM reaches the same conclusion from the other side: _free_request_blocks defers the return to the block pool when an in-flight step may still write those blocks (vllm/v1/core/sched/scheduler.py:L2463-L2475). Both engines pay one extra forward pass to avoid a use-after-free on KV pages.

Requests arrive faster than the engine retires them. Where do they pile up, what bounds the pile, and what does the client see?

Nothing bounds the transport. Both engines set SNDHWM and RCVHWM to 0 on every ZMQ socket in the request path (vllm/utils/network_utils.py:L336-L342; python/sglang/srt/utils/network.py:L248-L253). That is a deliberate choice — a blocked send inside the engine's IO thread would stall the step loop — and it means the transport will happily buffer until the machine runs out of RAM.

vLLM: unbounded, by design. There is no admission limit at the HTTP layer, no queue-depth cap in EngineCore, and no maximum on the scheduler's waiting queue. A thousand concurrent requests become a thousand RequestOutputCollectors and a waiting queue the scheduler drains at whatever rate max_num_seqs and the KV budget allow. What the client sees is TTFT growing without bound. The only defences are external: a router with a concurrency cap (§9.4) and client-side timeouts.

SGLang: an explicit cap that returns 503. Set --max-queued-requests (default None, i.e. unbounded, python/sglang/srt/server_args.py:L792-L796) and admission becomes a real decision:

    def _abort_on_queued_limit(self, recv_req: Req) -> bool:
        """Abort an incoming or existing request if the waiting queue is full. Returns True if the incoming request is aborted."""
        if (
            self.max_queued_requests is None
            or len(self.waiting_queue) + 1 <= self.max_queued_requests
        ):
            return False

        # Reject the incoming request by default.
        req_to_abort = recv_req
        message = "The request queue is full."
# ...
        self.ipc_channels.send_to_tokenizer.send_output(
            AbortReq(
                finished_reason={
                    "type": "abort",
                    "status_code": HTTPStatus.SERVICE_UNAVAILABLE,
                    "message": message,
                },
                rid=req_to_abort.rid,
            ),
            req_to_abort,
        )

With priority scheduling on, it will instead evict the least-preferred queued request and admit the new one, with the message "The request is aborted by a higher priority request.". There is a companion timeout: _abort_on_waiting_timeout (scheduler.py:L2878-L2905) sweeps the waiting queue against SGLANG_REQ_WAITING_TIMEOUT and 503s anything older, with "Request waiting timeout reached." — but that env var defaults to -1 and the function returns on the first line when it is <= 0, so on a stock server this sweep never runs. All three reach the client the same way — _handle_abort_finish_reason maps a SERVICE_UNAVAILABLE abort to a real fastapi.HTTPException for non-streaming requests and to an SSE error chunk for streaming ones (tokenizer_manager.py:L1644-L1683; serving_chat.py:L1590-L1605).

The slow-client case is separate, and both engines answer it identically in shape. A slow reader does not slow the engine; it fills its own mailbox, which coalesces — vLLM on put, SGLang on get. Memory grows by the undelivered text of that one request only. The alternative, letting a slow reader apply backpressure through to the scheduler, would let one client throttle a shared GPU.

You are reading the wrong file. At a556f3f, vllm/entrypoints/openai/api_server.py still exists but warns "`vllm.entrypoints.openai.api_server` is deprecated and will likely be unsupported in a future version.", and serving_chat.py is simply gone. Anything referencing those paths predates the restructure.

The first chunk is empty and you filtered it out. chat_completion_stream_generator has an explicit guard: if delta_text is empty, there are no token ids, and no tokens have been sent yet, continue — commented "Chunked prefill case, don't return empty chunks" (serving.py:L647-L653). With chunked prefill (§1.5) the engine emits per-step bookkeeping before any token exists. Client code that measures TTFT from the first SSE frame rather than the first frame with content will report a TTFT that is too low.

The stop string arrived one round-trip late. vLLM detects stop strings in the frontend detokeniser, so the engine has already scheduled the next step by the time reqs_to_abort makes it back (async_llm.py:L725-L729); SGLang checks inside the scheduler and stops immediately. Neither leaks the extra text, but vLLM burns the extra decode step. See §9.2.

The abort that arrived after the finish. Explicitly handled, not a bug: "Abort request for rid=%s not found in rid_to_state; likely already finished/removed." (tokenizer_manager.py:L3128-L3132), whose comment notes it is "Common under mass client disconnects, amplified by prefix / abort_all fan-out." In bulk it means a load balancer is timing clients out.

The detokeniser forgot your request. SGLang's DetokenizerManager keeps per-request incremental-decode state in a LimitedCapacityDict(capacity=DETOKENIZER_MAX_STATES) (python/sglang/srt/managers/detokenizer_manager.py:L143), default $2^{16}$. Exceed it with enough concurrent streams and the oldest entry is evicted while its request is still running:

            try:
                s = self.decode_status[rid]
            except KeyError:
                raise RuntimeError(
                    f"Decode status not found for request {rid}. "
                    "It may be due to the request being evicted from the decode status due to memory pressure. "
                    "Please increase the maximum number of requests by setting "
                    "the SGLANG_DETOKENIZER_MAX_STATES environment variable to a bigger value than the default value. "
                    f"The current value is {DETOKENIZER_MAX_STATES}. "
                    "For more details, see: https://github.com/sgl-project/sglang/issues/2812"
                )

Note where that raise lands: inside the detokeniser process's event loop, not in any one request's task. The blast radius is process-wide, which is part of the price of the extra hop.

A slow tokenizer serialises your frontend. vLLM's renderer thread pool defaults to one worker (vllm/config/model.py:L367), so one 200k-token prompt occupies it while every other arriving request queues behind — a TTFT spike on requests unrelated to the big one. Raise --renderer-num-workers. SGLang has the sharper version of the same exposure on the chat path: apply_chat_template and encode run directly on the event loop.

Start a server and watch each hop with the engine's own logs rather than a debugger.

# vLLM: request-level logging plus a single-token stream
vllm serve meta-llama/Meta-Llama-3-8B-Instruct --max-model-len 8192

curl -N -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"meta-llama/Meta-Llama-3-8B-Instruct",
       "messages":[{"role":"user","content":"Count to twenty."}],
       "stream":true,"stream_options":{"include_usage":true},"max_tokens":32}'

Three things to do with that, in order.

1. Find the boundary in ps. ps -ef | grep -E "VLLM::|sglang". vLLM shows the API server and an EngineCore child; SGLang shows the HTTP worker, a scheduler, and a detokenizer. Kill the detokeniser on a running SGLang server: generation continues, no bytes reach the client. That is Figure 3's third boundary, demonstrated.
2. Move the stream interval. Restart SGLang with --stream-interval 8 and re-run the curl: same total tokens, eight-fold fewer frames, an ITL distribution of seven zeros and one large gap. Compare vLLM's "stream_interval": 8 in the request body, applied at output_processor.py:L299-L313 — different process, identical user-visible effect.
3. Trip the queue limit. Start SGLang with --max-queued-requests 4, fire 64 concurrent long requests, and read the 503 bodies. You should see The request queue is full. verbatim. Then run the same load against vLLM and observe that you get no 503s at all — only rising TTFT.

Lab 12, 12-read-a-request-through-the-code, does this properly: it attaches to a live server and instruments each hop so the "unmeasured" rows in §7's table become real numbers on your hardware.

• vLLM V1 design notes — Architecture overview and the V1 blog post, for the reasoning behind moving EngineCore into its own process. Cross-check every path against the tree; the entrypoints layout changed under the docs.
• vLLM PR #9826, "[V1] AsyncLLM Implementation" — the commit that created vllm/v1/engine/core.py, and the thread where the out-of-process EngineCore and its ZMQ boundary were argued out.
• vLLM PR #11973 and PR #15156 — the first created output_processor.py; the second, "[V1][Perf] Simpler request output queues", is where RequestOutputCollector and its merge-on-put policy landed. Read #15156 for the clearest statement of the slow-client trade.
• vLLM PR #32240, "[Refactor] [6/N] to simplify the vLLM openai chat_completion serving architecture" — the commit that moved serving_chat.py into vllm/entrypoints/openai/chat_completion/. Read it before trusting any older write-up of the entrypoints layout.
• SGLang PR #7565, "throttle requests at scheduler based on --max_queued_requests", and PR #8746, which extended it to evict a queued request in favour of a higher-priority arrival — the two commits behind _abort_on_queued_limit.
• SGLang PR #21646, "Clean up TokenizerManager and req_time_stats" — where _coalesce_streaming_chunks and the backlog warning that opens this chapter were introduced.
• Orca (OSDI '22), "Orca: A Distributed Serving System for Transformer-Based Generative Models" — iteration-level scheduling, which is what makes the per-step fan-out in this chapter necessary in the first place.
• Neighbours: §9.1 for what the endpoint accepts, §9.2 for incremental detokenisation and stop strings, §11.2 and §12.2 for each engine's internals at repo-tour depth.