Inference is not training

An engineer who has trained models moves onto a serving team. Their first instinct is the one training rewarded for years: the GPU is idle, raise the batch size. So they start vLLM with --max-num-seqs 512 instead of a previously explicit cap of 128, point a load generator at it, and watch nvidia-smi report 99% GPU utilisation. Throughput barely moves. Median time-to-first-token triples. The p99 goes somewhere embarrassing. And the periodic engine log line grows a field it did not carry before — Preemptions: %d, which vLLM appends only once that counter is non-zero (vllm/v1/metrics/loggers.py:L283-L285) — because deep inside the scheduler this branch is firing, over and over, for requests that were already halfway through generating:

                    else:
                        preempted_req = self.running.pop()

                    self._preempt_request(
                        preempted_req,
                        scheduled_timestamp,
                        drop_stale_output=self.requires_kv_delivery,
                    )
                    preempted_reqs.append(preempted_req)
                    if preempted_req == request:
                        # No more request to preempt. Cannot schedule this request.

Three training intuitions failed at once. 99% utilisation meant nothing — that counter reports whether a kernel is resident, not whether the machine is doing useful arithmetic. Raising the batch did not raise throughput, because the thing that was full was not compute, it was the KV cache. And work already done got thrown away, because at inference the scheduler is allowed to evict a running request to make room, a move that has no analogue in a training step. Nothing here is a bug. It is what happens when you bring a fixed-batch, offline, throughput-only mental model to an online system with a latency contract.

Four inversions separate the two problems, and every later chapter of this book is a consequence of one of them.

The right-hand column is the whole game. Weights are a fixed cost you pay once at load time. The KV cache is a pool the engine carves out of what is left, and every concurrent request rents a slice of it that grows by one token per decode step. Batch size at inference is therefore not a number you set. It is whatever fits in that pool right now.

Take Llama-3-8B: $P = 8.03 \times 10^9$ parameters, $L = 32$ layers, $h_{kv} = 8$ KV heads, $d_h = 128$ head dimension, served in bf16 so $b = 2$ bytes per cached element. The symbols are the ones used throughout the book; see the formula sheet.

Training memory

Mixed-precision Adam costs about 16 bytes per parameter: 2 for the bf16 weight, 2 for the bf16 gradient, and 12 for the fp32 master copy plus the two Adam moments — the accounting from the ZeRO paper (Rajbhandari et al., 2019, §3). That is $8.03\times10^9 \times 16 = 128$ GB derived before a single activation is saved. This is why fine-tuning 8B on one 80 GB card means LoRA, ZeRO offload, or state/optimizer sharding or offload. Gradient checkpointing reduces saved activations, not the 128 GB of parameter/optimizer state, so checkpointing alone cannot make this full-training state fit.

Inference memory

Drop the backward pass and $128$ GB becomes $8.03\times10^9 \times 2 = 16.1$ GB of weights. What replaces the other 112 GB is the KV cache, whose per-token cost is derived:

The leading 2 is for K and V. Note what is absent: the number of query heads $h$. That asymmetry is why grouped-query attention exists, and the full treatment is §2.1.

Now budget an 80 GB H100. vLLM reserves a fraction of the device, defaulting to 0.92:

    gpu_memory_utilization: float = Field(default=0.92, gt=0, le=1)
    """The fraction of GPU memory to be used for the model executor, which can
    range from 0 to 1. For example, a value of 0.5 would imply 50% GPU memory
    utilization. If unspecified, will use the default value of 0.92. This is a
    per-instance limit, and only applies to the current vLLM instance. It does
    not matter if you have another vLLM instance running on the same GPU. For
    example, if you have two vLLM instances running on the same GPU, you can
    set the GPU memory utilization to 0.5 for each instance."""

The engine then subtracts everything that is not KV cache from that budget and keeps the remainder. The subtraction is literally this:

        self.available_kv_cache_memory_bytes = (
            self.requested_memory
            - profile_result.non_kv_cache_memory
            - cudagraph_memory_estimate_applied

An “80 GB” H100 actually presents $79.65$ GiB, so take $0.92 \times 79.65 = 73.28$ GiB requested, minus $14.96$ GiB of weights ($8.03 \times 10^{9}$ parameters at 2 bytes), minus roughly 6 GiB for peak activations, CUDA graph replay buffers (§8.1 — captured kernel sequences the engine replays instead of relaunching, which cost device memory to hold) and non-torch allocations — the three terms vLLM names in its startup message at vllm/v1/worker/gpu_worker.py:L782-L793. That leaves about $52.32$ GiB of KV pool, or

vLLM declares 128 as the fallback cap on concurrent sequences (vllm/config/scheduler.py:L44, DEFAULT_MAX_NUM_SEQS), but that value is rarely what a server runs: EngineArgs.get_batch_defaults (vllm/engine/arg_utils.py:L2580-L2671) resolves max_num_seqs to 1,024 on any device with at least 70 GiB that is not an A100, which includes the H100 we are budgeting. Either way the KV pool binds at 52 at 8k context, far below both numbers — so raising an explicitly configured 128 cap to 512 in §1 cannot remove this KV constraint. The H100 resolved default is already 1,024; 512 is not an increase from that default. The batch-size ceiling at inference is a memory ceiling, and it moves with context length. Doubling total resident context approximately halves the raw token-budget concurrency, holding the pool and other constraints fixed. Prompt plus generated/reserved tokens, not prompt length alone, determines that budget.

Why the compute intuition also fails

Consider one decode step at $B = 52$ — the concurrency the pool above allows. It reads all 16.1 GB of weights from HBM (the same 14.96 GiB, in decimal) and performs roughly $2PB \approx 8.4 \times 10^{11}$ FLOP, an arithmetic intensity of about 52 FLOP/byte derived. An H100 SXM does 989 TFLOP/s dense bf16 against 3.35 TB/s of HBM3 (NVIDIA H100 datasheet), a machine balance of about 295 FLOP/byte. Decode sits roughly 5.7× below that ridge: the GPU spends its time waiting on memory, and the tensor cores idle. Prefilling one 8192-token prompt over the same 16.1 GB of weight traffic gives about 8,200 FLOP/byte — an order of magnitude above the ridge, firmly compute-bound.

Same weights, same kernels, same GPU, two workloads on opposite sides of the roofline. That is the single most consequential fact in LLM serving, and it is why the next chapter is called Prefill and decode are two different computers. The roofline itself is derived properly in §0.4.

Both engines expose the same two entry points — an in-process Python object for offline batch work, and an HTTP server for online serving — but they factor them differently, and the difference tells you what each project optimised for.

vLLM: two engine wrappers over one core

The offline path is LLM.generate(). It adds every prompt, then runs a plain synchronous loop until the engine is empty:

        # Run the engine.
        outputs: list[_O] = []
        total_in_toks = 0
        total_out_toks = 0
        while self.llm_engine.has_unfinished_requests():
            step_outputs = self.llm_engine.step()
            for output in step_outputs:
                assert isinstance(output, output_type)
                if output.finished:
                    outputs.append(output)  # type: ignore[arg-type]
# ...
        # Sort the outputs by request ID.
        # This is necessary because some requests may be finished earlier than
        # its previous requests.
        return sorted(outputs, key=lambda x: int(x.request_id))

That closing comment is the whole chapter in three lines. Even offline, with every prompt known up front, requests finish out of order, because each one stops at its own EOS or stop string. The final sorted() exists only to hide continuous batching from a caller who thinks they passed in a batch.

The online path is AsyncLLM.generate(), one asyncio task per request, pulling from a per-request queue that a single background handler fills:

            # The output_handler task pushes items into the queue.
            # This task pulls from the queue and yields to caller.
            finished = False
            while not finished:
                # Note: drain queue without await if possible (avoids
                # task switching under load which helps performance).
                out = q.get_nowait() or await q.get()

                # Note: both OutputProcessor and EngineCore handle their
                # own request cleanup based on finished.
                assert isinstance(out, RequestOutput)
                finished = out.finished

And, uniquely to the online world, a cancellation path — because the other end of the socket is a browser tab that can be closed:

        # If the request is disconnected by the client, generate()
        # is cancelled or the generator is garbage collected. So,
        # we abort the request if we end up here.
        except (asyncio.CancelledError, GeneratorExit):
            if q is not None:
                await self.abort(q.request_id, internal=True)

The two wrappers differ only in how outputs are pumped. Underneath, both construct an EngineCoreClient — vllm/v1/engine/llm_engine.py:L105-L111 with asyncio_mode=False, vllm/v1/engine/async_llm.py:L149-L156 via make_async_mp_client — and the scheduling loop they drive is identical:

    def step(self) -> tuple[dict[int, EngineCoreOutputs], bool]:
        """Schedule, execute, and make output.

        Returns tuple of outputs and a flag indicating whether the model
        was executed.
        """

        # Check for any requests remaining in the scheduler - unfinished,
        # or finished and not yet removed from the batch.
        if not self.scheduler.has_requests():
            return {}, False
        scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())
        future = self.model_executor.execute_model(scheduler_output, non_block=True)

SGLang: one engine, two front doors

SGLang does not have an offline engine and an online engine. It has one process tree, described in the Engine docstring:

class Engine(EngineScoreMixin, EngineBase):
    """
    The entry point to the inference engine.

    - The engine consists of three components:
        1. TokenizerManager: Tokenizes the requests and sends them to the scheduler.
        2. Scheduler (subprocess): Receives requests from the Tokenizer Manager, schedules batches, forwards them, and sends the output tokens to the Detokenizer Manager.
        3. DetokenizerManager (subprocess): Detokenizes the output tokens and sends the result back to the Tokenizer Manager.

    Note:
    1. The HTTP server, Engine, and TokenizerManager all run in the main process.
    2. Inter-process communication is done through IPC (each process uses a different port) via the ZMQ library.
    """

The offline Engine.generate() builds a GenerateReqInput, hands it to the same tokenizer_manager.generate_request the HTTP handler uses, and drives the async generator from a private event loop:

        generator = self.tokenizer_manager.generate_request(obj, None)

        if stream:

            def generator_wrapper():
                while True:
                    try:
                        chunk = self.loop.run_until_complete(generator.__anext__())
                        yield chunk
                    except StopAsyncIteration:
                        break

            return generator_wrapper()
        else:
            ret = self.loop.run_until_complete(generator.__anext__())
            return ret

The HTTP server calls the identical coroutine at python/sglang/srt/entrypoints/http_server.py:L894-L898, and boots the identical process tree by calling the Engine classmethod directly:

    # Launch subprocesses
    (
        tokenizer_manager,
        template_manager,
        port_args,
        scheduler_init_result,
        subprocess_watchdog,
        _weight_cache_daemon_procs,
    ) = Engine._launch_subprocesses(
        server_args=server_args,
        init_tokenizer_manager_func=init_tokenizer_manager_func,
        run_scheduler_process_func=run_scheduler_process_func,
        run_detokenizer_process_func=run_detokenizer_process_func,
    )

What a request carries, and what an operator controls

A training batch is a tensor. An inference batch is a set of requests, and each one drags along per-request state that constrains how it can be batched with its neighbours:

    seed: int | None = None
    """Random seed to use for the generation."""
    stop: str | list[str] | None = None
    """String(s) that stop the generation when they are generated. The returned
    output will not contain the stop strings."""
    stop_token_ids: list[int] | None = None
    """Token IDs that stop the generation when they are generated. The returned
    output will contain the stop tokens unless the stop tokens are special
    tokens."""
    ignore_eos: bool = False
    """Whether to ignore the EOS token and continue generating
    tokens after the EOS token is generated."""
    max_tokens: int | None = 16
    """Maximum number of tokens to generate per output sequence."""
    min_tokens: int = 0
    """Minimum number of tokens to generate per output sequence before EOS or
    `stop_token_ids` can be generated"""

Read that as a list of ways your neighbours can ruin your latency. A seed forces a per-request RNG stream. stop strings mean a request's true length is unknown until it happens, so the scheduler cannot plan capacity. max_tokens is a bound the client picks, and structured_outputs at vllm/sampling_params.py:L337-L344 attaches a grammar-constrained logits processor to one row of a shared sampling kernel. None of these exist in training, where every row of the batch is treated identically.

Against that, what the operator gets is a handful of global knobs. SGLang's are grouped under a schedule namespace; the first of them, mem_fraction_static, is SGLang's counterpart to gpu_memory_utilization and is owned in full by §2.6:

    mem_fraction_static: A[
        Optional[float],
        "The fraction of the memory used for static allocation (model weights and KV cache memory pool). Use a smaller value if you see out-of-memory errors.",
        NS("schedule"),
    ] = None
    max_running_requests: A[
        Optional[int], "The maximum number of running requests.", NS("schedule")
    ] = None
    max_queued_requests: A[
        Optional[int],
        "The maximum number of queued requests. This option is ignored when using disaggregation-mode.",
        NS("schedule"),
    ] = None

Note the asymmetry. The client controls per-request semantics; the operator controls only aggregate resource policy. That gap — clients choosing shapes, operators choosing budgets — is what the scheduler exists to arbitrate (§1.4).

A timestamped lifecycle. An illustrative request arrives at 0 ms, leaves the queue at 40 ms, completes prefill and samples its first output at 65 ms, delivers it at 68 ms, and is cancelled at 90 ms. Queue delay, compute and delivery contribute separately to TTFT; cancellation must release live references while respecting in-flight work. See the latency definitions and scheduler lifecycle. Increasing concurrency can improve raw throughput yet lower goodput when extra requests cross their latency bounds.

Offline, vLLM. LLM.generate(prompts, sampling_params) (vllm/entrypoints/llm.py:L418-L480) validates that the runner type is "generate", fills in default sampling params, and calls _run_completion (vllm/entrypoints/offline_utils.py:L326-L349). That adds all prompts via _add_completion_requests, then hands off to _run_engine, which spins LLMEngine.step() until has_unfinished_requests() goes false. Each step() (vllm/v1/engine/llm_engine.py:L298-L336) pulls EngineCoreOutputs, runs the output processor, and aborts anything that hit a stop string. Inside the core, EngineCore.step() calls scheduler.schedule() then model_executor.execute_model(). Ten thousand prompts go in; the forward pass never sees ten thousand of anything. It sees whatever the scheduler admitted this iteration.

Online, SGLang. A POST to /generate lands on the FastAPI handler (python/sglang/srt/entrypoints/http_server.py:L894-L898), which awaits tokenizer_manager.generate_request(obj, request) — the same coroutine Engine.generate() drives offline. The TokenizerManager tokenizes in the main process and ZMQs the request to the Scheduler subprocess; the Scheduler batches, forwards, and ships output tokens to the DetokenizerManager, which sends text back. If the client disconnects, the streaming handler catches it explicitly rather than logging a 400. The full hop-by-hop trace is §12.2; vLLM's is §11.2.

    retraction_policy: A[
        str,
        Arg(
            help=(
                "The decode retraction policy to use when the KV cache is full. "
                "'length' preserves the existing behavior and retracts short-output, "
                "long-input requests first. 'priority' retracts lower-priority "
                "requests first, using the same priority direction as priority "
                "scheduling."
            ),
            choices=RETRACTION_POLICY_CHOICES,
        ),
        NS("schedule"),
    ] = "length"

The trigger is at python/sglang/srt/managers/scheduler.py:L3572-L3589: the scheduler checks batch.check_decode_mem() and, if it fails, calls batch.retract_decode(). vLLM's equivalent is the preemption block quoted in §1. Either way, a partially generated request loses its KV blocks and re-prefills later — so under memory pressure your effective throughput drops and your tail latency explodes, from the same cause. If you see rising p99 with flat throughput, look for preemption counters before you look at kernels.

Start a server and read the memory arithmetic out of its own log rather than trusting the derivation above:

vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.92 --max-model-len 8192 2>&1 | grep -i "kv cache"

The line to find is emitted by vllm/v1/worker/gpu_worker.py:L577-L581: "Available KV cache memory: %s GiB". Divide it by 128 KiB to get the token capacity, then by --max-model-len to get worst-case concurrency. Compare with the 429k / 52 figures above. Then rerun with --gpu-memory-utilization 0.80 and watch both numbers fall while the weights term does not move.

The SGLang equivalent, whose knob is the fraction of memory held statically rather than the fraction of the device:

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct \
  --mem-fraction-static 0.90 --max-running-requests 64

No GPU? The structural half of this chapter is readable offline. Run grep -n "has_unfinished_requests" vllm/entrypoints/offline_utils.py in a vLLM checkout and follow step() down to EngineCore.step() yourself; that call chain is the spine of Part 11.

• Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., SOSP 2023) — the paper vLLM grew out of. Read §2 and §3 now for the memory framing; the mechanism is §2.2.
• Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., OSDI 2022) — where iteration-level scheduling was introduced, i.e. why the batch changes every step.
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Rajbhandari et al., 2019) — §3 is the source of the 16 bytes-per-parameter training accounting used above.
• SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al., 2023) — the framing that produced the single-process-tree design.
• docs/design/arch_overview.md in the vLLM tree — the in-repo architecture overview, including the V1 process-count table for a given TP/DP/API-server configuration. Worth reading before Part 11.
• vLLM V1: A Major Upgrade to vLLM's Core Architecture — the rewrite that produced the EngineCore split quoted in §4.