Static, dynamic, and continuous batching

Take a batch of eight chat requests on Llama-3-8B and give them a realistic spread of output lengths — some one-line answers, one long explanation:

The batch runs for $\max_i n_i = 1024$ steps because that is when the last row stops. Every step processes all eight rows, so the machine performs $8 \times 1024 = 8192$ row-steps and only 1968 of them produce a token. 76% of the decode work is padding.

Put a wall-clock number on it. A Llama-3-8B decode step is bounded by streaming the weights: 8.03 B parameters at bf16 is 16.1 GB, and an H100 SXM has 3.35 TB/s of HBM bandwidth, so the floor is $16.1/3350 \approx 4.8$ ms per step at 100% of peak bandwidth, ignoring KV traffic entirely. That step cost is essentially independent of batch size in this range, which is the whole reason batching is the lever.

And that is only the throughput half. The latency half is worse: a request that arrives one microsecond after the batch launches waits 4.9 s before its first token, because there is no mechanism to put it into a batch that is already running.

A decode step is a function of a set of rows, not of a fixed tensor. Row $i$ contributes one query token and reads its own KV history; nothing in the arithmetic couples row $i$ to row $j$ except that they share the same weight matrices. That is the entire insight. If rows are independent, then the set of rows can change between steps at no cost — you are not resizing a persistent computation, you are calling a function with a different-sized argument. Static batching treats the batch as an object with a lifetime; continuous batching treats it as an argument.

Define the batch utilisation of a static batch of $B$ requests with output lengths $n_1 \dots n_B$ as the fraction of row-steps that produce a token:

where $\bar{n}$ is the mean output length and $n_{\max}$ the longest. The batch is only as efficient as the ratio of the mean to the maximum. For the table above, $1968 / 8192 = 0.240$.

For an illustrative length model, take independent exponential lengths of mean 1/lambda. Their expected maximum is H_B/lambda. Dividing the expected mean length by the expected maximum gives 1/H_B, but E[mean/max] is not exactly this ratio of expectations. This is a useful approximation, not a universal law of output lengths. Truncated, fixed-length, and Pareto distributions have different behavior.

Read the last column against §1.1: you raise $B$ specifically to move decode rightward on the roofline, and static batching gives you $B/\ln B$ of what you paid for. Going from $B=32$ to $B=256$ — an 8× increase in KV cache footprint and in the memory you must reserve — buys 5.3× effective batch. The padding tax grows exactly as fast as the thing you are trying to buy.

Fix 1: dynamic batching

The first fix, inherited from vision and recommendation serving, is to stop pre-forming batches. A dynamic batcher holds arriving requests in a queue and forms a batch when either $B_{\max}$ requests have accumulated or a timeout expires. This is genuinely useful for a model whose forward pass has fixed cost per input — an image classifier, an embedding model — because it removes the batch-boundary padding: you no longer pad a batch of 3 out to 8.

It does nothing for the problem above. Dynamic batching fixes when the batch is formed; the entire 76% loss comes from what happens after it is formed. Rows still cannot leave and new rows still cannot join. Note also the second-order damage: to make dynamic batching pay you raise the timeout, which adds queueing delay to TTFT on exactly the light-load requests where TTFT is most visible (§1.2).

Fix 2: iteration-level scheduling

Orca (Yu et al., OSDI 2022) made the scheduling granularity one iteration instead of one request. The scheduler runs before every forward pass; it may admit newly arrived requests into the next iteration and retire finished ones immediately after. A request's residency in the batch is $[\text{admit}, \text{finish}]$ rather than $[\text{batch start}, \text{batch end}]$, so the useful admitted-row utilization, ignoring padding and speculative/arrival effects, becomes

and the engineering problem shifts entirely from "how do I avoid padding" to "how do I keep the batch full". The paper reports up to 36.9× throughput over FasterTransformer at the same latency level for a 175B model — a cited number from the OSDI paper, on their hardware and their trace, not a claim about any current engine.

The catch: attention will not batch like a Linear will

If you try to implement iteration-level scheduling naively you hit a wall in the first attention layer. Consider a batch containing three rows mid-decode with KV lengths 1841, 96 and 5023, plus one row doing a 2048-token prefill. Every op except attention is per token: QKV projection, MLP, LayerNorm, residual add. They do not care which sequence a token came from, so you can flatten the whole batch into a $[\sum_i N_i,\; d]$ matrix and issue one GEMM. Attention is per sequence: row 2's query must attend to row 2's 96 KV entries and nothing else, and there is no rectangular tensor of shape $[B, S, \cdot]$ that holds 1841, 96 and 5023 without padding to 5023 — reintroducing the exact waste you set out to remove.

Orca's answer is selective batching: batch every op that is per-token, and split the batch before attention so attention runs per sequence. Modern engines keep the idea and drop the literal split. Instead the ragged layout is handed straight to a kernel that consumes offset and length arrays. This is why every attention interface you will meet in Part 3 takes vectors, not shapes:

class FlashAttentionMetadata:
    # NOTE(sang): Definition of context_len, query_len, and seq_len.
    # |---------- N-1 iteration --------|
    # |---------------- N iteration ---------------------|
    # |- tokenA -|......................|-- newTokens ---|
    # |---------- context_len ----------|
    # |-------------------- seq_len ---------------------|
    #                                   |-- query_len ---|

    num_actual_tokens: int  # Number of tokens excluding padding.
    max_query_len: int
    query_start_loc: torch.Tensor
    max_seq_len: int
    seq_lens: torch.Tensor
    block_table: torch.Tensor
    slot_mapping: torch.Tensor

query_start_loc is the exclusive prefix sum of per-request query lengths — it cuts the flat token axis back into sequences. seq_lens gives each sequence's KV length independently. block_table lets each sequence's KV live in scattered pages (§2.2). Together they are the data structure that makes a heterogeneous batch a single kernel launch. Note the field named num_actual_tokens: int # Number of tokens excluding padding. — padding has not vanished from the system, it has been demoted to a CUDA-graph alignment concern rather than a per-request one (§8.1 explains why replaying a captured graph requires a fixed token count in the first place).

Both engines implement iteration-level scheduling; they disagree about what the batch is. vLLM keeps a long-lived, slot-indexed InputBatch and mutates it in place. SGLang builds a ScheduleBatch of tensors and rebuilds them by gather. Both are correct; the tradeoff is real and shows up in different places.

vLLM: schedule running first, then admit

The V1 scheduler has no phase distinction at all — the comment at the top of schedule() is the clearest statement of the design in either codebase:

    def schedule(self, throttle_prefills: bool = False) -> SchedulerOutput:
        self.current_step += 1
        # NOTE(woosuk) on the scheduling algorithm:
        # There's no "decoding phase" nor "prefill phase" in the scheduler.
        # Each request just has the num_computed_tokens and
        # num_tokens_with_spec. num_tokens_with_spec =
        # len(prompt_token_ids) + len(output_token_ids) + len(spec_token_ids).
        # At each step, the scheduler tries to assign tokens to the requests
        # so that each request's num_computed_tokens can catch up its
        # num_tokens_with_spec. This is general enough to cover
        # chunked prefills, prefix caching, speculative decoding,
        # and the "jump decoding" optimization in the future.

A request is a pair (num_computed_tokens, num_tokens_with_spec) and the scheduler hands out tokens from a budget until the budget runs out. The two loops are literally labelled:

        # First, schedule the RUNNING requests.
        req_index = 0
        while req_index < len(self.running) and token_budget > 0:
            request = self.running[req_index]

        # Next, schedule the WAITING requests.
        if not preempted_reqs and self._pause_state == PauseState.UNPAUSED:
            step_skipped_waiting = create_request_queue(self.policy)

            while (self.waiting or self.skipped_waiting) and token_budget > 0:

Ordering matters and is a policy decision, not an accident. Running requests get first claim on token_budget, so an in-flight decode is never starved by a newly arrived 8k-token prompt. Waiting requests are admitted only with what is left over. The guard if not preempted_reqs is the safety interlock: if this very step had to evict someone to make room, do not turn around and admit someone new.

The retire side lives in update_from_output, after sampling:

        # Remove the stopped requests from the running and waiting queues.
        if stopped_running_reqs:
            self.running = remove_all(self.running, stopped_running_reqs)

with _free_request (vllm/v1/core/sched/scheduler.py:L2422-L2442) recording the id in self.finished_req_ids, which is the channel by which the worker learns to drop the row.

vLLM: the persistent batch and condense()

The scheduler owns request lifecycle; the GPU worker owns a slot-indexed mirror of it. Every per-request tensor on the worker — token ids, sampling temperature, top-p, block table rows — is allocated once at max_num_seqs rows and indexed by slot. Adding a request means writing one row; removing one means marking a slot free. Nothing is reallocated.

Admission picks the lowest free slot rather than appending:

    def _register_add_request(self, request: "CachedRequestState") -> int:
        """Track add-request operations for logits processors.
        Not applicable to pooling models.
        """

        # Fill the next empty index if there is one.
        if (new_req_index := self.batch_update_builder.pop_removed()) is None:
            # Append to end otherwise.
            new_req_index = self.num_reqs

        assert new_req_index < self.max_num_reqs
        self.batch_update_builder.batch_changed = True

Removal punches a hole and explicitly defers repair:

    def remove_request(self, req_id: str) -> int | None:
        """This method must always be followed by a call to condense().

        Args:
          req_id: request to remove

        Returns:
          Removed request index, or `None` if `req_id` not recognized
        """

        req_index = self.req_id_to_index.pop(req_id, None)
        if req_index is None:
            return None

        self.batch_update_builder.removed_append(req_index)
        self._req_ids[req_index] = None
        self.req_output_token_ids[req_index] = None
        self.spec_token_ids[req_index].clear()
        self.block_table.clear_row(req_index)

And condense() is the repair: slide the highest occupied slot down into the lowest hole, repeatedly, until the occupied slots are a dense prefix [0, num_reqs). Density is not cosmetic — every downstream kernel and every CUDA graph is launched over [0, num_reqs), so a hole in the middle would mean either launching over max_num_seqs rows or carrying an index indirection into every kernel.

    def condense(self) -> None:
        """Slide non-empty requests down into lower, empty indices.

        Any consecutive empty indices at the very end of the list are not
        filled.
# ...
        num_reqs = self.num_reqs

        if not (empty_req_indices := self.batch_update_builder.removed):
            # All removed requests were replaced by added requests, or else no
            # requests were removed at all. No condense() needed
            return
# ...
        # NOTE(woosuk): This function assumes that the empty_req_indices
        # is sorted in descending order.
        last_req_index = num_reqs + len(empty_req_indices) - 1
        while empty_req_indices:
            # Find the largest non-empty index.
            while last_req_index in empty_req_indices:
                last_req_index -= 1

            # Find the smallest empty index.
            empty_index = self.batch_update_builder.peek_removed()
            assert empty_index is not None
            if empty_index >= last_req_index:
                break
# ...
            num_tokens = self._get_active_token_count(last_req_index)
# ...
            self.token_ids_cpu[empty_index, :num_tokens] = self.token_ids_cpu[
                last_req_index, :num_tokens
            ]
# ...
            self.block_table.move_row(last_req_index, empty_index)

Three details worth stopping on. First, the early return: if every hole was already refilled by an add_request in the same iteration — the common steady-state case, one request out, one request in — condense() does no work at all. Second, self.token_ids_cpu[empty_index, :num_tokens] copies only the live token prefix, not the full max_model_len row; moving a slot in a 128×131072 int32 array would otherwise cost 512 KB per move. Third, the move is recorded in batch_update_builder.moved so that stateful logits processors can follow their request to its new slot.

vLLM V2: no condense, an index map instead

The V2 runner keeps the same idea — per-request state preallocated at max_num_reqs rows, indexed by slot — and drops the density invariant. RequestState (vllm/v1/worker/gpu/states.py:L9-L133) holds a plain free-list, self.free_indices = list(range(max_num_reqs)) at vllm/v1/worker/gpu/states.py:L29, and removal simply hands the slot back:

    def remove_request(self, req_id: str) -> int | None:
        """Return the freed slot index, or None if the request was not found."""
        req_idx = self.req_id_to_index.pop(req_id, None)
        if req_idx is None:
            return None
        self.index_to_req_id.pop(req_idx, None)
        self.free_indices.append(req_idx)
        return req_idx

There is no condense() anywhere in vllm/v1/worker/gpu/, because the holes are never repaired. Instead every step builds the very indirection condense() existed to avoid — a batch-position-to-slot map, materialised fresh per step and copied to the device:

        idx_mapping_iter = map(self.req_states.req_id_to_index.__getitem__, req_ids)
        idx_mapping_np = np.fromiter(idx_mapping_iter, dtype=np.intp, count=num_reqs)

idx_mapping goes to the GPU at vllm/v1/worker/gpu/model_runner.py:L1129 and is then threaded into every Triton kernel in prepare_inputs — prepare_prefill_inputs, prepare_pos_seq_lens, combine_sampled_and_draft_tokens — and into the sampler's per-request state. The kernels gather through it rather than assuming [0, num_reqs). That is exactly the “carry an index indirection into every kernel” option the V1 design rejected, and V2 took it: one intp gather per per-request read buys the removal of all slot-move bookkeeping, and the batch order is then free to be whatever the runner wants — which it uses, sorting decode-like requests ahead of prefills every step (sort_batch_req_ids, vllm/v1/worker/gpu/model_runner.py:L2012-L2024), a reordering condense()'s dense-prefix invariant would have fought.

So the three-way comparison is really: V1 pays row copies to keep kernels index-free; V2 pays an index map to make removal free; SGLang pays a re-gather of every field to make removal uniform. V2 and SGLang have converged from opposite directions on the same conclusion — the indirection is cheaper than the bookkeeping.

SGLang: filter and merge

SGLang's loop is a plain while True with the same three phases in the same order:

    def event_loop_normal(self):
        """A normal scheduler loop."""
        while True:
            if self.gracefully_exit:
                break

            # Receive requests
            recv_reqs = self.request_receiver.recv_requests()
            self.process_input_requests(recv_reqs)
            if self._engine_paused:
                continue

            # Get the next batch to run
            plan = self.get_next_batch_to_run(
                running_batch=self.running_batch, last_batch=self.last_batch
            )
            self.running_batch = plan.running_batch
            batch = plan.batch_to_run
            self.cur_batch_for_debug = batch

            # Launch the current batch
            if batch:
                result = self.run_batch(batch)
                self.process_batch_result(batch, result)
            else:
# ...
                self.on_idle()

            # Update last_batch
            self.last_batch = batch

The interesting structure is in get_next_batch_to_run. Where vLLM produces one mixed batch per step, SGLang decides between a prefill batch and a decode batch, and the merge of last step's prefill into the running set happens at the top of the next iteration:

            # Filter batch
            last_bs = last_batch.batch_size()
            last_batch.filter_batch(chunked_req_to_exclude=list(chunked_req_to_exclude))
            if last_batch.batch_size() < last_bs:
                running_batch.batch_is_full = False

            # Merge the new batch into the running batch.
            if not last_batch.is_empty():
                if running_batch.is_empty():
                    running_batch = last_batch
                else:
                    # Merge running_batch with prefill batch
                    running_batch.merge_batch(last_batch)

        if new_batch is not None:
            # Run prefill first if possible
            ret = new_batch
        else:
            # Run decode (skip for prefill-only batches)
            if not running_batch.is_empty() and not running_batch.is_prefill_only:
                running_batch = self.update_running_batch(running_batch)
                ret = running_batch if not running_batch.is_empty() else None
            else:
                ret = None

Retirement is filter_batch, called unconditionally at the head of every decode step:

    def update_running_batch(self, batch: ScheduleBatch) -> Optional[ScheduleBatch]:
        """Update the current running decoding batch."""
        initial_bs = batch.batch_size()

        batch.filter_batch()
        if batch.is_empty():
            batch.batch_is_full = False

and filter_batch itself is a gather, not a compaction:

    def filter_batch(
        self,
        chunked_req_to_exclude: Optional[Union[Req, List[Req]]] = None,
        keep_indices: Optional[List[int]] = None,
    ):
        if keep_indices is None:
# ...
            keep_indices = [
                i
                for i in range(len(self.reqs))
                if not self.reqs[i].finished()
                and self.reqs[i] not in chunked_req_to_exclude
            ]
# ...
        if len(keep_indices) == len(self.reqs):
            # No need to filter
            return

        keep_indices_device = torch.tensor(
            keep_indices,
            dtype=torch.int64,
            pin_memory=is_pin_memory_available(self.device),
        ).to(self.device, non_blocking=True)
# ...
        self.reqs = [self.reqs[i] for i in keep_indices]
# ...
        self.req_pool_indices = self.req_pool_indices[keep_indices_device]
        self.req_pool_indices_cpu = self.req_pool_indices_cpu[keep_indices]
        self.seq_lens = self.seq_lens[keep_indices_device]
        self.orig_seq_lens = self.orig_seq_lens[keep_indices_device]
        self.out_cache_loc = None
        # Sum is recomputed lazily by ForwardBatch.init_new.
        self.seq_lens_sum = None

and merge_batch is a concatenation:

    def merge_batch(self, other: ScheduleBatch):
        # Penalizer orchestrator must be merged before Batch.reqs is merged. This is because
        # orchestrator.merge() depends on Batch.reqs during preparation of each penalizers, so it
        # needs to be called with pre-merged Batch.reqs.
        self.sampling_info.merge_batch(other.sampling_info)
# ...
        self.req_pool_indices = torch.cat(
            [self.req_pool_indices, other.req_pool_indices]
        )
        self.req_pool_indices_cpu = torch.cat(
            [self.req_pool_indices_cpu, other.req_pool_indices_cpu]
        )
        self.seq_lens = torch.cat([self.seq_lens, other.seq_lens])
        self.orig_seq_lens = torch.cat([self.orig_seq_lens, other.orig_seq_lens])
        self.out_cache_loc = None

Why the designs differ

The two are duals of the same operation and the difference is where the cost lands.

Neither is obviously better and both have converged on the same escape hatch: skip the work entirely when the batch did not change. vLLM's V2 runner is a third point on the same line — it skips the repair altogether and pays a gather instead, which is closer to SGLang's position than to its own predecessor's. The real divergence is upstream — vLLM's single mixed budget versus SGLang's prefill-or-decode choice, which is §1.5's subject.

The engine core loop is three calls (vllm/v1/engine/core.py:L1391-L1399 drives _process_input_queue then _process_engine_step):

    def step(self) -> tuple[dict[int, EngineCoreOutputs], bool]:
        """Schedule, execute, and make output.
# ...
        # Check for any requests remaining in the scheduler - unfinished,
        # or finished and not yet removed from the batch.
        if not self.scheduler.has_requests():
            return {}, False
        scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())
        future = self.model_executor.execute_model(scheduler_output, non_block=True)
# ...
        engine_core_outputs = self.scheduler.update_from_output(
            scheduler_output, model_output
        )

Say slots 0..7 hold R0..R7, and at iteration $t$ the sampler emits EOS for R0 while R8 sits in waiting. The scheduler half is runner-independent; the worker half below is the V1 runner, chosen because its slot arithmetic is explicit. On V2 steps 1 and 2 are identical and step 3 becomes “free_indices gets slot 0 back, add_request pops it again, and the next idx_mapping_np names slot 0 in position 0” — same end state, no condense to early-return from. In order:

1. Iteration $t$, update_from_output detects the stop, appends R0 to stopped_running_reqs, and at vllm/v1/core/sched/scheduler.py:L2060-L2062 rebuilds self.running without it. _free_request puts "R0" into self.finished_req_ids and releases its KV blocks.
2. Iteration $t{+}1$, schedule() walks the seven remaining running requests first, then reaches the WAITING loop at L755, finds R8, allocates its blocks, appends it to self.running, and puts it in scheduled_new_reqs. The SchedulerOutput carries finished_req_ids = {"R0"} alongside it.
3. _update_states on the worker consumes both halves in that order:

        # Remove the finished requests from the persistent batch.
        # NOTE(woosuk): There could be an edge case where finished_req_ids and
        # scheduled_req_ids overlap. This happens when a request is aborted and
        # then resubmitted with the same ID. In this case, we treat them as two
        # distinct requests - clearing the cached states for the first request
        # and handling the second as a new request.
        for req_id in scheduler_output.finished_req_ids:
            self.input_batch.remove_request(req_id)

        # Add the new or resumed requests to the persistent batch.
        # The smaller empty indices are filled first.
        for request in reqs_to_add:
            self.input_batch.add_request(request)
            self.input_batch.update_req_spec_token_ids(request, scheduled_spec_tokens)

        # Condense the batched states if there are gaps left by removed requests
        self.input_batch.condense()
        # Allow attention backend to reorder the batch, potentially
        self._may_reorder_batch(scheduler_output)
        # Refresh batch metadata with any pending updates.
        self.input_batch.refresh_metadata()

remove_request("R0") marks slot 0 removed. add_request(R8) calls _register_add_request, which pops slot 0 off the removed list and writes R8 there. condense() then hits its early return: the removed list is empty, so no rows move. The batch is still a dense [0, 8), the block table row for slot 0 now points at R8's freshly allocated blocks, and refresh_metadata() pushes the dirty rows to GPU. The attention kernel at iteration $t{+}1$ sees seq_lens[0] equal to R8's prompt length instead of R0's — one array element changed, no tensor was reallocated, and the forward pass has unchanged decode-only cost only after R8's prompt KV is available. Prefilling a new prompt consumes compute and token budget; slot replacement does not make that work free.

SGLang reaches the same end state by a different route: update_finish_state() sets req.finished_reason and release_kv_cache returns the pages (python/sglang/srt/managers/scheduler_components/batch_result_processor.py:L273-L278); the next call to update_running_batch runs filter_batch(), which builds keep_indices = [1,2,3,4,5,6,7] and re-gathers seq_lens, req_pool_indices and friends into fresh length-7 tensors. R8, having been prefilled as its own batch in some earlier iteration, is torch.cat-ed on by merge_batch.

Continuous batching has no answer for KV exhaustion

Iteration-level scheduling solves padding. It does not create memory. Every admitted row needs its KV cache to stay resident for as long as it is in the batch, and each row grows by one token's worth every step. Admit aggressively enough and the pool runs dry mid-decode, with the requests already halfway through their answers. Both engines then have to take memory away from somebody.

In vLLM the trigger is allocate_slots returning None inside the running loop:

            # Schedule newly needed KV blocks for the request.
            with record_function_or_nullcontext("schedule: allocate_slots"):
                while True:
                    new_blocks = self.kv_cache_manager.allocate_slots(
                        request,
                        num_new_tokens,
                        num_lookahead_tokens=self.num_lookahead_tokens,
                    )

                    if new_blocks is not None:
                        # The request can be scheduled.
                        break

                    # The request cannot be scheduled.
                    # Preempt the lowest-priority request.
                    if self.policy == SchedulingPolicy.PRIORITY:

In SGLang it is check_decode_mem() failing, and it prints a line you will see in production logs:

            msg_prefix = (
                "KV cache pool is full. Retract requests. "
                if kv_full_retract_flag
                else "Testing retraction. "
            )
            msg_details = f"#retracted_reqs: {len(retracted_reqs)}, #new_tokens_gained: {new_token_gained}"
# ...
            logger.warning(msg_prefix + msg_details)

A steady stream of KV cache pool is full. Retract requests. is not a memory bug — it is the scheduler telling you it admitted more than it can carry, and you are now paying to recompute prefills you already paid for. Preemption policy, retraction versus swapping, and the new_token_ratio feedback loop that damps the oscillation are §1.4's subject.

A long prefill still stalls every decode in the batch

Continuous batching makes the batch composition dynamic; it does not make an iteration cheap. If one iteration includes an 8k-token prefill, every decode row in that same iteration waits for the whole prefill GEMM to finish. Inter-token latency spikes for the entire batch, and the spike is proportional to the longest prompt anyone submitted. Chunked prefill — splitting the prompt so no single iteration is long — is §1.5. You can see the hook already in the code above: vLLM's long_prefill_token_threshold caps num_new_tokens in both loops.

The persistent batch assumes overlap between consecutive steps

The vLLM model runner spells out the assumption behind its design, and the failure mode when it does not hold: unscheduled requests are removed from the persistent batch too (vllm/v1/worker/gpu_model_runner.py:L1286-L1308), so a workload that alternates between two disjoint request sets — some priority-scheduling configurations, some multi-LoRA setups where max_loras forces a round-robin — will move nearly every row every step. The comment is blunt: "If batches have low request overlap (e.g., alternating between two distinct sets of requests), this optimization becomes very inefficient." Nothing crashes; you just lose CPU time in condense() and H2D copies that a gather-based design would not have spent.

Max batch size is two independent limits

Both engines cap the batch twice: by request count and by token count. In vLLM these are max_num_seqs and max_num_batched_tokens (vllm/config/scheduler.py:L42-L44 holds the library defaults of 128 and 2048, which EngineArgs.create_engine_config overrides for real deployments). In SGLang the request cap is --max-running-requests (python/sglang/srt/server_args.py:L789-L791). Raising the request cap without KV cache to back it converts a throughput knob into a preemption generator.

Reproduce the padding tax by turning continuous batching off, which you do by pinning the batch to one row so that no interleaving is possible, and comparing against a normal batch on the same trace.

# Serial baseline: one row at a time, no batching at all.
# This measures batching benefit, NOT a static B>1 straggler-padding baseline.
vllm serve meta-llama/Meta-Llama-3-8B-Instruct --max-num-seqs 1 --max-num-batched-tokens 8192 --port 8000

# Stop the first server before launching this comparison on the SAME GPU.
# Continuous batching with a real batch
vllm serve meta-llama/Meta-Llama-3-8B-Instruct --max-num-seqs 128 --max-num-batched-tokens 8192 --port 8001

# Drive both with the same request stream
vllm bench serve --model meta-llama/Meta-Llama-3-8B-Instruct --dataset-name sharegpt --num-prompts 500 --request-rate 8 --port 8000
vllm bench serve --model meta-llama/Meta-Llama-3-8B-Instruct --dataset-name sharegpt --num-prompts 500 --request-rate 8 --port 8001

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --max-running-requests 1 --port 30000
python -m sglang.bench_serving --backend sglang --dataset-name sharegpt --num-prompts 500 --request-rate 8 --port 30000

# then re-launch with --max-running-requests 128 and repeat

What to watch, beyond the output-token throughput line: sweep --max-num-seqs over 1, 4, 16, 64, 256 and plot throughput against it. The curve should climb steeply and then flatten — the flattening point is where decode stops being weight-bound and starts being bound by KV reads plus attention, which is the crossover derived in §0.4. If instead throughput falls at the top of the sweep, check the logs for retraction or preemption warnings: you have admitted more rows than the KV pool can carry.

• Orca: A Distributed Serving System for Transformer-Based Generative Models — Yu et al., OSDI 2022. Sections 3 (iteration-level scheduling) and 4 (selective batching) are the source of everything in this chapter. Read the scheduler pseudocode in Fig. 8 against vllm/v1/core/sched/scheduler.py.
• vLLM announcement post — the original statement of continuous batching plus PagedAttention as a pair, and why the memory half was the blocker.
• vLLM RFC #8779, "vLLM's V1 Engine Architecture" — the design discussion behind the V1 scheduler, the persistent batch, and the removal of the prefill/decode distinction from the scheduler.
• SGLang server_args.py at the pinned SHA — the scheduling knobs (--max-running-requests, --schedule-policy, --schedule-conservativeness) that control admission aggressiveness, and therefore how often you see retraction.
• Anyscale, "How continuous batching enables 23x throughput in LLM inference" — a clear public writeup of the static-versus-continuous comparison with published measurements. Their numbers, their hardware; treat them as cited, not as something this chapter reproduced.