Pipeline parallelism and bubbles

Llama-3-70B in bf16 is 131 GiB of weights. One H100 SXM has 80 GB. Eight of them, on one node, with NVLink between them, hold it comfortably at TP=8: 16.4 GiB of weights per GPU, ~52 GiB left over for KV. That configuration works and it is what most people run.

Double the context or the concurrency and you need sixteen GPUs. Assume an eight-GPU HGX H100 node with one local NVSwitch domain, so sixteen GPUs span two such nodes. You have two nodes and 400 Gb/s InfiniBand NDR between them (50 GB/s per direction, NVIDIA Quantum-2 NDR, cited). Two things go wrong at once if you answer "TP=16".

The KV heads run out. Llama-3-70B has $h_{kv}=8$. Past TP=8 the KV heads are replicated, not split — §5.1 owns this. Each of the sixteen GPUs still stores a full KV head's worth of every layer. You bought eight more GPUs and the per-token KV footprint per GPU did not move.

The all-reduces cross the wire. A Llama block does two all-reduces — after o_proj, after down_proj — so $L=80$ means 160 collectives per forward pass on $[T, 8192]$ bf16 tensors. Ring all-reduce moves $2\frac{N-1}{N}M$ bytes out of every rank: at $T=64$ and $N=8$ that is 1.84 MB per collective and 294 MB per rank per decode step. Over NVLink at 450 GB/s per direction (H100 datasheet: 900 GB/s bidirectional, cited) that is 0.65 ms against a 5.2 ms step. Over 50 GB/s InfiniBand it is 5.9 ms — more than the compute, before counting 320 synchronisation latencies. Derived, not measured.

And if you try PP on a model whose vLLM implementation was never wired for it, you get this instead of a server:

        pipeline_parallel_size = parallel_config.pipeline_parallel_size
        if pipeline_parallel_size > 1 and not self.registry.is_pp_supported_model(
            self.architectures, self
        ):
            raise NotImplementedError(
                "Pipeline parallelism is not supported for this model. "
                "Supported models implement the `SupportsPP` interface."
            )

Tensor parallelism cuts every weight matrix the short way and makes eight GPUs cooperate on one layer. Pipeline parallelism cuts the stack: GPU 0 owns layers 0–19, GPU 1 owns 20–39, and so on. No matrix is split, so no collective is needed inside a layer; the only thing that ever leaves a stage is the activation at the seam, a point-to-point send of shape $[T, d]$. That is why the wire cost collapses: TP pays $2L$ collectives per forward pass, PP pays $S-1$ sends. For Llama-3-70B at $L=80$, $S=4$, that is 160 all-reduces versus 3 sends.

What actually crosses the seam

Read the model, not the paper. vLLM's Llama returns this when it is not the last stage:

        if get_pp_group().is_first_rank:
            if inputs_embeds is not None:
                hidden_states = inputs_embeds
            else:
                hidden_states = self.embed_input_ids(input_ids)
            residual = None
        else:
            assert intermediate_tensors is not None
            hidden_states = intermediate_tensors["hidden_states"]
            residual = intermediate_tensors["residual"]
# ...
        if not get_pp_group().is_last_rank:
            return IntermediateTensors(
                {"hidden_states": hidden_states, "residual": residual}
            )

Two tensors, not one. The second is the un-added residual, and it exists because both engines fuse the residual add into the next RMSNorm. SGLang's Llama does the identical thing and leaves a note about it:

        if self.pp_group.is_first_rank:
            if input_embeds is None:
                hidden_states = self.embed_tokens(input_ids)
            else:
                hidden_states = input_embeds
            residual = None
        else:
            assert pp_proxy_tensors is not None
            # FIXME(@ying): reduce the number of proxy tensors by not fusing layer norms
            hidden_states = pp_proxy_tensors["hidden_states"]
            residual = pp_proxy_tensors["residual"]

So the seam payload is $2 \cdot T \cdot d \cdot b$ bytes: at $d=8192$, $b=2$, that is 32 KiB per token per seam. Compare against TP=8's $160 \times 2\frac{7}{8} d b = 4.375$ MiB per rank per token. 140×, derived.

Both engines then shard that seam send across the TP group and rebuild it with a local all-gather:

        for key, tensor in zip(tensor_keys, tensor_list):
            if tensor.numel() == 0:
                continue

            if self._should_use_all_gather(
                key, tensor.numel(), all_gather_group, all_gather_tensors
            ):
                tensor = tensor.reshape(all_gather_size, -1)[all_gather_rank]

With all_gather_group=get_tp_group() (vllm/v1/worker/gpu_worker.py:L1134-L1138) each of the eight TP ranks pushes 1/8 of the seam tensor over its own NIC and the receiving node all-gathers over NVLink. The cross-node bytes per rank per token drop to 4 KiB. SGLang passes attn_tp_group for the same reason (python/sglang/srt/managers/scheduler_pp_mixin.py:L1044-L1052).

The bubble

Define $S$ = number of stages, $M$ = number of micro-batches, $t$ = time one stage takes on one micro-batch (assume balanced stages). In the naive GPipe schedule, micro-batch $j$ enters stage $s$ at time $(s + j)t$, so the last micro-batch leaves the last stage at $(S - 1 + M)t$. Makespan:

Total device-time is $S \cdot T_{\text{total}}$; useful device-time is $S \cdot M \cdot t$. The idle fraction — the bubble — is

At $S=4$: $M=1$ gives $\beta = 75\%$, $M=4$ gives $42.9\%$, $M=16$ gives $15.8\%$, $M=64$ gives $4.5\%$. The training remedy is simply "make $M$ big".

Why the training remedy does not transfer to decode

In training, $t$ scales down with $M$ because the forward pass is compute-bound: half the tokens, half the FLOPs, half the time. The bubble shrinks and the makespan barely moves.

Decode is not compute-bound. A decode step is below the ridge point — ~98% of the traffic is reading weights, and weight traffic is independent of batch size. So halving a decode micro-batch does not halve $t$; it leaves $t$ essentially unchanged. Feed the formula that fact:

Nor is there a smaller unit to split: in decode each sequence contributes exactly one token per step, and you cannot subdivide a token. What both engines do instead is batch-level pipelining: keep $S$ independent batches in flight, drawn from the continuous-batching queue (§1.3), and let each stage work on a different one. There is no fill/drain per step because the stream never ends. The bubble does not leave the formula — it moves into a load condition: if fewer than $S$ batches exist, stages idle.

Capacity, not latency — the blunt version

Let $W$ be the weight bytes read per forward pass and $B_{\text{HBM}}$ the per-GPU bandwidth. With balanced stages, $t = W / (S \cdot B_{\text{HBM}})$, so

$S$ cancels out of the latency. In this fixed-TP, weight-bandwidth-only model, adding pipeline stages does not reduce per-token latency. A token still traverses all 80 layers, and the total weight bytes it forces off HBM is the same $W$ whether those bytes sit on one GPU or sixteen. What $S$ buys is throughput, linearly, and the ability to host $S\times$ the parameters.

TP, by contrast, does cut latency: at TP=8 each GPU reads $W/8$ concurrently, giving $T_{\text{token}} = 5.17$ ms plus collective time. That is the entire trade. TP buys latency at the cost of a link fast enough to carry 160 collectives; PP buys capacity and throughput at no link cost and no latency benefit.

Each stage allocates KV only for its own layers. Both engines resolve a [start_layer, end_layer) range at load time and size the pool from it. SGLang makes it explicit:

    pp_range = _resolve_pp_layer_range(model=model, model_num_layers=model_num_layers)
    num_effective_layers = pp_range.end_layer - pp_range.start_layer
# ...
    return ModelLayerInfo(
        start_layer=pp_range.start_layer,
        end_layer=pp_range.end_layer,
        num_effective_layers=num_effective_layers,
    )

and num_effective_layers is what the KV pool is sized with (python/sglang/srt/model_executor/pool_configurator.py:L150-L160, python/sglang/srt/mem_cache/kv_cache_configurator.py:L1123). vLLM gets there structurally: get_kv_cache_spec walks the attention modules actually present in the static forward context, and on a PP rank the absent ones are PPMissingLayer — torch.nn.Identity subclasses with no attention inside (vllm/v1/worker/gpu_model_runner.py:L7896-L7909).

For Llama-3-70B, per token per layer the KV is $2 \cdot h_{kv} \cdot d_h \cdot b = 2 \cdot 8 \cdot 128 \cdot 2 = 4$ KiB, so 320 KiB for the whole 80-layer stack. That divides by $S$. Here is why it matters, on 16 H100s at --gpu-memory-utilization 0.9 — an H100 80GB reports 79.65 GiB (§2.1), so 71.69 GiB usable, of which 3 GiB is reserved for activations and graphs:

Doubling TP from 8 to 16 buys 16% more concurrency, because past $h_{kv}=8$ every extra rank stores a replicated KV head and only the weight shard shrinks. Adding a pipeline stage instead buys 132%, on the same sixteen GPUs, over the slow link. This is the most under-appreciated property of PP in serving.

vLLM: layer stubbing plus a future queue in the engine core

Layer assignment is one function. Note the docstring's reasoning about where the remainder goes — it is the balance argument from Figure 4:

def get_pp_indices(
    num_hidden_layers: int, pp_rank: int, pp_size: int
) -> tuple[int, int]:
    """Try to evenly distribute layers across partitions.

    If the number of layers is not divisible by the number of partitions,
    the remaining layers are evenly distributed across all but the last
    partition. The last partition is excluded because it often contains an
    additional norm layer and we are attempting to balance compute.

    If `pp_size > 2` and the number of remaining layers is
    `0 < x <= pp_size - 2` then the remaining layers are evenly distributed
    across the middle partitions. The first and last partitions are excluded
    because they contain the input and output embeddings respectively and we
    are attempting to reduce maximum memory consumption across partitions.
    """

Every model then builds a full-length ModuleList in which the foreign layers are placeholders, so state-dict keys and layer indices stay global:

class PPMissingLayer(torch.nn.Identity):
    """
    A placeholder layer for missing layers in a pipeline parallel model.
    """

    def __init__(self, *args, **kwargs):
        super().__init__()

    def forward(self, *args, **kwargs):
        """Return the first arg from args or the first value from kwargs."""
        return args[0] if args else next(iter(kwargs.values()))
# ...
    start_layer, end_layer = get_pp_indices(
        num_hidden_layers, get_pp_group().rank_in_group, get_pp_group().world_size
    )

    modules = torch.nn.ModuleList(
        [PPMissingLayer() for _ in range(start_layer)]
        + get_offloader().wrap_modules(
            layer_fn(prefix=f"{prefix}.{idx}") for idx in range(start_layer, end_layer)
        )
        + [PPMissingLayer() for _ in range(end_layer, num_hidden_layers)]
    )

The pipelining itself lives in the engine core, not the worker. max_concurrent_batches is the whole scheduler-side contract:

    @property
    def max_concurrent_batches(self) -> int:
        # PP requires PP-size concurrent batches to fill the pipeline.
        # Async scheduling requires 2 concurrent batches to overlap.
        pp_size = self.parallel_config.pipeline_parallel_size
        if self.scheduler_config.async_scheduling:
            if self.use_v2_model_runner:
                return pp_size + 1
            # V1 Model Runner does not fully support async scheduling with PP.
            if pp_size <= 1:
                return 2
        return pp_size

That number sizes a deque of in-flight futures, and step_with_batch_queue prefers filling the pipeline over collecting results:

        # Setup batch queue for pipeline parallelism.
        # Batch queue for scheduled batches. This enables us to asynchronously
        # schedule and execute batches, and is required by pipeline parallelism
        # to eliminate pipeline bubbles.
        self.batch_queue_size = vllm_config.max_concurrent_batches
        self.batch_queue: (
            deque[tuple[Future[ModelRunnerOutput], SchedulerOutput, Future[Any]]] | None
        ) = None
        if self.batch_queue_size > 1:
            logger.debug("Batch queue is enabled with size %d", self.batch_queue_size)
            self.batch_queue = deque(maxlen=self.batch_queue_size)

SGLang: a scheduler process per stage, running a ring

SGLang does not centralise. Every PP stage runs its own Scheduler with its own event loop, and requests, activations and sampled tokens all travel around the ring stage-to-stage. The docstring of event_loop_pp is the clearest single description of inference-time pipelining in either repository:

    def event_loop_pp(self: Scheduler):
        """
        A scheduler loop for pipeline parallelism.
        Notes:
        1. Each stage runs in the same order and is notified by the previous stage.
        2. We use async send but sync recv to avoid desynchronization while minimizing the communication overhead.
        3. We can use async batch depth to buffer the outputs in the last stage for to allow overlapping the GPU computation and CPU processing and avoid last PP rank staggler.

        Unified Schedule:
        ====================================================================
        Stage P
        recv ith req from previous stage
        recv ith proxy from previous stage
        run ith batch
        recv prev (i+1)% mb_size th outputs
        process batch result of prev (i+1)% mb_size th batch (can be run in parallel with the curr batch GPU computation)
        send ith req to next stage
        send ith proxy to next stage
        send current stage's outputs to next stage(can be stashed and delayed to send later)

        the above order can be optimized and reordered to minimize communication-related CPU stall and overhead bubbles.

        ====================================================================
        """

The in-flight count is explicit, and can exceed $S$:

    def init_pp_loop_state(self: Scheduler):
        self.pp_loop_size: int = self.ps.pp_size + get_parallel().pp_async_batch_depth
        # In CP mode, attention weights are duplicated, eliminating the need for the attention TP all-gather operation.
        self.require_attn_tp_allgather = (
            not get_parallel().enable_dsa_prefill_context_parallel
        )
        self.mbs = [None] * self.pp_loop_size
        self.last_mbs = [None] * self.pp_loop_size
        self.running_mbs = [
            ScheduleBatch(reqs=[], batch_is_full=False)
            for _ in range(self.pp_loop_size)
        ]
        self.mb_metadata: List[Optional[PPBatchMetadata]] = [None] * self.pp_loop_size

--pp-async-batch-depth (default 0, python/sglang/srt/server_args.py:L1054-L1056) adds slots beyond $S$ so the last rank can buffer sampled tokens and overlap its CPU post-processing with the next GPU batch — the "staggler" in the docstring. It is a bubble that only exists in inference: the last stage does the sampling and detokenisation handoff no other stage does, so its slot is systematically longer.

Where they differ, and why. vLLM keeps one scheduler and treats the pipeline as an asynchronous execution backend: the scheduler never knows which stage a batch is on, only that a future has not resolved. Preemption and prefix-cache bookkeeping stay in one place, at the cost of a scheduler-to-worker round trip per stage. SGLang replicates the scheduler per stage and forwards the request objects themselves down the ring (_pp_send_pyobj_to_next_stage, scheduler_pp_mixin.py:L967-L982): no central coordinator, cheaper per step, but every stage must independently reach the same batching decision — which is why the pipeline demultiplexes message kinds to survive the resulting reordering:

    def _pp_recv_typed_dict(
        self: Scheduler,
        expected_kind: str = "default",
        all_gather_group: Optional = None,
    ) -> Dict[str, torch.Tensor]:
        """Receive a typed tensor dict, demultiplexing by msg_type.

        If a message of the wrong kind is received, it's stashed in the queue
        and we continue receiving until we get the expected kind.
        """

The admission consequence

With $S$ batches in flight, a single stage's running batch can only be $1/S$ of the engine's total concurrency, or the pipeline oversubscribes the KV pool. SGLang encodes exactly that:

        if not get_parallel().pp_max_micro_batch_size:
            get_context().override(
                "scheduler.pp_max_micro_batch_size_default",
                pp_max_micro_batch_size=max(
                    self.max_running_requests // self.ps.pp_size, 1
                ),
            )
# ...
    def get_num_allocatable_reqs(self, running_bs):
        res = get_parallel().pp_max_micro_batch_size - running_bs
        res = min(res, self.req_to_token_pool.available_size())
        return res

Set --max-running-requests 256 at PP=4 and each micro-batch admits at most 64. Forget this and you will conclude PP "made batching worse" when what actually happened is that your per-step batch got divided by $S$ while your total concurrency did not.

One decode step for one batch through vLLM at PP=2, TP=8, in call order.

1. EngineCore.step_with_batch_queue (vllm/v1/engine/core.py:L624-L680) finds len(batch_queue) < 2, calls self.scheduler.schedule(), then execute_model(scheduler_output, non_block=True) and pushes the future. Because the queue is not yet full it returns None immediately — the scheduler schedules a second batch before collecting the first.
2. On PP rank 0, Worker.execute_model (vllm/v1/worker/gpu_worker.py:L1098-L1103) sees is_first_rank, skips the receive, and calls self.model_runner.execute_model(...) with intermediate_tensors=None.
3. GPUModelRunner._preprocess (gpu_model_runner.py:L3733-L3739) keeps intermediate_tensors = None for the first rank. LlamaModel.forward embeds the tokens, runs layers 0–39 via islice(self.layers, self.start_layer, self.end_layer), and because not get_pp_group().is_last_rank returns IntermediateTensors({"hidden_states": ..., "residual": ...}) (llama.py:L430-L433).
4. Back in the worker, the result is an IntermediateTensors, so the branch at gpu_worker.py:L1127-L1138 fires: isend_tensor_dict with all_gather_group=get_tp_group(). Each of the 8 TP ranks slices its 1/8 of both tensors (parallel_state.py:L1061-L1064) and posts a non-blocking torch.distributed.isend to the matching rank on the next node. execute_model returns None — rank 0 produces no sampler output at all.
5. On PP rank 1, Worker.execute_model hits not get_pp_group().is_first_rank and calls irecv_tensor_dict, wrapping the handles in an AsyncIntermediateTensors so the receive overlaps with the runner's own input preparation.
6. GPUModelRunner.sync_and_gather_intermediate_tensors (gpu_model_runner.py:L3472-L3500) copies the arrivals into the persistent buffers and, if sequence parallelism scattered the residual, all-gathers it back over the TP group first.
7. Layers 40–79 run, self.norm is a real RMSNorm on this rank, lm_head produces logits, the sampler runs, and a real ModelRunnerOutput comes back — which is what the future in step 1 finally resolves to.

PP is incompatible with the things that make a single-stage engine fast. SGLang refuses to start:

        if self.pp_size > 1:
            assert (
                self.disable_overlap_schedule and self.speculative_algorithm is None
            ), "Pipeline parallelism is not compatible with overlap schedule, speculative decoding"
            assert self.min_free_slots_delay is None, (
                "--min-free-slots-delay is not supported with pipeline "
                "parallelism: allocatable slots per microbatch are bounded by "
                "pp-max-micro-batch-size, so the threshold may never be reached"
            )

The overlap scheduler and PP want the same resource — another batch in flight — and fight over it. Speculative decoding (§6.2) is worse: draft-then-verify needs the sampled token back at the start of the model, an extra full lap of the ring per round. Piecewise CUDA graphs are also off (python/sglang/srt/server_args.py:L4640).

Uneven partitions. $L=80$ divides by 2, 4, 5, 8, 10 but not 3, 6, 7. At PP=6 vLLM logs "Hidden layers were unevenly partitioned: [...]. This can be manually overridden using the VLLM_PP_LAYER_PARTITION environment variable" (vllm/distributed/utils.py:L160-L166). The slowest stage sets the slot time, so the [13,13,13,14,14,13] split the code produces runs at the 14-layer stage's pace on every stage — 4.8% of your throughput lost to a rounding decision. If your stages have different KV footprints or a heavy lm_head, tune the partition by hand.

Preemption gets harder. With several batches in flight, a request the scheduler thinks it has finished may still be mid-pipeline:

            # With overlapping batches (async scheduling or PP), a step may
            # still be writing a freed request's KV blocks. A consumer KV
            # Connector can reallocate and fill those blocks via a load that
            # isn't ordered against that write, so defer freeing them.
            multiple_inflight_batches = self.vllm_config.max_concurrent_batches > 1
            if multiple_inflight_batches and kv_transfer_config.is_kv_consumer:
                self.defer_block_free = True

"PP made my TPOT worse." The ideal fixed-TP comparison predicts added boundary latency plus per-stage Python and launch overhead. If your dashboard shows unchanged-to-slightly-worse TPOT and much higher throughput after enabling PP, the system is working correctly.

# vLLM: TP inside the node, PP across nodes.
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 8 --pipeline-parallel-size 2 \
  --max-model-len 8192 --gpu-memory-utilization 0.9

# Confirm the layer split from the worker log prefixes (Worker_PP0_TP0 ... Worker_PP1_TP7),
# set by vllm/v1/executor/multiproc_executor.py:L1065-L1084.

# Force a deliberately unbalanced split and watch the slot time follow the slowest stage:
VLLM_PP_LAYER_PARTITION=48,32 vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 8 --pipeline-parallel-size 2

# SGLang: same shape. Note the mandatory --disable-overlap-schedule.
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-70B-Instruct \
  --tp-size 8 --pp-size 2 --disable-overlap-schedule \
  --nnodes 2 --node-rank 0 --dist-init-addr $HEAD:5000

# Raise the in-flight count past pp_size to hide the last-rank sampling stall:
  --pp-async-batch-depth 1

# Retune admission: at pp_size=2 each micro-batch defaults to max_running_requests // 2.
  --pp-max-micro-batch-size 128

Launch scope: the multi-node commands above are configuration sketches, not a complete cluster bootstrap. Provision matching environments and model access, run the complementary node-rank command on every node, verify rendezvous and readiness, and stop all workers before trying another configuration. The GPU runs have not been executed in this checkout.

The measurement worth taking: sweep request rate from 1 to saturation at PP=1 and PP=2 on the same total GPU count, plotting TPOT and output-token throughput separately. Changing PP at fixed GPU count also changes TP: there is no general doubling prediction. Separately, hold TP fixed and add a second pipeline stage on additional GPUs. In the ideal weight-bandwidth model, per-request latency is nearly unchanged while a filled pipeline can approach twice the aggregate throughput. At low load the added boundary increases latency; communication, KV traffic and stage imbalance can change both comparisons.

• Huang et al., GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (2018) — the source of the $(S-1)/(S+M-1)$ bubble.
• Narayanan et al., Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021) — interleaved 1F1B, which reduces the bubble to $\frac{1}{v}\frac{S-1}{M}$ for $v$ model chunks per device. Nothing in either serving engine implements it, for the reason in §5.2.3.
• Narayanan et al., PipeDream (2018) — asynchronous 1F1B; the batch-level pipelining the engines use is closer to this than to GPipe.
• vLLM PR #4412 — the original pipeline-parallel implementation, including the discussion of why the batch queue lives in the engine core.
• SGLang PR #5724 — the pipeline-parallel scheduler loop that became scheduler_pp_mixin.py.
• Neighbouring chapters: §5.1 (tensor parallelism and the $h_{kv}$ wall), §5.3 (DP and EP), §5.4 (what the collectives actually cost on real topologies), §5.5 (how the ranks get launched and wired), §2.1 (the KV sizing formula used above).