Data parallelism, attention-DP, expert parallelism

Consider an MLA cache-sizing thought experiment with eight ranks and a fixed 40 GiB KV pool on each rank. It is not a runnable eight-H100 DeepSeek-V3 deployment: approximately 671 GB of FP8 weights alone exceeds eight 80 GB GPUs, before scales, workspaces and KV. First isolate cache replication; then include the weight budget below.

§3.5 established why and verified it in both engines: MLA's KV cache is replicated across tensor-parallel ranks, not sharded. vLLM's get_num_kv_heads returns 1 at the use_mla branch before the TP division; SGLang's MLA branch of _compute_cell_size takes no parallel argument at all. The per-token cell is $L \cdot (r_{kv} + d_{\text{rope}}) \cdot b = 61 \times 576 \times 2 = 70{,}272$ bytes on every rank, so a 40 GiB pool holds $42{,}949{,}672{,}960 / 70{,}272 = 611{,}192$ tokens per rank — and all eight ranks hold the same 611,192. At fixed per-rank pool size, adding TP ranks adds no distinct KV capacity. Weight sharding can still free memory and enlarge each pool.

The second symptom is on the FLOP side. DeepSeek-V3's MoE layer has 256 routed experts of 2,048 intermediate width; shard each eight ways with TP and every rank runs a $K = 7168 \to N = 512$ GEMM per expert — a shape that leaves most of a tensor core idle, repeated over the experts receiving tokens in the current batch. A token uses its selected eight routed experts, not all 256; a sufficiently large, balanced batch may activate nearly all experts. Both symptoms have the same fix, and it is not a kernel: stop splitting the tensor, start splitting the data and the experts. §5.1 owns TP, §5.2 PP, §5.4 the collectives, and §7.1 MoE routing and grouped GEMM as a modelling topic. What follows is the parallelism of it.

Every parallelism axis answers one question: what do I split, and what must I therefore communicate? TP splits the weight matrices and pays an all-reduce over activations. PP splits the layers and pays a point-to-point send. DP splits the batch and pays nothing — until the ranks must share a weight matrix, when it pays a gather. EP splits the experts and pays an all-to-all, because which rank a token needs is decided at run time by the router.

The interesting deployments are hybrids. DP attention runs attention data-parallel — each rank owns whole sequences, so each KV entry lives on exactly one rank — while the MoE stays tensor- or expert-parallel. That needs a shape change at the attention↔FFN boundary of every layer, and everything hard about DP attention lives in that shape change.

Plain data parallelism

The baseline: $N$ replicas, each a complete engine with a complete copy of the weights and its own KV pool, behind a router. Throughput can approach linear scaling in $N$ when load balances and the router, tokenizer and external dependencies do not bottleneck; model execution needs no inter-replica collectives. SGLang's data-parallel controller is exactly that router:

class LoadBalanceMethod(Enum):
    """Load balance method."""

    ROUND_ROBIN = auto()
    FOLLOW_BOOTSTRAP_ROOM = auto()
    TOTAL_REQUESTS = auto()
    TOTAL_TOKENS = auto()

Dispatch is a greedy pick over shared-memory load snapshots — least outstanding requests, or least resident tokens with requests as tie-break (python/sglang/srt/managers/data_parallel_controller.py:L114-L130). Two consequences. $N$ replicas hold $N$ copies of the weights, so plain DP is only affordable for small models. And prefix caches do not share across replicas: a conversation routed to replica 3 on turn one and replica 6 on turn two re-prefills from scratch. None of these four methods knows anything about prefixes, which is why cache-aware routing exists — §9.4.

DP attention: turning a replicated cache into a sharded one

Attention is embarrassingly parallel over sequences: nothing in the computation for sequence $a$ touches sequence $b$. So if rank 0 owns $\{a, b\}$ and rank 1 owns $\{c, d\}$, each rank holds only its own KV — and the eight copies of the MLA latent that TP forced on you become eight different caches instead of eight identical ones. A dense FFN needs its complete transformation per token; a routed MoE needs only the selected experts. If those required weights are sharded across ranks, communication must assemble the corresponding computation. The layer therefore has two halves with incompatible data layouts. SGLang names them:

class ScatterMode(Enum):
    """
    Suppose we have TP=4, DP=2, enable-dp-attention, and the system handles seq a,b,c,d
    Model input/output: [ab, ab, cd, cd] for four ranks respectively
    SCATTERED: [a, b, c, d]
    TP_ATTN_FULL: [ab, ab, cd, cd], i.e. all ranks inside a TP attn group have full data of the group
    FULL: [abcd, abcd, abcd, abcd]
    MOE_FULL: full within the MoE group (cp_per_moe CP chunks), used when moe_dp_size < attn_cp_size
    """

    SCATTERED = auto()
    TP_ATTN_FULL = auto()
    FULL = auto()
    MOE_FULL = auto()

Attention runs in TP_ATTN_FULL, a TP-sharded FFN in FULL; getting between them is an all-gather in and a reduce-scatter out, once per layer. The rank arithmetic that carves the world into an attention-DP grid is four lines:

def compute_dp_attention_world_info(
    enable_dp_attention, tp_rank, tp_size, dp_size, attn_cp_size: int = 1
):
    attn_dp_size = dp_size if enable_dp_attention else 1
    attn_tp_size = tp_size // attn_dp_size // attn_cp_size
    attn_tp_rank = tp_rank % attn_tp_size

    if not enable_dp_attention:
        attn_dp_rank = 0
    else:
        # Rank layout is (dp, cp, tp) where tp is the fastest-changing dim:
        # tp_rank = (attn_dp_rank * attn_cp_size + attn_cp_rank) * attn_tp_size + attn_tp_rank
        attn_dp_rank = tp_rank // (attn_tp_size * attn_cp_size)

    return attn_tp_rank, attn_tp_size, attn_dp_rank, attn_dp_size

The canonical DeepSeek configuration is tp_size = dp_size = 8, giving attn_tp_size = 1: attention is pure DP, each rank alone with its own sequences and KV pool. The flag's help text says exactly this — "Enabling data parallelism for attention and tensor parallelism for FFN. The dp size should be equal to the tp size." (python/sglang/srt/server_args.py:L1163-L1170).

The capacity arithmetic, done twice

The clean version first, holding the KV pool fixed at 40 GiB per GPU exactly as §3.5 did. Under TP=8 all eight ranks hold identical latents, so cluster distinct resident tokens is 611,192; under DP=8 each pool holds different sequences, so the cluster holds $8 \times 611{,}192 = 4{,}889{,}536$. Eight times in this fixed-pool layout comparison, not a universal flag-level speedup.

That flatters DP attention, because it hides the cost: with attn_tp_size = 1 every rank keeps a full copy of the attention weights instead of an eighth. Do it again with weights accounted, from the published shapes ($d = 7168$, $n_h = 128$, $r_q = 1536$, $r_{kv} = 512$, $d_{\text{rope}} = 64$, $d_{\text{nope}} = d_v = 128$):

DeepSeek-V3 in fp8 is 671 GB, of which 11.41 GB is attention. That does not fit on eight 80 GB H100s at all, so compare at 16 GPUs, budgeting 72 GB per GPU for weights plus KV (8 GB reserved for activations and workspaces — an assumption, not a measurement):

Replicating attention costs 10.7 GB per rank and cuts the per-rank pool by 36%. It still wins by 10.3× cluster-wide, because the factor of 16 from sharding dominates. That is the whole argument, and it is specific to models where attention weights are small relative to the experts: on a dense Llama-3-70B you would replicate attention to gain nothing, since GQA at $\text{TP} \le h_{kv} = 8$ already shards the cache.

What the boundary costs

Per layer, per rank, the DP-attention boundary with a TP-sharded FFN moves

with $N$ the full TP world size, $T_g$ the global token count this step, $d$ hidden size, $b$ bytes per element — an all-gather in, a reduce-scatter out. For DeepSeek-V3 at $N = 8$, $T_g = 1024$, $d = 7168$, bf16: $2 \times \tfrac{7}{8} \times 1024 \times 7168 \times 2 = 25.7$ MB per layer per rank, or 1.49 GB per rank per decode step over 58 MoE layers. At NVLink 4's 900 GB/s aggregate per H100 (450 GB/s one direction, from the NVIDIA H100 product page) that is 3.3 ms of ideal transfer time per decode iteration producing 1,024 tokens (about 3.2 µs per output token when amortizing aggregate throughput, not per-request TPOT) — which is why DP attention is almost always paired with an all-to-all MoE that eliminates the gather entirely.

Under EP, expert $e$ lives on exactly one rank (a few, once EPLB starts replicating); rank $r$ holds $E/N$ of the $E$ experts and nothing else. The router picks $k$ experts per token at run time, so before the grouped GEMM every token must be sent to the ranks holding its experts, and afterwards sent back and weighted-summed. That pair — dispatch and combine — is an all-to-all: every rank sends a different amount to every other rank, and the amounts are unknown until the router has run.

Communication volume

Naively, dispatch sends each token $k$ times. Real libraries deduplicate by destination rank: two chosen experts on the same rank cost one crossing, not two. So per-rank dispatch volume is

Here $T_\ell$ is the local token count. Uniform selection of $k$ distinct experts from $E$ gives the hypergeometric probability above. Network traffic excludes the local rank. The simpler $N[1-(1-1/N)^k]$ is a with-replacement approximation counting all destinations, including local ones. Real grouped routing, skew and placement require empirical destination counts. We count bytes sent per rank; received bytes are a separate equal expectation under balance. Combine is symmetric in rows but not bytes: dispatch is narrowed to fp8 while combine returns model dtype. In normal mode SGLang narrows it itself with sglang_per_token_group_quant_fp8(hidden_states, 128, ...) (python/sglang/srt/layers/moe/token_dispatcher/deepep.py:L527-L534); in low-latency mode it passes use_fp8=True into the kernel. SGLang's Ascend buffer-sizing formula budgets TOTAL_SEQ_LEN * (hidden_size + hidden_size) * topk — two passes of $T k d$ (Expert Parallelism guide).

Contrast with TP. At $N=8$, $T_\ell = 128$, $d = 7168$: EP moves $128 \times 4.63282 \times 7168 \times 1.03125 = 4.38$ MB dispatched and 8.50 MB combined, 12.88 MB sent per layer per rank, against 25.7 MB for DP-gather-plus-TP-FFN. EP is about 2.0× smaller in ideal sent bytes under these assumptions. The all-to-all is not the expensive primitive by volume; it is expensive because it is irregular (per-peer sizes are data-dependent, so the kernel cannot be shape-specialised), latency-sensitive (two round trips × 58 layers = 116 serialised collectives per decode step), and because at scale it crosses node boundaries where bandwidth drops an order of magnitude.

Why EP is really about weight bandwidth

What actually decides EP size at decode is not communication but HBM reads. Each DeepSeek-V3 expert is $3 \times 2048 \times 7168 = 44.04$ MB in fp8; all 256 are 11.27 GB per layer. For a large balanced batch that activates all experts, each resident expert's weights may need to be streamed once per step. This is not a universal decode lower bound: kernels can skip experts with no assigned tokens, and cache reuse changes HBM traffic. Under independent uniform top-$k$ routing across $B$ tokens, the expected number of active experts is $E[1-(1-k/E)^B]$. The following is the all-experts-active traffic model:

In that all-experts-active model, TP=8 reads the same 1.41 GB per rank — it holds a slice of every expert rather than all of a few — so its bandwidth floor matches EP=8. What differs is arithmetic intensity: 256 skinny $K{=}7168 \to N{=}512$ expert GEMMs on the tokens routed to each expert, versus 32 wider local experts. Both TP and EP reduce resident expert-weight bytes per rank, approximately as $1/N$; EP changes ownership and GEMM shapes, not that capacity scaling law. Wide EP trades kernel efficiency and weight traffic against routing imbalance, topology and collective latency. The best deployment depends on batch size, routing and hardware.

SGLang: one process world, modes per layer

SGLang runs one TP world and re-partitions it per layer. Whether a layer needs the DP gather at all is decided once, at model construction:

    @classmethod
    def _compute_mlp_mode(cls, context: _LayerModeComputationContext):
        if context.is_layer_sparse:
            if (
                # Token dispatch/combine will be handled outside of LayerCommunicator for these modes.
                not get_moe_a2a_backend().is_none()
                or should_use_flashinfer_cutlass_moe_fp4_allgather()
                or enable_dwdp()
            ):
                return ScatterMode.SCATTERED
            # DSA CP and MLA CP both don't support MOE_FULL yet; fall back to FULL.
            if is_enable_moe_cp_allgather() and not (
                is_dsa_enable_prefill_cp() or is_mla_prefill_cp_enabled()
            ):
                return ScatterMode.MOE_FULL
            return ScatterMode.FULL

With --moe-a2a-backend deepep sparse layers are SCATTERED — no gather, no scatter, the dispatcher owns all movement. Without it they are FULL and you pay the 25.7 MB per layer computed above. One flag, two different communication graphs.

When the gather is needed, SGLang picks between two implementations based on how uneven the ranks are:

        # we choose the mode that minimizes the communication cost
        # prefer MAX_LEN when communication cost is equal to enable symmetric memory
        max_len = max(global_num_tokens)
        sum_len = sum(global_num_tokens)
        if sum_len * 2 >= max_len * dp_size:
            return cls.MAX_LEN
        else:
            return cls.SUM_LEN

MAX_LEN pads every rank to the longest and uses all_gather_into_tensor, moving $N \cdot \max_i T_i$ rows; SUM_LEN zero-fills a $\sum_i T_i$-row buffer and all-reduces. The test picks whichever is cheaper, and the gap between them is exactly the cost of DP-attention imbalance — Exercise 1 works a case where it is 8×.

The other half of DP attention lives in the scheduler and is the operationally important half. Because the FFN is a collective, every DP rank must enter every forward pass, work or no work:

    # Decide whether to emit idle batch
    if skip_all_gather:
        # Skip idle batch when attn-dp=1 (and always under DWDP: ranks run independently)
        need_idle_batch = not dwdp and dp_size > 1
    else:
        need_idle_batch = max(mlp_sync_info.global_num_tokens) > 0

    batch_to_gather = local_batch
    if need_idle_batch:
        if local_batch is None:
            batch_to_gather = local_batch = get_idle_batch()

Every scheduler iteration all-gathers each rank's token count (python/sglang/srt/managers/scheduler_components/dp_attn.py:L131-L193); if any rank has tokens, ranks with none fabricate an empty ScheduleBatch to reach the collectives. DP attention is not $N$ independent engines — it is one engine in lockstep whose attention happens to be partitioned, and that lockstep is the source of most DP-attention production pain.

vLLM: N engine processes, one flattened MoE world

vLLM arrives from the opposite direction. Its DP ranks are separate EngineCore processes with separate schedulers and separate KV caches — plain DP by construction — and the MoE layer is what reaches across them:

    data_parallel_size: int = Field(default=1, ge=1)
    """Number of data parallel groups. MoE layers will be sharded according to
    the product of the tensor, prefill-context, and data parallel sizes."""

The flattening is literal: DP and PCP are folded into the MoE's TP rank space, and with enable_expert_parallel that flattened size becomes the EP size.

            When TP = 2, DP(PCP) = 2 and EP = True, the configuration on different
                devices:

            - device 0: TP = {1, 0} DP = {2, 0} EP = {4, 0}
            - device 1: TP = {1, 0} DP = {2, 0} EP = {4, 1}
            - device 2: TP = {1, 0} DP = {2, 1} EP = {4, 2}
            - device 3: TP = {1, 0} DP = {2, 1} EP = {4, 3}
            - Comment: There are 2 engine instances and the experts are split
                between the 4 devices.

Note TP = {1, 0} in every EP row: turning on EP sets the MoE's tensor-parallel size to 1 outright (vllm/model_executor/layers/fused_moe/config.py:L1243-L1253). EP and TP are alternatives inside the MoE layer, not composable there. And vLLM's simplest all-to-all backend is SGLang's gather/scatter under a different name:

class AgRsAll2AllManager(All2AllManagerBase):
    """
    An implementation of all2all communication based on
    all-gather (dispatch) and reduce-scatter (combine).
    """

Because vLLM's DP ranks are separate processes, its lockstep problem is worse than SGLang's and it needs a dedicated process to solve it:

    * Keeps track of the current DP "request wave" number and running state
      of the engines. This is received from the DP rank 0 engine and published
      to the front-end processes along with the current load stats.

      The engines alternate between a global running/paused state. The global
      "request wave" number is a count of the number of times that the workers
      collectively move from a running state to a paused state. This transition
      is synchronized via the all-reduce operation performed in the
      DPEngineCoreProc._has_global_unfinished_reqs method.

An idle vLLM DP rank runs execute_dummy_batch() (vllm/v1/engine/core.py:L2202-L2206) — the same fabricated forward pass SGLang calls an idle batch. The world only stops when a synchronising all-reduce agrees, amortised to once every 32 steps:

    def _has_global_unfinished_reqs(self, local_unfinished: bool) -> bool:
        # Optimization - only perform finish-sync all-reduce every 32 steps.
        self.step_counter += 1
        if self.step_counter % 32 != 0:
            return True

Every EP number above assumed uniform routing. Routing is a learned, data-dependent function, and on real traffic some experts receive several times the mean while others receive almost none. Because dispatch, grouped GEMM and combine are all collective, the slowest rank sets the step time for the whole cluster: one rank holding two hot experts stalls seven idle peers. Both engines measure this with the same statistic. SGLang:

def compute_utilization_rate(
    gpu_physical_count_of_batch: torch.Tensor,  # (..., num_layer, num_gpu)
):
    """output: utilization_rate (..., num_layer)"""
    gpu_physical_count_of_batch = gpu_physical_count_of_batch.float()
    max_gpu_physical_count = einops.reduce(
        gpu_physical_count_of_batch,
        "... num_layer num_gpu -> ... num_layer",
        "max",
    )
    avg_gpu_physical_count = einops.reduce(
        gpu_physical_count_of_batch,
        "... num_layer num_gpu -> ... num_layer",
        "mean",
    )
    return (avg_gpu_physical_count + 1e-5) / (max_gpu_physical_count + 1e-5)

vLLM computes the identical ratio and calls it balancedness, logging "EPLB step: %d for model %s: avg_tokens=%.2f, max_tokens=%d, balancedness=%.4f" (vllm/distributed/eplb/eplb_state.py:L600-L612). It is the fraction of your purchased MoE FLOPs you are actually using: 0.55 means 45% of expert compute is stall.

The rebalancing idea

EPLB — DeepSeek's Expert-Parallel Load Balancer, vendored verbatim into SGLang — has two moves. Replication: allocate more physical expert slots than logical experts, and hand each spare slot to whichever expert currently has the highest load per replica. A greedy loop:

    n, num_log = weight.shape
    num_redundant = num_phy - num_log
    assert num_redundant >= 0
    device = weight.device
    phy2log = torch.arange(num_phy, dtype=torch.int64, device=device).repeat(n, 1)
    rank = torch.zeros(n, num_phy, dtype=torch.int64, device=device)
    logcnt = torch.ones(n, num_log, dtype=torch.int64, device=device)
    arangen = torch.arange(n, dtype=torch.int64, device=device)
    for i in range(num_log, num_phy):
        redundant_indices = (weight / logcnt).max(dim=-1).indices
        phy2log[:, i] = redundant_indices
        rank[:, i] = logcnt[arangen, redundant_indices]
        logcnt[arangen, redundant_indices] += 1

Permutation: pack the physical slots onto GPUs so per-GPU load sums are as equal as possible. The full algorithm is hierarchical and topology-aware — groups onto nodes, then replication within a node, then slots onto GPUs within a node, so traffic prefers NVLink:

    # Step 1: pack groups to nodes
    tokens_per_group = weight.unflatten(-1, (num_groups, group_size)).sum(-1)
    group_pack_index, group_rank_in_pack = balanced_packing(tokens_per_group, num_nodes)
# ...
    # Step 2: construct redundant experts within nodes
    # [num_layers * num_nodes, num_logical_experts // num_nodes]
    tokens_per_mlog = weight.gather(-1, mlog2log).view(
        -1, num_logical_experts // num_nodes
    )
    phy2mlog, phyrank, mlogcnt = replicate_experts(
        tokens_per_mlog, num_physical_experts // num_nodes
    )

    # Step 3: pack physical_experts to GPUs
    # [num_layers * num_nodes, num_physical_experts // num_nodes]
    tokens_per_phy = (tokens_per_mlog / mlogcnt).gather(-1, phy2mlog)
    pack_index, rank_in_pack = balanced_packing(tokens_per_phy, num_gpus // num_nodes)

Replication is what makes this work. Permutation alone cannot fix a single expert taking 2× the mean — no packing puts it anywhere but on some GPU. Replication splits it in two, halving per-replica load, for one extra 44 MB slot. The trade is HBM for balancedness, budgeted as ep_num_redundant_experts added to the logical count (python/sglang/srt/eplb/expert_location.py:L237-L241) or vLLM's eplb_config.num_redundant_experts (vllm/config/parallel.py:L72-L73).

What rebalancing costs

The manager is a generator that yields once per forward pass and falls into a rebalance every eplb_rebalance_num_iterations steps (default 1000):

    # can be more complex if needed
    def _entrypoint(self):
        while True:
            for _ in range(self._rebalance_num_iterations):
                yield

            yield from self.rebalance()

Moving expert $e$ from rank 3 to rank 5 means copying 44 MB of fp8 weights over the fabric, and a full permutation can move a large fraction of every layer. SGLang does it with batched point-to-point sends, chunked to keep NCCL from choking:

        # Submit P2P ops in batches to prevent NCCL/RCCL GPU-side accumulation
        # hangs on large rebalances. All ranks use the same expert_id ranges
        # (based on num_physical_experts) so matching send/recv pairs land in
        # the same batch. Set batch_chunk_size >= num_physical_experts to disable.
        batch_chunk_size = envs.SGLANG_EPLB_P2P_BATCH_CHUNK_SIZE.get()
        ops_by_expert = {eid: ops for eid, ops in sorted_infos}
        for start in range(0, num_physical_experts, batch_chunk_size):
            batch_ops = []
            for eid in range(
                start, min(start + batch_chunk_size, num_physical_experts)
            ):
                if eid in ops_by_expert:
                    batch_ops.extend(ops_by_expert[eid])
            if batch_ops:
                reqs = torch.distributed.batch_isend_irecv(batch_ops)
                for req in reqs:
                    req.wait()

eplb_rebalance_layers_per_chunk spreads the move across several forward passes so no single step spikes, and eplb_min_rebalancing_utilization_threshold (default 1.0) suppresses the rebalance entirely when measured utilisation already exceeds it (python/sglang/srt/server_args.py:L2435-L2450). vLLM uses a 1000-step sliding window with a 3000-step rearrangement interval and an async path (vllm/config/parallel.py:L62-L87). Same design points, different defaults, and no published measurement behind either set that I could find in these trees.

Wide EP is brittle: with 64 ranks in the all-to-all, one dead GPU takes down the deployment. SGLang's python/sglang/srt/elastic_ep/ is the machinery for surviving that and for growing the EP world without a restart. What it concretely does, from source:

DeepSeek-V3 on SGLang, --tp 8 --dp-size 8 --ep 8 --enable-dp-attention --moe-a2a-backend deepep. Follow one decode token that landed on DP rank 3, through one sparse layer.

1. DataParallelController.total_tokens_scheduler routed the request at admission to the rank with the fewest resident tokens (python/sglang/srt/managers/data_parallel_controller.py:L114-L130). Rank 3 owns this sequence for life; its KV blocks never leave.
2. prepare_mlp_sync_batch_raw all-gathers every rank's token count into global_num_tokens; ranks with nothing to do build an idle batch (python/sglang/srt/managers/scheduler_components/dp_attn.py:L359-L369). All eight now agree on the global shape. initialize_dp_attention resolved the grid at startup: attn_tp_size = 8 // 8 // 1 = 1, so attn_dp_rank = tp_rank = 3 (python/sglang/srt/layers/dp_attention.py:L326-L340).
3. MLA runs on rank 3 alone against rank 3's KV pool, emitting [1, 7168] inside a [T_3, 7168] local tensor. LayerScatterModes._compute_mlp_mode sees a sparse layer and a non-none a2a backend, so it returns ScatterMode.SCATTERED (python/sglang/srt/layers/communicator.py:L385-L400): no dp_gather happens and Figure 2's 12.85 MB all-gather is skipped.
4. The router produces topk_ids [T_3, 8] over 0..255, and ExpertLocationDispatcher maps each logical expert to one of its physical replicas (python/sglang/srt/eplb/expert_location_dispatch.py) — with EPLB running a hot expert has several, and the choice spreads load.
5. _DeepEPDispatcherImplLowLatency.dispatch_a reaches _dispatch_core, which calls buffer.low_latency_dispatch(hidden_states, topk_ids, self.num_max_dispatch_tokens_per_rank, self.num_experts, use_fp8=self.use_fp8, ...) (python/sglang/srt/layers/moe/token_dispatcher/deepep.py:L766-L776) — the fp8 narrowing happens inside the DeepEP kernel here, while the normal-mode path does it explicitly with sglang_per_token_group_quant_fp8. Our token's hidden vector targets an expected 5.29 distinct ranks, of which 4.63 are remote, under uniform distinct-expert routing. Actual destination counts follow the router. Every rank contributes to the same collective — which is why step 2's idle batches were mandatory.
6. On each destination rank, the received rows land in per-expert buckets (packed_recv_count) and a grouped GEMM runs 32 groups, streaming 1.41 GB of fp8 expert weights. The straggler rank — the one whose 32 experts happened to attract the most rows — finishes last, and everyone waits.
7. combine_a / combine_b send the eight partial outputs back and apply the topk_weights sum, producing [1, 7168] on rank 3 again. Along the way on_deepep_dispatch_low_latency incremented this layer's per-expert counters (python/sglang/srt/layers/moe/token_dispatcher/deepep.py:L726-L730); after 1000 such steps EPLBManager._entrypoint falls out of its yield loop and starts moving 44 MB expert tensors over batch_isend_irecv.

Real deployments mix all four axes. The decision procedure, in order, because each step constrains the next:

1. PP only if you must. It exists to cross a fabric too slow for TP. Inside an NVLink domain, skip it — bubbles are pure loss (§5.2).
2. EP as wide as the experts allow, subject to $E \bmod \text{EP} = 0$. This sets per-rank expert HBM and therefore the decode latency floor.
3. Attention: DP if MLA, TP if GQA. With MLA, TP adds zero KV capacity, so DP wins if you can afford replicated attention weights. With GQA at $\text{TP} \le h_{kv}$, TP already shards the cache and DP only wastes memory.
4. TP fills what is left. SGLang requires tp_size % dp_size == 0 (python/sglang/srt/server_args.py:L6742-L6744) and the DeepEP, Mooncake, NIXL and pplx backends additionally require ep_size == tp_size. vLLM instead derives EP $= \text{TP} \times \text{PCP} \times \text{DP}$ automatically (vllm/model_executor/layers/fused_moe/config.py:L1202-L1210).
5. Then EPLB. It optimises placement within a fixed EP world, so settle the topology first, then measure balancedness.

Worked: Qwen3-235B-A22B fp8 on one 8×H100 node

235 B parameters in fp8 is 235 GB — 29.4 GB per GPU across eight, comfortable. The shapes are confirmed in-tree by SGLang's grouped-GEMM benchmark: $d = 4096$ (the k of the gate-up GEMM), 128 experts, $2 \times 768 \times 4/2 = 1536$ intermediate width at TP=4:

            # Prefill, Qwen3-235B-A22B-FP8, gateup, chunk_size = 16384, TP = 4
            ShapeArg(expected_m_per_group=1024, n=768, k=4096, num_groups=128),
            # Prefill, Qwen3-235B-A22B-FP8, down, chunk_size = 16384, TP = 4
            ShapeArg(expected_m_per_group=1024, n=4096, k=384, num_groups=128),
            # Decode, Qwen3-235B-A22B-FP8, gateup, bs = 256, TP = 4
            ShapeArg(expected_m_per_group=16, n=768, k=4096, num_groups=128),
            # Decode, Qwen3-235B-A22B-FP8, down, bs = 256, TP = 4
            ShapeArg(expected_m_per_group=16, n=4096, k=384, num_groups=128),

(The prefill row also pins top-$k$: $16384 \times k / 128 = 1024 \Rightarrow k = 8$.) Applying the procedure: no PP; EP=8, giving 16 experts per rank at $3 \times 1536 \times 4096 = 18.9$ MB each, so 302 MB of expert weights per layer per rank. Qwen3 uses GQA with $h_{kv} = 4$, so step 3 says attention TP, not DP — but the a2a backends may demand ep_size == tp_size == 8. That equality alone does not imply dp_size = 8; additional backend and DP-attention validation determines legal combinations. So: either accept replicated attention weights (small here) to get the DeepEP path, or set --moe-a2a-backend none and keep attn_tp = 8 with an all-gather/reduce-scatter MoE. A genuine fork with no universal answer — measure both.

This source-derived DeepSeek configuration is a hardware-dependent sketch, not an eight-H100 recipe. Budget the FP8 weights, replicated attention, 32 redundant experts per layer, scales, workspace and KV before launching. Eight 80 GB H100s cannot fit the default checkpoint. Use a sufficiently provisioned topology, pinned compatible DeepEP/DeepGEMM builds and the environment preflight. Run one configuration at a time and verify readiness; this GPU launch has not been executed locally.

python -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-V3 \
    --tp 8 --dp-size 8 --ep 8 \
    --enable-dp-attention \
    --moe-a2a-backend deepep \
    --moe-runner-backend deep_gemm \
    --enable-eplb \
    --ep-num-redundant-experts 32 \
    --expert-distribution-recorder-mode stat

Three things to measure, none of which need a benchmark harness:

1. The capacity claim. Start with and without --enable-dp-attention, read the KV pool line the scheduler logs, and multiply by dp_size in the DP case. You must compare actual logged pools rather than assume 8×; Figure 3's left pair deliberately fixes the pool size.
2. Balancedness. --expert-distribution-recorder-mode stat accumulates the per-expert counts that compute_utilization_rate turns into Figure 5's ratio; on vLLM, --eplb-config.log_balancedness=true prints it. Watch it either side of the rebalance at step 1000.
3. The gather that disappears. Profile one decode step with --moe-a2a-backend deepep and again with none, counting NCCL kernels per layer. The deepep run has no AllGather in the sparse layers at all — that is _compute_mlp_mode returning SCATTERED.

On vLLM the equivalent single-node start is vllm serve <model> --data-parallel-size 8 --enable-expert-parallel; the offline form is in examples/features/data_parallel/data_parallel_offline.py:L4-L9.

• LMSYS, "Deploying DeepSeek with PD Disaggregation and Large-scale Expert Parallelism" — the primary write-up behind SGLang's EP stack, including its EPLB and two-batch-overlap sections. Cited by SGLang's own expert parallelism guide.
• deepseek-ai/EPLB — the upstream of python/sglang/srt/eplb/eplb_algorithms/deepseek.py, vendored because it is not a PyPI package (see the file's first line).
• deepseek-ai/DeepEP — the all-to-all library, including the normal/low-latency split and the NVLink-forwarding design for inter-node dispatch. SGLang's dispatcher wraps its low_latency_dispatch / low_latency_combine directly.
• DeepSeek-V3 Technical Report — the source for the 671 B total / 37 B active parameter counts and the group-limited routing that caps how many nodes a token can reach.
• SGLang PR #6121 — DP attention extended beyond MLA to standard-attention Qwen models, cited from SGLang's DP/DPA guide.
• SGLang PR #9269 — the MoE refactor behind the BaseDispatcher / MoeRunnerCore split in python/sglang/srt/layers/moe/token_dispatcher/base.py; PR #13327 for the dispatcher hooks behind single-batch overlap.
• Next: §5.4 on the collectives themselves and what topology does to them, and §7.1 on the routing function that produces the imbalance this chapter spends so much effort correcting.