Worker orchestration and multi-node serving

Llama-3-70B on 8×H100 with TP=8. The server has been serving for two hours. Then throughput goes to zero. nvidia-smi shows all eight GPUs pinned at 100% SM utilisation and constant power draw. No exception has been raised. No request has returned. Five minutes later vLLM finally says something:

        def get_response():
            responses = []
            for mq in response_mqs:
                dequeue_timeout = (
                    None if deadline is None else max(0.0, deadline - time.monotonic())
                )
                try:
                    status, result = mq.dequeue(timeout=dequeue_timeout)
                except TimeoutError as e:
                    raise TimeoutError(f"RPC call to {method} timed out.") from e
                if status != WorkerProc.ResponseStatus.SUCCESS:
                    raise RuntimeError(
                        f"Worker failed with error '{result}', please check the"
                        " stack trace above for the root cause"
                    )
                responses.append(result)

The deadline is VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS, default 300 (vllm/envs.py:L243). SGLang has the same shape of failure and the same 300-second default, arriving through a watchdog thread instead of an RPC deadline:

        if self.dump_info is not None and (info_msg := self.dump_info()):
            logger.error(f"{self.debug_name} debug info:\n{info_msg}")

        pyspy_dump_schedulers()
        logger.error(
            f"{self.debug_name} watchdog timeout "
            f"({self.watchdog_timeout=}, {self.soft=})"
        )
        print(file=sys.stderr, flush=True)
        print(file=sys.stdout, flush=True)

        if not self.soft:
            # Wait for some time so that the parent process can print the error.
            time.sleep(5)
            self.parent_process.send_signal(signal.SIGQUIT)

Both engines default this timeout to 300 s (vllm/envs.py:L243; python/sglang/srt/server_args.py:L1261-L1265). Both then kill the whole server. Neither can do better, because the GPUs are not hung — they are waiting. One rank entered an all-reduce that another rank never entered, and NCCL's spin-wait looks exactly like useful work.

This chapter owns the process architecture that makes lockstep the default: how ranks get launched, who decides what they run, how the decision travels, and what happens when one dies. The parallelism strategies belong to §5.1–§5.4; admission logic belongs to §1.4.

A collective requires compatible participation, but is not a host-side barrier: asynchronous CUDA operations can return after enqueueing, before device completion. Completion and safe use of results depend on stream ordering or explicit synchronization. Disagreement can cause a hang, an error, or silently incorrect output when shapes agree but request identities differ. See PyTorch distributed operation semantics. If rank 0 schedules a batch of 32 sequences and rank 1 schedules 31, the first all-reduce inside the first attention layer will have mismatched shapes — or, worse, matched shapes and mismatched semantics — and the job may abort, hang until a watchdog intervenes, or reduce unrelated activations without detecting the semantic error.

So the whole architecture reduces to one requirement: every rank must execute the same sequence of steps, with the same shapes, in the same order. Two useful architectural approaches are centralized plans and replicated scheduling; the engines emphasize different approaches.

Read at these SHAs: vLLM's serving path is Model A and SGLang is Model B. That is the single most consequential architectural difference between the two engines, and almost everything else in this chapter follows from it. vLLM also ships a Model B path — ExecutorWithExternalLauncher — but only for offline torchrun jobs, and with an explicit caveat about determinism that we will come back to.

Take Llama-3-70B (L=80, d=8192, h=64, h_kv=8) at TP=8 on H100 SXM. Tensor parallelism inserts one all-reduce after the attention output projection and one after the MLP down projection, so a single forward pass crosses

collective barriers per rank per step, where $L$ is the number of transformer layers. How often does that happen? The model's 131 GiB of bf16 weights shard to $131/8 = 16.4$ GiB per rank; at the H100's 3.35 TB/s of HBM bandwidth the pure memory-bound floor for one decode step is

which is 190 steps/s. So at the roofline this job crosses roughly $160 \times 190 \approx 30{,}000$ collective barriers per second per rank (arithmetic from the roofline formula in FORMULAS plus published Llama-3-70B shapes; real systems land well below this, and §5.4 owns the all-reduce cost itself). Every one of those 30,000 barriers is a chance for two ranks to disagree, and a disagreement can stall the server or corrupt results, depending on whether the protocol detects it.

That asymmetry is what justifies paying for a plan broadcast. The cost side is small: vLLM's broadcast is a write into a shared-memory ring buffer sized by VLLM_MQ_MAX_CHUNK_BYTES_MB, default 16 MB, with a ZMQ fallback for objects that do not fit (vllm/envs.py:L242, vllm/envs.py:L1771-L1775). One such write per step against a 5.25 ms floor is not where your throughput went.

The benefit side is that the plan is defined to be identical on every rank, but it does not make arbitrary rank-dependent kernels harmless: collective order, metadata interpretation and persistent worker state still have to agree. Model B does not have that guarantee, and SGLang says so, in a comment that is the most honest sentence in either codebase about the risk it is taking:

    def _sync_token_ids_across_tp(
        self, batch_next_token_ids: torch.Tensor, sampling_info: SamplingBatchInfo
    ):
        if SYNC_TOKEN_IDS_ACROSS_TP or sampling_info.grammars:
            # For performance reasons, SGLang does not sync the final token IDs across TP ranks by default.
            # This saves one all-reduce, but the correctness of this approach depends on the determinism of several operators:
            # the last all-reduce, the last lm_head matmul, and all sampling kernels.
            # These kernels are deterministic in most cases, but there are some rare instances where they are not deterministic.
            # In such cases, enable this env variable to prevent hanging due to TP ranks becoming desynchronized.
            # When using xgrammar, this becomes more likely so we also do the sync when grammar is used.

            torch.distributed.all_reduce(
                batch_next_token_ids,
                op=dist.ReduceOp.MIN,
                group=self.tp_sync_group,
            )

In Model B the sampled token is part of the scheduling input for the next step: it determines sequence lengths, stop conditions, and therefore batch composition. If two ranks sample different tokens because a reduction reassociated differently, they will schedule different batches on the next step and the job hangs. SGLang's default is to bet that the kernels are deterministic, and to provide SYNC_TOKEN_IDS_ACROSS_TP=1 — one extra all-reduce per step — as the insurance policy. It forces the sync whenever grammar-constrained decoding is on, because that path is more likely to diverge.

vLLM: one scheduler, N worker processes

vLLM picks an executor class from one string:

            executor_class = distributed_executor_backend
        elif distributed_executor_backend == "ray":
            if envs.VLLM_USE_RAY_V2_EXECUTOR_BACKEND:
                from vllm.v1.executor.ray_executor_v2 import RayExecutorV2

                executor_class = RayExecutorV2
            else:
                from vllm.v1.executor.ray_executor import RayDistributedExecutor

                executor_class = RayDistributedExecutor
        elif distributed_executor_backend == "mp":
            from vllm.v1.executor.multiproc_executor import MultiprocExecutor

            executor_class = MultiprocExecutor
        elif distributed_executor_backend == "uni":
            from vllm.v1.executor.uniproc_executor import UniProcExecutor

            executor_class = UniProcExecutor
        elif distributed_executor_backend == "external_launcher":
            # TODO: make v1 scheduling deterministic
            # to support external launcher
            executor_class = ExecutorWithExternalLauncher

Note the TODO. vLLM's V1 scheduler is not currently guaranteed deterministic, which is exactly why the plan must be broadcast rather than recomputed.

MultiprocExecutor is the default for TP>1. It creates one shared-memory MessageQueue that it alone writes to and every worker reads from, exports a handle, then spawns one process per local rank:

            max_chunk_bytes = envs.VLLM_MQ_MAX_CHUNK_BYTES_MB * 1024 * 1024
            mq_connect_ip = get_ip()
            logger.info(
                "DP group leader: node_rank=%d, node_rank_within_dp=%d, "
                "master_addr=%s, mq_connect_ip=%s (local), "
                "world_size=%d, local_world_size=%d",
                self.parallel_config.node_rank,
                self.parallel_config.node_rank_within_dp,
                self.parallel_config.master_addr,
                mq_connect_ip,
                self.world_size,
                self.local_world_size,
            )
            self.rpc_broadcast_mq = MessageQueue(
                self.world_size,
                self.local_world_size,
                max_chunk_bytes=max_chunk_bytes,
                connect_ip=mq_connect_ip,
            )

The queue itself is a hybrid: a shared-memory ring buffer for local readers, a ZMQ XPUB socket for large payloads and for readers on other nodes (vllm/distributed/device_communicators/shm_broadcast.py:L488-L533). Each worker is a plain multiprocessing process with two extra pipes — a ready pipe child-to-parent, and a death pipe whose EOF tells the child its parent died:

        context = get_mp_context()
        # Ready pipe to communicate readiness from child to parent
        ready_reader, ready_writer = context.Pipe(duplex=False)
        # Death pipe to let child detect parent process exit
        death_reader, death_writer = context.Pipe(duplex=False)

Every worker then runs the same three-line loop forever. This is the entire control plane:

    def worker_busy_loop(self):
        """Main busy loop for Multiprocessing Workers"""
        assert self.rpc_broadcast_mq is not None
        while True:
            method, args, kwargs, output_rank = self.rpc_broadcast_mq.dequeue(
                indefinite=True
            )
            try:
                if isinstance(method, str):
                    func = getattr(self.worker, method)
                elif isinstance(method, bytes):
                    func = partial(cloudpickle.loads(method), self.worker)

                output = func(*args, **kwargs)

                if output_rank is None or self.rank == output_rank:
                    self.handle_output(output)

A worker is a method dispatcher. It has no admission scheduler, but it does have input queues and persistent model-runner request/KV state. The tuple on the wire is (method_name, args, kwargs, output_rank), enqueued once by the executor at multiproc_executor.py:L420, and the per-step call is just collective_rpc("execute_model", args=(scheduler_output,), ...). Only one rank replies, chosen as the first TP rank of the last PP stage (multiproc_executor.py:L541-L555) — the others compute and stay silent, because their outputs are identical by construction.

The Ray backends differ only in how processes come into existence. RayExecutorV2 subclasses MultiprocExecutor — workers are Ray actors, control plane is the same message queue (vllm/v1/executor/ray_executor_v2.py:L219-L231). The legacy RayDistributedExecutor instead compiles a Ray DAG whose input node is the SchedulerOutput and whose edges are PP stages (vllm/v1/executor/ray_executor.py:L574-L590). Either way the plan is still broadcast; only the transport changed.

SGLang: N schedulers, no executor

SGLang has no executor object at all. The launcher loops over ranks and starts one full Scheduler process for each:

            for pp_rank in pp_rank_range:
                for tp_rank in tp_rank_range:
                    reader, writer = mp.Pipe(duplex=False)
                    gpu_id = (
                        server_args.base_gpu_id
                        + ((pp_rank % pp_size_per_node) * tp_size_per_node)
                        + (tp_rank % tp_size_per_node) * server_args.gpu_id_step
                    )
                    attn_cp_rank, moe_dp_rank, moe_ep_rank = _compute_parallelism_ranks(
                        server_args, tp_rank
                    )

                    with maybe_reindex_device_id(gpu_id) as gpu_id:
                        proc = mp.Process(
                            target=run_scheduler_process_func,
                            args=(
                                server_args,
                                port_args,
                                gpu_id,
                                tp_rank,
# ...
                                writer,
                            ),
                        )

Each of those processes builds a Scheduler, sends its init info back up the pipe, and enters an event loop that receives, schedules, and executes — all of it, on every rank (python/sglang/srt/managers/scheduler.py:L5159-L5176). The loop is the same one a single-GPU SGLang server runs:

    def event_loop_normal(self):
        """A normal scheduler loop."""
        while True:
            if self.gracefully_exit:
                break

            # Receive requests
            recv_reqs = self.request_receiver.recv_requests()
            self.process_input_requests(recv_reqs)
            if self._engine_paused:
                continue

            # Get the next batch to run
            plan = self.get_next_batch_to_run(
                running_batch=self.running_batch, last_batch=self.last_batch
            )
            self.running_batch = plan.running_batch
            batch = plan.batch_to_run
            self.cur_batch_for_debug = batch

            # Launch the current batch
            if batch:
                result = self.run_batch(batch)
                self.process_batch_result(batch, result)

The only cross-rank step is recv_requests. Rank 0 owns the ZMQ socket to the tokenizer; the other ranks pass None and receive the list by gloo broadcast:

        elif self.ps.tp_size != 1:
            recv_reqs = broadcast_pyobj(
                recv_reqs,
                self.tp_group.rank,
                self.tp_cpu_group,
                src=self.tp_group.ranks[0],
            )
        return recv_reqs

The IPC asymmetry is visible in channel construction: only the rank-zero scheduler creates sockets at all.

        if is_rank_zero:
            recv_from_tokenizer = get_zmq_socket(
                context, zmq.PULL, port_args.scheduler_input_ipc_name, False
            )
            if enable_scripted_runtime:

The consequence matters: the gloo broadcast is a barrier too. Every rank calls recv_requests once per loop iteration, and when TP>1 that call blocks in a collective. SGLang's loop is therefore implicitly synchronised even when idle — ranks cannot drift apart in loop count, only in the decisions they make from identical inputs.

One decode step at TP=4

vLLM, function by function. EngineCore.step() (vllm/v1/engine/core.py:L583-L595) calls self.scheduler.schedule(), then hands the result to MultiprocExecutor.execute_model, which is a thin wrapper over collective_rpc (multiproc_executor.py:L340-L351). collective_rpc does exactly one enqueue:

        if isinstance(method, str):
            send_method = method
        else:
            send_method = cloudpickle.dumps(method, protocol=pickle.HIGHEST_PROTOCOL)
        self.rpc_broadcast_mq.enqueue((send_method, args, kwargs, output_rank))

        response_mqs: Sequence[MessageQueue] = self.response_mqs
        if output_rank is not None:
            response_mqs = (response_mqs[output_rank],)

All four workers wake in worker_busy_loop, resolve execute_model on their Worker, and enter the forward pass together. Only rank 0's response queue is read.

SGLang, same step: all four Scheduler processes are already inside event_loop_normal. Each calls recv_requests → broadcast_pyobj (a barrier), then each independently calls get_next_batch_to_run and run_batch. There is no centralized plan on the wire. The ranks have already participated in the input broadcast before reaching the first model all-reduce.

One cold start

Startup is where multi-rank systems actually break, because it is a long sequence of steps of which several are collective. vLLM's order is fixed by WorkerProc.__init__ and EngineCore.__init__: construct worker → init_device() → load_model() → message queues → READY. Then, from the controller: determine_available_memory() → get_kv_cache_configs() → initialize_from_config() → compile_or_warm_up_model().

init_device is the first collective. Note the ordering comment — the process group is created before the memory snapshot, so NCCL's buffers are counted as used rather than as free KV space:

            # Initialize the distributed environment BEFORE taking
            # memory snapshot
            # This ensures NCCL buffers are allocated before we measure
            # available memory
            init_worker_distributed_environment(
                self.vllm_config,
                self.rank,
                self.distributed_init_method,
                self.local_rank,
                current_platform.dist_backend,
            )

That call resolves to init_distributed_environment then ensure_model_parallel_initialized (vllm/v1/worker/gpu_worker.py:L1423-L1437), which is where a TP/PP mismatch is caught with an assertion rather than a hang (vllm/distributed/parallel_state.py:L2023-L2027).

Memory profiling is per-rank and can legitimately differ between ranks — a stray process on one GPU, an uneven PP split. Since vLLM's controller holds one block table for all ranks, the configs must be reconciled, and the reconciliation is a plain Python min:

    # Change the num_blocks of each rank to the smallest among all ranks.
    # We also need to shrink the tensor size proportionally to avoid
    # allocating unused memory.
    min_num_blocks = min(
        kv_cache_config.num_blocks for kv_cache_config in kv_cache_configs
    )
    for kv_cache_config in kv_cache_configs:
        num_blocks_old = kv_cache_config.num_blocks
        kv_cache_config.num_blocks = min_num_blocks

        # Shrink tensor size proportionally
        for tensor in kv_cache_config.kv_cache_tensors:
            assert tensor.size % num_blocks_old == 0
            tensor.size = tensor.size // num_blocks_old * min_num_blocks

SGLang has no controller that sees all ranks, so the same reconciliation is a collective. get_available_gpu_memory(..., distributed=True) min-reduces free memory across the world group before anyone sizes a pool:

    if distributed:
        tensor = torch.tensor(free_gpu_memory, dtype=torch.float32)
        torch.distributed.all_reduce(
            tensor, op=torch.distributed.ReduceOp.MIN, group=cpu_group
        )
        free_gpu_memory = tensor.item()

    return free_gpu_memory / (1 << 30)

This is the same design difference showing up in a different place: Model A reconciles in the controller, Model B reconciles with a collective. Note also that vLLM's ExecutorWithExternalLauncher — the one Model B path in vLLM — does it exactly SGLang's way (vllm/v1/executor/uniproc_executor.py:L199-L207), which is a good sanity check that the two models really are the two options.

Neither engine elects a leader. Rank 0 is declared, by you, on the command line, and everything else is derived arithmetic. Both engines take the same three inputs: how many nodes, which node this is, and where the rendezvous lives.

SGLang calls them --nnodes, --node-rank, and --dist-init-addr (python/sglang/srt/server_args.py:L1015-L1024). The address is turned into a torch tcp:// init method:

def _resolve_dist_init_method(*, server_args: ServerArgs, dist_port: int) -> str:
    # Allow external orchestrators (e.g. trainpi) to override the distributed
    # init method.  When set to "env://", torch uses MASTER_ADDR/MASTER_PORT
    # env-vars and an externally-created TCPStore, completely avoiding port
    # conflicts with intra-host collocation.
    dist_init_method_override = envs.SGLANG_DISTRIBUTED_INIT_METHOD_OVERRIDE.get()
    if dist_init_method_override:
        dist_init_method = dist_init_method_override
    elif server_args.dist_init_addr:
        na = NetworkAddress.parse(server_args.dist_init_addr)
        dist_init_method = na.to_tcp()
    else:
        dist_init_method = NetworkAddress(
            server_args.host or "127.0.0.1", dist_port
        ).to_tcp()
    return dist_init_method

Which ranks live on which node is pure arithmetic from node_rank (python/sglang/srt/entrypoints/engine.py:L1824-L1856), and the divisibility precondition is asserted up front:

        if self.ep_join_mode != "scale":
            assert (
                self.tp_size * self.pp_size
            ) % self.nnodes == 0, "tp_size must be divisible by number of nodes"

vLLM's flags are --nnodes/-n, --node-rank/-r, --master-addr, --master-port (vllm/engine/arg_utils.py:L1029-L1032), and — a finding worth stating plainly — the multiprocessing backend now does multi-node TP and PP without Ray. Non-zero nodes run in headless mode, constructing a MultiprocExecutor with no scheduler above it:

    if parallel_config.node_rank_within_dp > 0:
        from vllm.version import __version__ as VLLM_VERSION

        # Run headless workers (for multi-node PP/TP).
        host = parallel_config.master_addr
        head_node_address = f"{host}:{parallel_config.master_port}"
        logger.info(
            "Launching vLLM (v%s) headless multiproc executor, "
            "with head node address %s for torch.distributed process group.",
            VLLM_VERSION,
            head_node_address,
        )

        executor = MultiprocExecutor(vllm_config, monitor_workers=False)
        executor.start_worker_monitor(inline=True)
        return

The plan still has to reach those remote ranks. It does, because the message queue on a follower node is built on top of the torch.distributed world group instead of shared memory (vllm/v1/executor/multiproc_executor.py:L620-L629) — so the shm ring buffer for local workers and a distributed broadcaster for remote ones present the same dequeue API to worker_busy_loop.

vLLM validates the launch arithmetic with error strings you will actually see:

            if world_size % self.nnodes != 0:
                raise ValueError(
                    "Invalid data-parallel launch options: "
                    f"`--nnodes {self.nnodes}` must evenly divide the total "
                    f"world size ({world_size}). Adjust `--nnodes`, "
                    "`--data-parallel-size`, `--pipeline-parallel-size`, or "
                    "`--tensor-parallel-size`."
                )
            if not 0 <= self.node_rank < self.nnodes:
                raise ValueError(
                    "Invalid data-parallel launch options: `--node-rank` must "
                    f"be between 0 and {self.nnodes - 1}; got "
                    f"`--node-rank {self.node_rank}`. Set it to this node's "
                    "zero-based index."
                )

And the classic single-node mistake — asking for more world size than you have GPUs — has its own message at vllm/config/parallel.py:L953-L958, which formats as World size (N) is larger than the number of available GPUs (M) in this node and then names the two remedies, --distributed-executor-backend ray or --nnodes.

What must match across nodes. Model path and revision, every parallelism size, dtype and quantisation, block/page size, CUDA-graph capture sizes, and the engine version. There is no cross-node version handshake in either engine at these SHAs: skew surfaces as a decode error on the plan broadcast in vLLM, or a shape mismatch inside NCCL in SGLang. Identical images are a hard requirement.

A worker dies

The hard part of partial failure is not detection — it is that the survivors are stuck. When rank 3 segfaults, ranks 0, 1, and 2 are blocked inside an NCCL collective that will never complete. You cannot ask them to clean up, because they are not running Python. The only reliable move is to kill everything.

vLLM does exactly that. A monitor thread waits on the worker sentinels; the first death takes the executor down and fires a callback into the engine:

        def monitor_workers():
            sentinels = [h.proc.sentinel for h in workers]
            died = multiprocessing.connection.wait(sentinels)
            _self = self_ref()
            if not _self or getattr(_self, "shutting_down", False):
                logger.debug("MultiprocWorkerMonitor: shutdown already initiated")
                return
            _self.is_failed = True
            proc = next(h.proc for h in workers if h.proc.sentinel == died[0])
            logger.error(
                "Worker proc %s died unexpectedly (exit code: %s), "
                "shutting down executor.",
                proc.name,
                proc.exitcode,
            )
            _self.shutdown()

SGLang's equivalent is the rank's own exception handler signalling the parent, plus an opt-in process-group kill so the surviving ranks do not fill the log with NCCL teardown noise first:

    except Exception:
        traceback = get_exception_traceback()
        logger.error(f"Scheduler hit an exception: {traceback}")
        parent_process.send_signal(signal.SIGQUIT)
        # Opt-in: SIGKILL the pgroup so sibling ranks don't spew thousands
        # of NCCL/TCPStore tracebacks before they finally die.
        if envs.SGLANG_KILLPG_ON_SCHEDULER_EXCEPTION.get():
            try:
                os.killpg(os.getpgrp(), signal.SIGKILL)
            except Exception:
                pass

SGLang additionally runs a SubprocessWatchdog because a C++-level abort — a NCCL timeout calling std::terminate() — never runs a Python handler, and would otherwise leave a zombie service (python/sglang/srt/utils/watchdog.py:L166-L175, which cites sgl-project/sglang#18421).

vLLM's fault-tolerance path

vllm/v1/fault_tolerance/ is small — 222 lines across three files — and its scope is narrower than the directory name suggests. It wraps the engine's busy loop so a raised exception parks the engine instead of killing it:

def fault_tolerant_wrapper(busy_loop_func: Callable):
    """Wrap the busy loop to catch faults and delegate recovery."""

    def run_with_fault_tolerance(self: "EngineCoreProc"):
        while True:
            try:
                busy_loop_func(self)
            except SystemExit:
                raise
            except Exception as exc:
                if not self.enable_fault_tolerance:
                    raise
                self.ft_sentinel.on_fault(exc)
                recovered = self.ft_sentinel.resumed.wait(
                    timeout=self.ft_sentinel.engine_recovery_timeout_sec
                )
                if recovered:
                    continue
                logger.error(
                    "[FT] No recovery within %ds timeout.",
                    self.ft_sentinel.engine_recovery_timeout_sec,
                )
                raise

The crucial detail is in on_fault: it asks the executor whether it failed and marks the engine DEAD if so (vllm/v1/fault_tolerance/engine_core_sentinel.py:L92-L98). If a worker process died, recovery is not attempted. In-flight requests are aborted, and the recovery that is implemented — retry — rebuilds the stateless DP process group and RPCs every worker to clear its runner state (engine_core_sentinel.py:L139-L171; vllm/v1/worker/sentinel/gpu_worker_sentinel.py:L52-L69). That is a story about a DP replica hiccuping, not about surviving a lost TP rank; losing a TP rank still means restarting the engine. Monitoring all of this is §9.5's job.

Divergence, and how to catch it

Model B's hazard — ranks quietly disagreeing — is hard to debug because the symptom appears one barrier later, in unrelated code. SGLang ships an opt-in checker: ranks hash the sequence of consensus-marked events they observed and reduce that hash with both MIN and MAX. If the two differ, the ranks did not agree.

    value_bytes = hasher.digest()
    min_value = torch.tensor(list(hasher.digest()), dtype=torch.uint8)
    max_value = min_value.clone()
    for group in _sync_groups:
        dist.all_reduce(min_value, op=dist.ReduceOp.MIN, group=group)
        dist.all_reduce(max_value, op=dist.ReduceOp.MAX, group=group)
    if not torch.equal(min_value, max_value):
        # When divergence, all rank should output the following log.
        logger.critical(
            f"Found rank divergence for {len(events)} events(s)! local hash: {value_bytes.hex()}, events = {events}"
        )
        for handler in logger.handlers:
            handler.flush()

        # os._exit instead of sys.exit: this runs in a background thread, where
        # SystemExit would only kill the thread, not the process. os._exit tears
        # down the whole scheduler process so a TP/PP mismatch can never
        # silently keep serving.
        os._exit(1)

Turn it on with SGLANG_ENABLE_RANK_CONSENSUS_CHECKER=1 (python/sglang/srt/environ.py:L332). The functions it guards are the ones most likely to diverge: radix-cache prefix matching and HiCache prefetch progress, both decorated @rank_consensus in python/sglang/srt/mem_cache/unified_radix_cache.py:L495-L498 and python/sglang/srt/mem_cache/unified_radix_cache.py:L1753. That is a useful hint about where divergence actually comes from in practice — not the model, but the host-side cache bookkeeping.

Non-NVIDIA backends

Both engines abstract the accelerator behind a platform object that supplies the distributed backend string: nccl for CUDA and ROCm, xccl for XPU, gloo for CPU (vllm/platforms/cuda.py:L214, vllm/platforms/xpu.py:L111, vllm/platforms/cpu.py:L48). SGLang resolves it through get_default_distributed_backend (python/sglang/srt/distributed/parallel_state.py:L2157-L2164) and keeps per-vendor code under python/sglang/srt/hardware_backend/ — npu (Ascend), xpu, musa, cpu, mlx — with Ascend P/D transfer in python/sglang/srt/disaggregation/ascend/. Nothing above changes on these backends: same process tree, same broadcast, same barriers; only the collective library and the graph-capture mechanism swap out.

1. See the process tree. Start each engine at TP=2 and look at what actually exists. vLLM names its worker processes, and the names encode the ranks (multiproc_executor.py:L1075-L1084):

# Terminal A: run one engine in the foreground.
vllm serve meta-llama/Llama-3.1-8B-Instruct --tensor-parallel-size 2

# Terminal B: wait for successful service readiness, then inspect.
ps -eo pid,ppid,comm,args | grep -E "VLLM|EngineCore|Worker" | grep -v grep

# Stop vLLM with Ctrl-C, confirm every worker exits and GPU memory is released.
# Only then run SGLang in Terminal A; inspect from Terminal B after readiness.
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --tp 2
# Terminal B:
ps -eo pid,ppid,comm,args | grep sglang

vLLM shows one EngineCore plus two Worker_TP* processes; SGLang shows two sglang::scheduler processes plus a detokenizer. Count them — the count is the architecture.

2. Make the divergence hazard visible. Run SGLang at TP=2 with the consensus checker on and note that it costs two extra all-reduces per checked event:

SGLANG_ENABLE_RANK_CONSENSUS_CHECKER=1 python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct --tp 2 --enable-metrics

3. Shorten the timeout so hangs are loud. The 300 s defaults are tuned for slow cold starts, not for production hang detection. In staging, drop them and see the failure mode immediately:

VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30 vllm serve ... --tensor-parallel-size 2
python -m sglang.launch_server ... --tp 2 --watchdog-timeout 30 --soft-watchdog-timeout 10

The soft watchdog is the more useful: it dumps py-spy stacks for every scheduler process without killing anything, which is how you find out which rank is somewhere else.

4. Break it on purpose, only on an isolated disposable deployment. Set a bounded external timeout, retain logs, and terminate all ranks after each test. Never run this against shared or production workers. On two nodes, set --node-rank 1 on both and read what you get. Then point node 1's --dist-init-addr somewhere unreachable and time how long before anything is logged.

• vllm-project/vllm#11400 — the motivation for ExecutorWithExternalLauncher, cited directly in the class docstring. The best short statement of the torchrun-SPMD case for inference.
• sgl-project/sglang#18421 — cited in SubprocessWatchdog: a NCCL timeout aborts in C++, no Python handler runs, and the server becomes a zombie. The canonical multi-rank failure report.
• vLLM V1: A Major Upgrade to vLLM's Core Architecture — the design note that introduced the EngineCore/Executor/Worker split this chapter traces.
• Shoeybi et al., Megatron-LM — the source of the "two all-reduces per layer" structure that sets the barrier count.
• In-repo, worth reading end to end: vllm/v1/executor/multiproc_executor.py (1,122 lines and the entire vLLM control plane) and python/sglang/srt/managers/scheduler_components/request_receiver.py (the only place SGLang ranks exchange scheduling input).
• Next: §11.2 walks vLLM's AsyncLLM → EngineCore → Executor → Worker layering in repo-tour depth, and §12.1 does the same for SGLang's process tree. Operational monitoring of everything that goes wrong here is §9.5.