AsyncLLM → EngineCore → Executor → Worker

This is the entire error message a vLLM user gets when the thing that actually broke was a CUDA illegal memory access inside an attention kernel on TP rank 3:

class EngineDeadError(VLLMServerError):
    """Raised when the EngineCore dies. Unrecoverable."""

    def __init__(self, *args, suppress_context: bool = False, **kwargs):
        ENGINE_DEAD_MESSAGE = "EngineCore encountered an issue. See stack trace (above) for the root cause."  # noqa: E501

        super().__init__(ENGINE_DEAD_MESSAGE, *args, **kwargs)
        # Make stack trace clearer when using with LLMEngine by
        # silencing irrelevant ZMQError.
        self.__suppress_context__ = suppress_context

"See stack trace (above)" is doing a lot of work there. The stack trace is not above — it is in a different process, under a different logger prefix, and at TP > 1 in a third process below that. The frontend genuinely does not know what happened; all it knows is that a socket delivered a sentinel byte-string.

That is not sloppiness; it is the unavoidable price of a design decision, and the decision is worth paying for. But you cannot debug vLLM, extend it, or reason about its tail latency until you can say which object lives in which process, what state each owns exclusively, and what may cross between them.

Five layers, named top to bottom: AsyncLLM, EngineCoreClient, EngineCore, Executor, Worker. Two of those names are boundaries pretending to be objects.

The stack has exactly two interesting cuts in it, and they are cut for different reasons.

The frontend/engine process boundary separates Python-heavy work. Independent interpreters can reduce GIL contention, but GPU and native work may already overlap inside one process. The sum/max comparison below is a serial-versus-perfect-pipeline model, not an inevitable consequence of process count.

The second cut — between Executor and Worker — gets one Python interpreter per GPU, because a CUDA context belongs to a process. It exists only when there is more than one rank: at world_size == 1 the backend resolves to uni (vllm/config/parallel.py:L980-L981) and UniProcExecutor builds the worker inline in the EngineCore process (vllm/v1/executor/uniproc_executor.py:L52-L73), so a single-GPU server is two processes wide, not three. That cut is §5.5's subject; this chapter treats it as given.

Everything else is ordinary in-process composition: AsyncLLM holds an EngineCoreClient, EngineCore holds an Executor, an Executor holds worker handles. The stack looks deep, but only two of the four joints are real boundaries.

Let $T_{\text{step}}$ denote the measured GPU/engine service term for the configuration being modeled. The illustrative 15 GB streamed-weight floor is 4.48 ms on the stated H100 bandwidth, not a universal actual step period. KV traffic, achieved bandwidth, launch work and batch shape can change it even below the compute ridge. Let $c$ be the host cost per emitted token frame and $B$ the scheduled batch; streaming intervals and speculative multi-token outputs require corresponding accounting.

Per step the frontend does $B$ units of $c$: one forward pass yields one token per sequence, and every sequence has its own socket. Compare the two placements:

These are explicit execution models: the first assumes no overlap and the second assumes independent resources and perfect pipelining, before IPC overhead. The GIL serializes Python bytecode in a conventional interpreter, but GPU execution and native operations can run while it is released. A single process is therefore not necessarily serial, and a process split is not automatically faster.

At $c=T_{\text{step}}/B$ the ideal serial and perfect-overlap models differ by two. This is their largest relative benefit, not a break-even threshold with IPC: that threshold depends on transfer overhead and achievable overlap. For very large $Bc$, both periods are dominated by frontend work.

The bill for the split

1. Serialisation and wire bytes — small, and shrinking per token as batch grows. EngineCoreOutputs carries the whole batch's tokens for one step, so the boundary is crossed once per step, not once per token — the single most important structural fact about the split. Each EngineCoreOutput is a msgspec struct with array_like=True, omit_defaults=True, so a plain decode output is a two-element msgpack array: a request_id and a one-element token list. The id is the client-facing one with eight random hex characters appended (vllm/v1/engine/input_processor.py:L262-L279), so an OpenAI chat request carries chatcmpl- + 16 hex + - + 8 hex = 34 characters, a 2-byte str8 header plus 34; the token list is a 1-byte header plus 3 for a mid-range Llama-3 id. About 41 bytes including the outer array header. Multiply by $B$, add ~50 bytes for the outer struct's engine_index, timestamp and SchedulerStats:

At $B=128$ the boundary costs one encode and one decode per 128 tokens — 0.016 operations per token — moving 1.2 MB/s over a Unix domain socket. The same token leaves the frontend as an OpenAI SSE chunk on the order of a few hundred bytes of JSON (uncited: count it on your own deployment with curl -N), so the IPC payload is several times smaller than the HTTP payload the frontend produces from it anyway. Batching amortises the IPC hop, which is why vLLM can afford to put one there.

The input direction is per-request and dominated by prompt_token_ids: a 2,048-token Llama-3 prompt at roughly 3 msgpack bytes per id is about 6.2 KB, sent once. Generating 256 tokens occupies 256 × 4.48 ms = 1.15 s, so the amortised input rate is 5.4 KB/s per in-flight request — derived, and again negligible.

2. Admission and abort latency. IPC adds transport and queueing delay. A regularly progressing loop may drain the queue at a step boundary, but 4.48 ms is an idealized GPU lower bound, not an upper bound on waiting. Prefill, CPU scheduling, outstanding work or failure can extend the delay. Abort handling has its own drain points; trace arrival and acknowledgement instead of inferring a bound from bandwidth.

3. A materially worse debugging story. This is the real cost, and the opening error message is what it looks like. No stack trace spans the boundary; a py-spy dump on the API server shows an idle event loop no matter what the engine is doing.

The in-tree comment where the engine's IO threads are created states the benefit side of the trade in vLLM's own words:

            # Background Threads and Queues for IO. These enable us to
            # overlap ZMQ socket IO with GPU since they release the GIL,
            # and to overlap some serialization/deserialization with the
            # model forward pass.
            # Threads handle Socket <-> Queues and core_busy_loop uses Queue.
            ready_event = threading.Event()
            input_thread = threading.Thread(
                target=self.process_input_sockets,
                args=(
                    addresses.inputs,
                    addresses.coordinator_input,
                    identity,
                    ready_event,
                ),
                daemon=True,
            )
            input_thread.start()

            self.output_thread = threading.Thread(
                target=self.process_output_sockets,
                args=(
                    addresses.outputs,
                    addresses.coordinator_output,
                    self.engine_index,
                ),
                daemon=True,

The busy loop never touches a socket: it puts a tuple on a queue.Queue and moves on. The msgpack encode happens on the output thread, which can run it during the next forward pass because zmq.send and the C-level encoder both release the GIL. The GIL argument applied recursively — the same reasoning that split the processes splits the threads.

Two struct types cross the boundary in steady state, and a one-byte tag says which. The tag is an enum whose values are the wire bytes, so there is no framing layer:

class EngineCoreRequestType(enum.Enum):
    """
    Request types defined as hex byte strings, so it can be sent over sockets
    without separate encoding step.
    """

    ADD = b"\x00"
    ABORT = b"\x01"
    START_DP_WAVE = b"\x02"
    UTILITY = b"\x03"
    # Sentinel used within EngineCoreProc.
    EXECUTOR_FAILED = b"\x04"
    # Sentinel to wake up input_queue.get() during shutdown.
    WAKEUP = b"\x05"

UTILITY is the escape hatch: every out-of-band control call — get_supported_tasks, profile, reset_prefix_cache, sleep, add_lora, pause_scheduler — is a (client_index, call_id, method_name, args) tuple dispatched by getattr on the EngineCore instance (vllm/v1/engine/core.py:L1534-L1548), answered by a UtilityOutput keyed on call_id. The whole protocol is two hot-path structs plus a generic RPC.

Both hot-path structs carry three flags, all performance decisions:

class EngineCoreRequest(
    msgspec.Struct,
    array_like=True,  # type: ignore[call-arg]
    omit_defaults=True,  # type: ignore[call-arg]
    gc=False,
):  # type: ignore[call-arg]
    request_id: str
    prompt_token_ids: list[int] | None
    mm_features: list[MultiModalFeatureSpec] | None
    sampling_params: SamplingParams | None
    pooling_params: PoolingParams | None
    arrival_time: float
    lora_request: LoRARequest | None
    cache_salt: str | None
    data_parallel_rank: int | None
    prompt_embeds: torch.Tensor | None = None

array_like=True drops field names, making the struct a positional msgpack array — which is why every newly added field carries a comment saying "appended last so array_like positional serialization stays backward compatible." Field order in these two classes is wire-protocol ABI. omit_defaults=True trims trailing defaults, turning a decode output into 41 bytes instead of the struct's two required fields plus fifteen defaulted ones. gc=False keeps these objects out of CPython's cyclic collector, which matters when you allocate $B$ of them every 4.48 ms.

class EngineCoreOutputs(
    msgspec.Struct,
    array_like=True,  # type: ignore[call-arg]
    omit_defaults=True,  # type: ignore[call-arg]
    gc=False,
):  # type: ignore[call-arg]
    # NOTE(Nick): We could consider ways to make this more compact,
    # e.g. columnwise layout

    engine_index: int = 0

    # [num_reqs]
    outputs: list[EngineCoreOutput] = []
    scheduler_stats: SchedulerStats | None = None
    timestamp: float = 0.0

    utility_output: UtilityOutput | None = None
    finished_requests: set[str] | None = None

    # In DP case, used to signal that the current wave of requests
    # has finished and the engines are paused.
    wave_complete: int | None = None
    # In DP case, used to signal that a request was received for an
    # "old" wave, so the next wave needs to be started in other engines.
    start_wave: int | None = None

    def __post_init__(self):
        if self.timestamp == 0.0:
            self.timestamp = time.monotonic()

The engine stamps time.monotonic(). The quoted historical comment is more restrictive than modern CPython's same-host clock contract: separate processes on the same compatible host and clock namespace can compare readings. Different hosts, time namespaces or incompatible runtimes require clock identification, synchronization and an uncertainty budget. Neither monotonic value is a Unix timestamp. finished_requests reconciles frontend and engine lifetimes.

Zero-copy, and where it stops

Tensors and ndarrays skip msgpack's normal path. MsgpackEncoder stashes backing buffers, encodes an index into that stash in place of the data, and returns the buffers as extra ZMQ frames:

    def _encode_tensor(
        self, obj: torch.Tensor
    ) -> tuple[str, tuple[int, ...], int | dict | memoryview]:
        oob_consumer = self.oob_tensor_consumer
        # view the tensor as a contiguous 1D array of bytes
        if obj.nbytes < self.size_threshold and obj.is_cpu:
            # Smaller tensors are encoded inline, just like ndarrays.
            data = msgpack.Ext(CUSTOM_TYPE_RAW_VIEW, tensor_data(obj))
        elif oob_consumer is not None and (data := oob_consumer(obj)) is not None:
            assert isinstance(data, dict)
        else:
            # Otherwise encode index of backing buffer to avoid copy.
            assert self.aux_buffers is not None
            data = len(self.aux_buffers)
            self.aux_buffers.append(tensor_data(obj))
        dtype = str(obj.dtype).removeprefix("torch.")
        return dtype, obj.shape, data

size_threshold defaults to VLLM_MSGPACK_ZERO_COPY_THRESHOLD = 256 bytes (vllm/envs.py:L224). Below it, copy inline — a separate frame is not worth a syscall. Above it, hand ZMQ a pointer, and then keep that buffer alive until ZMQ is done with it. Hence the message-tracker bookkeeping in the output thread:

                # Reclaim buffers that zmq is finished with.
                while pending and pending[-1][0].done:
                    reclaimed = pending.pop()[1]
                    if len(reuse_buffers) < max_reuse_bufs:
                        reuse_buffers.append(reclaimed)

                buffer = reuse_buffers.pop() if reuse_buffers else bytearray()
                buffers = encoder.encode_into(outputs, buffer)
                tracker = self._send_msg_tracking_payload(
                    sockets[client_index], buffers
                )
                if not tracker.done:
                    pending.appendleft((tracker, buffer))
                elif len(reuse_buffers) < max_reuse_bufs:
                    # Limit the number of buffers to reuse.
                    reuse_buffers.append(buffer)

The payload buffer is recycled, not reallocated: encode_into writes into a bytearray from a free list of size len(sockets) + 1. The 223 allocations per second it saves are trivial; what matters is 223 fewer objects per second entering the GC's young generation in the process whose pause times set $T_{\text{step}}$.

AsyncLLM — the frontend

Process: API server. Concurrency: asyncio, single-threaded. It constructs four things and owns three of them:

        self.renderer = renderer = renderer_from_config(self.vllm_config)

        # Convert EngineInput --> EngineCoreRequest.
        self.input_processor = InputProcessor(self.vllm_config, renderer)

        # Converts EngineCoreOutputs --> RequestOutput.
        self.output_processor = OutputProcessor(
            renderer.tokenizer,
            log_stats=self.log_stats,
            stream_interval=self.vllm_config.scheduler_config.stream_interval,
            tracing_enabled=tracing_endpoint is not None,
        )

        # EngineCore (starts the engine in background process).
        self.engine_core = EngineCoreClient.make_async_mp_client(
            vllm_config=vllm_config,
            executor_class=executor_class,
            log_stats=self.log_stats,
            client_addresses=client_addresses,
            client_count=client_count,
            client_index=client_index,
        )

Read those four lines as a boundary declaration. InputProcessor is the only producer of an EngineCoreRequest; OutputProcessor is the only consumer of an EngineCoreOutputs. The prompt tokeniser lives here and only here (the engine process loads its own for structured output, vllm/v1/structured_output/__init__.py:L71-L80) — which is why prompt_token_ids crosses the wire as a list of ints, never a string.

The exclusively-owned state is the per-request mailbox: one RequestOutputCollector per add_request (vllm/v1/engine/output_processor.py:L48-L55), whose docstring names the invariant that makes streaming survive backpressure — "RequestOutputs are merged if the producer gets ahead of the consumer." A collector is one slot, not a queue; put either fills it or merges into it. That is vLLM's coalescing answer to the problem §9.3 opened with.

Demultiplexing is one background task. output_handler() (vllm/v1/engine/async_llm.py:L686-L747) is the only consumer of the engine's output stream, fanning one batched EngineCoreOutputs out to $B$ collectors in chunks so it does not monopolise the event loop:

        async def output_handler():
            try:
                while True:
                    # 1) Pull EngineCoreOutputs from the EngineCore.
                    outputs = await engine_core.get_output_async()
                    num_outputs = len(outputs.outputs)

                    iteration_stats = (
                        IterationStats() if (log_stats and num_outputs) else None
                    )

                    # Split outputs into chunks of at most
                    # VLLM_V1_OUTPUT_PROC_CHUNK_SIZE, so that we don't block the
                    # event loop for too long.
                    engine_core_outputs = outputs.outputs
                    for start in range(0, num_outputs, chunk_size):
                        end = start + chunk_size
                        outputs_slice = engine_core_outputs[start:end]
                        # 2) Process EngineCoreOutputs.
                        processed_outputs = output_processor.process_outputs(
                            outputs_slice, outputs.timestamp, iteration_stats
                        )

The await asyncio.sleep(0) between chunks is the frontend admitting that $Bc$ can exceed $T_{\text{step}}$, and that when it does the SSE writers must still get a turn. Note the threshold: VLLM_V1_OUTPUT_PROC_CHUNK_SIZE defaults to 128 (vllm/envs.py:L169, :L1430-L1431) and the yield is guarded by if end < num_outputs, so at every batch size discussed above — $B \le 128$ — there is exactly one chunk and the yield never fires. It is insurance for the large-batch case, not part of the steady-state path.

EngineCoreClient — the boundary itself

This layer has no behaviour of its own. Its job is to make "call the engine" look identical whether the engine is in this process, in another reached synchronously, or in another reached from an event loop. The class docstring enumerates those three (the factory itself is make_client at vllm/v1/engine/core_client.py:L88-L110):

class EngineCoreClient(ABC):
    """
    EngineCoreClient: subclasses handle different methods for pushing
        and pulling from the EngineCore for asyncio / multiprocessing.

    Subclasses:
    * InprocClient: In process EngineCore (for V0-style LLMEngine use)
    * SyncMPClient: ZMQ + background proc EngineCore (for LLM)
    * AsyncMPClient: ZMQ + background proc EngineCore w/ asyncio (for AsyncLLM)
    """

InprocClient states plainly what the abstraction hides: "get the next batch of outputs" becomes "run one step of the engine, synchronously, right here."

    def __init__(self, *args, **kwargs):
        self.engine_core = EngineCore(*args, **kwargs)

    def get_output(self) -> EngineCoreOutputs:
        outputs, model_executed = self.engine_core.step_fn()
        self.engine_core.post_step(model_executed=model_executed)
        return outputs and outputs.get(0) or EngineCoreOutputs()

    def get_supported_tasks(self) -> tuple[SupportedTask, ...]:
        return self.engine_core.get_supported_tasks()

    def add_request(self, request: EngineCoreRequest) -> None:
        req, request_wave = self.engine_core.preprocess_add_request(request)
        self.engine_core.add_request(req, request_wave)

AsyncMPClient owns the ROUTER input socket, the PULL output socket, an encoder, a decoder, and an asyncio.Queue filled by a dedicated task (vllm/v1/engine/core_client.py:L1041-L1095). The asymmetry matters: the client binds ROUTER and addresses each send to an engine identity, which is what makes data-parallel routing a client-side concern (DPLBAsyncMPClient).

EngineCore — the loop

Owns exclusively: the Scheduler (and through it the KVCacheManager and every block table), the StructuredOutputManager, the pipeline-parallel batch_queue, and the two queue.Queues that face the IO threads. The busy loop is eleven lines:

    @fault_tolerant_wrapper
    def run_busy_loop(self):
        """Core busy loop of the EngineCore."""
        while self._handle_shutdown():
            # 1) Poll the input queue until there is work to do.
            self._process_input_queue()
            # Publish request counts before and after GPU step to ensure freshness.
            self._maybe_publish_request_counts()
            # 2) Step the engine core and return the outputs.
            self._process_engine_step()
            self._maybe_publish_request_counts()

        raise SystemExit

And step() is the whole engine in twenty:

    def step(self) -> tuple[dict[int, EngineCoreOutputs], bool]:
        """Schedule, execute, and make output.

        Returns tuple of outputs and a flag indicating whether the model
        was executed.
        """

        # Check for any requests remaining in the scheduler - unfinished,
        # or finished and not yet removed from the batch.
        if not self.scheduler.has_requests():
            return {}, False
        scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())
        future = self.model_executor.execute_model(scheduler_output, non_block=True)
        grammar_output = self.scheduler.get_grammar_bitmask(scheduler_output)
        with (
            self.capture_iteration_details(scheduler_output) as iteration_details,
            self.log_error_detail(scheduler_output),
        ):
            model_output = future.result()
            if model_output is None:
                model_output = self.model_executor.sample_tokens(grammar_output)

        # Before processing the model output, process any aborts that happened
        # during the model execution.
        self._process_aborts_queue()
        engine_core_outputs = self.scheduler.update_from_output(
            scheduler_output, model_output
        )
        self._attach_iteration_details(engine_core_outputs, iteration_details)

        return engine_core_outputs, scheduler_output.total_num_scheduled_tokens > 0

Two things to notice. execute_model(..., non_block=True) returns a Future, and the grammar bitmask is computed between the launch and the .result(). How much that actually hides depends on the executor: under MultiprocExecutor the call returns as soon as the request is enqueued on the broadcast MQ (vllm/v1/executor/multiproc_executor.py:L310) and the overlap is real, but UniProcExecutor runs the method inline and then calls future.set_result(...) (vllm/v1/executor/uniproc_executor.py:L108-L117) — on one GPU the only genuine deferral is async scheduling's AsyncOutputFuture (:L32-L47). And _process_aborts_queue() sits between the model output and update_from_output, so an abort that arrived during a 4.48 ms forward pass takes effect on the same step. The input thread writes aborts to both queues (vllm/v1/engine/core.py:L1763-L1772) because "aborting in the scheduler is idempotent."

Executor — rank fan-out

Owns exclusively: the set of ranks, the broadcast channel to them, the response channels back. Nothing else. Its contract is one primitive, collective_rpc; every other base-class method is a one-liner over it:

    def execute_model(
        self, scheduler_output: SchedulerOutput, non_block: bool = False
    ) -> ModelRunnerOutput | None | Future[ModelRunnerOutput | None]:
        output = self.collective_rpc(  # type: ignore[call-overload]
            "execute_model", args=(scheduler_output,), non_block=non_block
        )
        return output[0]

    @overload
    def sample_tokens(
        self, grammar_output: GrammarOutput | None, non_block: Literal[False] = False
    ) -> ModelRunnerOutput:
        pass

Its docstring states the boundary rule: "It is recommended to use this API to only pass control messages, and set up data-plane communication to pass data" (vllm/v1/executor/abstract.py:L183-L185). KV tensors never cross here — NCCL and the KV connectors carry them. The executor moves a SchedulerOutput down and a ModelRunnerOutput up, and nothing else.

Worker — the device

Owns exclusively: the torch.device, the CUDA context, the distributed groups, the memory snapshot used for KV sizing, and the model runner. Worker.init_device (vllm/v1/worker/gpu_worker.py:L315-L441) does them in an order that is itself an invariant: resolve the device index, set the device, initialise distributed, then snapshot memory — "This ensures NCCL buffers are allocated before we measure available memory." Get that order wrong and the KV cache is sized against memory NCCL is about to take.

Executor.get_class dispatches on one config string, and is the fastest way to see what topologies vLLM supports at this SHA:

        elif distributed_executor_backend == "ray":
            if envs.VLLM_USE_RAY_V2_EXECUTOR_BACKEND:
                from vllm.v1.executor.ray_executor_v2 import RayExecutorV2

                executor_class = RayExecutorV2
            else:
                from vllm.v1.executor.ray_executor import RayDistributedExecutor

                executor_class = RayDistributedExecutor
        elif distributed_executor_backend == "mp":
            from vllm.v1.executor.multiproc_executor import MultiprocExecutor

            executor_class = MultiprocExecutor
        elif distributed_executor_backend == "uni":
            from vllm.v1.executor.uniproc_executor import UniProcExecutor

            executor_class = UniProcExecutor
        elif distributed_executor_backend == "external_launcher":
            # TODO: make v1 scheduling deterministic
            # to support external launcher
            executor_class = ExecutorWithExternalLauncher

The last one inverts the architecture: instead of one scheduler fanning out to $N$ ranks, you launch $N$ complete engines under torchrun and each schedules for itself. Its docstring says what that requires:

class ExecutorWithExternalLauncher(UniProcExecutor):
    """An executor that uses external launchers to launch engines,
    specially designed for torchrun-compatible launchers, for
    offline inference with tensor parallelism.

    see https://github.com/vllm-project/vllm/issues/11400 for
    the motivation, and examples/features/torchrun/torchrun_example_offline.py
    for the usage example.

    The key idea: although it is tensor-parallel inference, we only
    create one worker per executor, users will launch multiple
    engines with torchrun-compatible launchers, and all these engines
    work together to process the same prompts. When scheduling is
    deterministic, all the engines will generate the same outputs,
    and they don't need to synchronize the states with each other.
    """

    def _init_executor(self) -> None:
        """Initialize the worker and load the model."""
        assert not envs.VLLM_ENABLE_V1_MULTIPROCESSING, (
            "To get deterministic execution, "
            "please set VLLM_ENABLE_V1_MULTIPROCESSING=0"
        )
        super()._init_executor()

"When scheduling is deterministic" is the load-bearing clause, and the # TODO: make v1 scheduling deterministic beside the dispatch at vllm/v1/executor/abstract.py:L79-L80 is vLLM stating in its own source that it currently is not. §5.5 identified that comment as the honest reason vLLM broadcasts a SchedulerOutput rather than letting each rank derive it: replicating a scheduler is only correct if every replica makes bit-identical decisions, and vLLM does not guarantee that. Broadcasting one authoritative decision costs a shared-memory write per step and needs no such guarantee — which is why external_launcher is a documented offline path with an assert, and mp is the default once there is more than one rank. On a single GPU there is no second process at all: vllm/config/parallel.py:L980-L981 sets the backend to uni whenever world_size == 1, and UniProcExecutor constructs the worker inline in the EngineCore process (vllm/v1/executor/uniproc_executor.py:L52-L73).

The consequence, also from §5.5: because mp is the multi-rank default and now supports multi-node TP/PP without Ray (via --nnodes, --node-rank and --headless on non-zero nodes), the Ray executors have narrowed from "required for multi-node" to "useful when Ray already owns your scheduling."

The worker side of mp is smaller than people expect:

    def worker_busy_loop(self):
        """Main busy loop for Multiprocessing Workers"""
        assert self.rpc_broadcast_mq is not None
        while True:
            method, args, kwargs, output_rank = self.rpc_broadcast_mq.dequeue(
                indefinite=True
            )
            try:
                if isinstance(method, str):
                    func = getattr(self.worker, method)
                elif isinstance(method, bytes):
                    func = partial(cloudpickle.loads(method), self.worker)

                output = func(*args, **kwargs)

                if output_rank is None or self.rank == output_rank:
                    self.handle_output(output)

A name, a getattr, a call. The isinstance(method, bytes) branch is the extension point: collective_rpc will cloudpickle an arbitrary callable and run it on every rank (vllm/v1/executor/multiproc_executor.py:L415-L420) — how §11.5's plugin surfaces and every "run this on all workers" debugging trick work.

The order of construction encodes real dependencies: you cannot size the KV cache before you know how much memory the model left, and you cannot build the scheduler before you know how many blocks it may allocate.

Step 5 is the one that surprises people: EngineCore.__init__'s first action builds the executor — spawning every worker and loading every weight — before the scheduler exists at all.

        # Setup Model.
        self.model_executor = executor_class(vllm_config)
        self._pooler_config_logged = False
        if executor_fail_callback is not None:
            self.model_executor.register_failure_callback(executor_fail_callback)

Only then is memory profiled and the scheduler constructed with the resulting block count:

        # Setup KV Caches and update CacheConfig after profiling.
        kv_cache_config = self._initialize_kv_caches(vllm_config)
        self.structured_output_manager = StructuredOutputManager(vllm_config)

        # Setup scheduler.
        Scheduler = vllm_config.scheduler_config.get_scheduler_cls()

        if len(kv_cache_config.kv_cache_groups) == 0:  # noqa: SIM102
            # Encoder models without KV cache don't support
            # chunked prefill. But do SSM models?
            if vllm_config.scheduler_config.enable_chunked_prefill:
                logger.warning("Disabling chunked prefill for model without KVCache")
                vllm_config.scheduler_config.enable_chunked_prefill = False

        scheduler_block_size, hash_block_size = resolve_kv_cache_block_sizes(
            kv_cache_config, vllm_config
        )

        self.scheduler: SchedulerInterface = Scheduler(
            vllm_config=vllm_config,
            kv_cache_config=kv_cache_config,
            structured_output_manager=self.structured_output_manager,
            include_finished_set=include_finished_set,
            log_stats=self.log_stats,
            block_size=scheduler_block_size,
            hash_block_size=hash_block_size,
        )

This dependency is why EngineCoreReadyResponse exists: it "Contains post-initialization config that may differ from the original values (e.g. max_model_len after KV cache auto-fitting)" (vllm/v1/engine/__init__.py:L72-L78). The frontend validates request lengths against max_model_len, but the true value is only knowable after profiling in the engine process, so it has to travel back up the wire before the server can accept traffic.

Abort: two owners, one message

Every request exists twice: as a RequestState in the frontend's OutputProcessor.request_states, and as a Request in the engine's scheduler. Aborting must retire both, and only one can be retired synchronously.

    async def abort(
        self, request_id: str | Iterable[str], internal: bool = False
    ) -> None:
        """Abort RequestId in OutputProcessor and EngineCore."""

        request_ids = (
            (request_id,) if isinstance(request_id, str) else as_list(request_id)
        )
        all_request_ids = self.output_processor.abort_requests(request_ids, internal)
        await self.engine_core.abort_requests_async(all_request_ids)

The frontend goes first and its removal is authoritative: abort_requests pops the RequestState, expands parents into their n>1 children, and pushes a final FinishReason.ABORT output into the collector so the waiting generate() unblocks (vllm/v1/engine/output_processor.py:L508-L529). Only the surviving internal IDs go on the wire; the engine's copy dies asynchronously, and the KV blocks come back then — not before.

The frontend can drop its state before the engine releases KV. Reclamation depends on abort delivery, loop progress and outstanding GPU holders. A measured step period can help estimate this gap in normal operation; the 4.48 ms bandwidth floor cannot bound it. Under mass disconnect, verify eventual idempotent cleanup and queue backpressure.

Death: the sentinel chain

When a worker dies, the failure climbs four layers and crosses two process boundaries. Every link is a different mechanism:

        def monitor_workers():
            sentinels = [h.proc.sentinel for h in workers]
            died = multiprocessing.connection.wait(sentinels)
            _self = self_ref()
            if not _self or getattr(_self, "shutting_down", False):
                logger.debug("MultiprocWorkerMonitor: shutdown already initiated")
                return
            _self.is_failed = True
            proc = next(h.proc for h in workers if h.proc.sentinel == died[0])
            logger.error(
                "Worker proc %s died unexpectedly (exit code: %s), "
                "shutting down executor.",
                proc.name,
                proc.exitcode,
            )
            _self.shutdown()
            callback = _self.failure_callback
            if callback is not None:
                _self.failure_callback = None
                callback()

1. A daemon thread blocks on the workers' process sentinels. One dies; wait() returns.
2. The executor marks itself failed, shuts down the remaining workers, and calls the failure callback EngineCore.__init__ registered — a one-liner at vllm/v1/engine/core.py:L1028-L1030 that puts (EngineCoreRequestType.EXECUTOR_FAILED, b"") into input_queue, the only safe way a background thread can interrupt the busy loop.
3. The loop picks it up and _handle_client_request does the bluntest thing available: raise RuntimeError("Executor failed.") (vllm/v1/engine/core.py:L1548-L1549).
4. run_engine_core's handler logs "EngineCore encountered a fatal error" and calls _send_engine_dead(), which pushes ENGINE_CORE_DEAD onto output_queue and joins the output thread with a 5-second timeout — the process is about to exit and the sentinel must win that race (vllm/v1/engine/core.py:L1618-L1631).
5. The frontend's socket task sees a single frame and translates it:

    def validate_alive(self, frames: Sequence[zmq.Frame]):
        if len(frames) == 1 and (frames[0].buffer == EngineCoreProc.ENGINE_CORE_DEAD):
            self.engine_dead = True
            raise EngineDeadError()

That EngineDeadError lands in outputs_queue, is raised out of get_output_async(), kills the output_handler task, and reaches every in-flight request via OutputProcessor.propagate_error, which puts the exception into each collector (vllm/v1/engine/output_processor.py:L468-L474). Every open stream fails at once, which is correct: the engine's state is gone, so nothing in flight can complete. §9.5 owns the operational response — the process is unrecoverable by design and restart is the answer.

Graceful shutdown: drain or abort

The clean path is driven by a signal handler that sets a state enum rather than raising. _handle_shutdown then picks one of two modes from one config value:

        if self.shutdown_state == EngineShutdownState.REQUESTED:
            shutdown_timeout = self.vllm_config.shutdown_timeout
            mode = "abort" if shutdown_timeout == 0 else "drain"

            logger.info(
                "[shutdown] EngineCore: start mode=%s timeout=%ds",
                mode,
                shutdown_timeout,
            )

            if shutdown_timeout == 0:
                num_requests = self.scheduler.get_num_unfinished_requests()
                if num_requests > 0:
                    logger.info(
                        "[shutdown] EngineCore: aborting in-flight requests count=%d",
                        num_requests,
                    )
                aborted_reqs = self.scheduler.finish_requests(
                    None, RequestStatus.FINISHED_ABORTED
                )
                self._send_abort_outputs(aborted_reqs)

Drain is opt-in: shutdown_timeout defaults to 0 (vllm/config/vllm.py:L451), so the out-of-the-box graceful path is abort. In drain mode the loop stops accepting ADD requests (_reject_add_in_shutdown) and keeps stepping until has_work() goes false. Frontend-side, BackgroundResources.__call__ sets engine_dead = True before closing sockets, so late output reads as death rather than as a hang (vllm/v1/engine/core_client.py:L428-L440).

§11.1 owns the model-runner rewrite. What belongs here is why it is tractable: the new runner is selected inside Worker.init_device, below every boundary in this chapter.

        # Construct the model runner
        if self.use_v2_model_runner:
            from vllm.v1.worker.gpu.model_runner import (
                GPUModelRunner as GPUModelRunnerV2,
            )

            # HACK(woosuk): This is a temporary fix to avoid type errors.
            self.model_runner: GPUModelRunner = GPUModelRunnerV2(  # type: ignore
                self.vllm_config, self.device
            )
        else:
            from vllm.v1.worker.gpu_model_runner import (
                GPUModelRunner as GPUModelRunnerV1,
            )

            self.model_runner = GPUModelRunnerV1(self.vllm_config, self.device)

The upper runner interface and process topology remain recognizable, but V2 is not implementation-local: scheduler branches change resumed-request packaging, full token-ID transfer and bookkeeping, and attention paths also differ. See the cross-directory grep exercise. A shared interface limits coupling; it does not mean all its producers are untouched.

Three objects carry the invariants that matter most. EngineCoreRequest/EngineCoreOutput: field order is ABI. RequestOutputCollector: a one-slot mailbox with merge-on-overflow, so a slow consumer changes chunk shape, never token content — making put drop instead of merge is a correctness bug. Executor.collective_rpc: control messages only, because a tensor on the RPC channel serialises every rank's step behind one Python pickle.

1. See both processes, and prove which is busy.

vllm serve meta-llama/Meta-Llama-3-8B-Instruct --max-model-len 4096 &
# in another shell, once it is up:
ps -eo pid,comm,args | grep -E 'EngineCore|APIServer|VLLM'
py-spy dump --pid $(pgrep -f 'EngineCore')      # the busy loop
# NB: with the default --api-server-count 1 the frontend runs in the main
# `vllm serve` process and sets no process title; use that PID directly.
# With --api-server-count 2+ the workers are titled VLLM::APIServer_0, ...
py-spy dump --pid $(pgrep -f 'VLLM::APIServer')  # the event loop

Under load the engine dump sits in execute_model or update_from_output; the frontend in epoll or process_outputs. If the frontend is ever in epoll while requests are queued, $Bc < T_{\text{step}}$ and the split is doing its job.

2. Collapse the split and watch the arithmetic. VLLM_ENABLE_V1_MULTIPROCESSING=0 forces InprocClient, putting the scheduler and the detokeniser under one GIL:

python -c "
from vllm import LLM, SamplingParams
llm = LLM('meta-llama/Meta-Llama-3-8B-Instruct', max_model_len=2048)
print(type(llm.llm_engine.engine_core).__name__)
"
# with VLLM_ENABLE_V1_MULTIPROCESSING=0 -> InprocClient
# default (=1)                          -> SyncMPClient

The offline LLM path makes the model executor directly reachable in-process (vllm/v1/engine/llm_engine.py:L122-L125), which is why every vLLM debugging recipe starts by setting this to 0. Prediction to test: at batch 1 the two settings should be near-identical, since $Bc$ is one token of host work. The gap opens with batch.

3. Measure the wire, not the model. Compare throughput at VLLM_MSGPACK_ZERO_COPY_THRESHOLD=1 and =1000000 on a workload with logprobs=20, which puts real tensors in EngineCoreOutput.new_logprobs. At 1 they ride as separate zero-copy frames; at 1000000 they are copied inline.

• vllm-project/vllm#11400 — the issue ExecutorWithExternalLauncher's docstring points at, on running vLLM under torchrun-compatible launchers. Read it alongside the determinism TODO.
• vLLM V1 alpha release notes — the project's own account of why the frontend and engine were separated, and what the isolated EngineCore loop was meant to buy.
• vLLM architecture overview — the maintained diagram of the same stack; a cross-check on which layers have moved since this SHA.
• msgspec documentation — specifically the pages on array_like, omit_defaults and gc=False, which explain the three flags on the wire structs better than any comment in the vLLM tree.
• ZeroMQ Guide, chapter 3 — ROUTER/DEALER and PUSH/PULL semantics, including what a PUSH socket does when the puller is slow.
• §5.5 for the executor/worker boundary in depth, §9.3 for one request across both engines, §9.5 for the failure semantics, and §11.3 for what happens inside schedule().