Multimodal serving

Run a vision-language model under vLLM's defaults and the first thing that surprises you is not latency. It is that the engine refuses to schedule.

The V1 scheduler has an encoder budget as well as a token budget. If encoder work cannot fit now, it can defer that work by stopping this prefill chunk before the image. That is scheduling delay, not necessarily permanent rejection of the request:

            if not self.encoder_cache_manager.can_allocate(
                request, i, encoder_compute_budget, num_embeds_to_schedule
            ):
                # The encoder cache is full or the encoder budget is exhausted.
                # NOTE(woosuk): We assume that the encoder input tokens should
                # be processed altogether, as the encoder usually uses
                # bidirectional attention.
                if num_computed_tokens + shift_computed_tokens < start_pos:
                    # We only schedule the decoder tokens just before the
                    # encoder input.
                    num_new_tokens = start_pos - (
                        num_computed_tokens + shift_computed_tokens
                    )
                else:
                    # Because of prefix caching, num_computed_tokens is greater
                    # than start_pos even though its encoder input is not
                    # available. In this case, we can't schedule any token for
                    # the request in this step.
                    num_new_tokens = 0
                break

num_new_tokens = 0 means the request is skipped this step entirely. That path opens up because the encoder cache is a fixed, small pool that does not grow with the KV pool. It is sized in embedding slots, and its starting point is the batched-token budget — set unconditionally in SchedulerConfig.__post_init__, with no CLI flag behind it (both fields are Field(init=False), vllm/config/scheduler.py:L86-L97):

        self.max_num_encoder_input_tokens = self.max_num_batched_tokens
        self.encoder_cache_size = self.max_num_batched_tokens

That value is then floored at one worst-case item — the effective size is max(max_num_batched_tokens, max_tokens_per_mm_item), derived in §4 — so the count of images the cache holds depends on both numbers. At max_num_batched_tokens=8192 with a model whose per-item cap sits at or below that, the cache is 8,192 slots, which is five 1,369-token images (derived: $\lfloor 8192/1369 \rfloor = 5$). Raise the cap and the floor takes over: a Qwen2-VL whose processor allows 16,384 tokens per image gets a 16,384-slot cache, or eleven such images. Either way the number is small, fixed at startup, and counted in images across the whole engine rather than per request — and when every resident entry is still referenced by a running request, the next image's request does not advance.

The second surprise is an error string. When the number of embeddings the encoder produced does not equal the number of placeholder positions the tokenizer emitted, the splice fails loudly:

    try:
        # If is_multimodal is on CPU this avoids a D2H sync
        inputs_embeds[is_multimodal] = mm_embeds_flat.to(dtype=input_dtype)
    except RuntimeError as e:
        num_actual_tokens = len(mm_embeds_flat)
        num_expected_tokens = is_multimodal.sum().item()

        if num_actual_tokens != num_expected_tokens:
            expr = _embedding_count_expression(multimodal_embeddings)

            raise ValueError(
                f"Attempted to assign {expr} = {num_actual_tokens} "
                f"multimodal tokens to {num_expected_tokens} placeholders"

Both symptoms come from the same structural fact: a multimodal request is two models in one request, glued together by an exact-length contract on a token count that neither model owns.

A text request has one compute stage. A multimodal request has three: a CPU stage (decode the bytes, resize, tile, normalise), a GPU encoder stage (a vision transformer, run once per image), and the usual language-model prefill. The encoder is not a smaller version of the language model. It has fixed-size input, no KV cache, no incremental state, bidirectional attention, and it runs once per image rather than once per token. Its output — a dense [N, d_model] tensor — is not KV. It is a block of input embeddings, and it must be spliced into the language model's embedding sequence at exactly the positions the tokenizer reserved with placeholder token ids.

Encoder embeddings must stay alive until the language model consumes them. Bidirectional attention prevents independently finalizing a prefix without its future keys; it does not prohibit exact tiled attention over the full image. The model/API may still schedule an image as one encoder item. Reusing its language-model KV additionally requires the same preceding causal prefix, positions, model/adapter and processing identity, not only the same image at the same offset. Encoder-embedding reuse has a different, less contextual key.

Note where the two roofline regimes sit. The encoder reads its weights once and applies them to thousands of patches; the decode loop reads its weights once and applies them to a handful of tokens. They are on opposite sides of the ridge point, in the same engine, contending for the same SMs.

Define the tiling. Qwen2-VL and Qwen2.5-VL use dynamic resolution: the image is resized so both dimensions are divisible by $f = p \cdot m$ where $p = 14$ is the ViT patch size and $m = 2$ is the spatial merge factor, subject to a pixel-count cap. vLLM computes the resulting token count directly:

        grid_t = max(padded_num_frames // temporal_patch_size, 1)
        grid_h = preprocessed_size.height // patch_size
        grid_w = preprocessed_size.width // patch_size

        num_patches = grid_t * grid_h * grid_w
        num_vision_tokens = num_patches // (merge_size**2)

        return preprocessed_size, num_vision_tokens

The resize itself is smart_resize, which SGLang carries in tree so we can read the exact rounding rule:

    h_bar = max(factor, round_by_factor(height, factor))
    w_bar = max(factor, round_by_factor(width, factor))
    if h_bar * w_bar > max_pixels:
        beta = math.sqrt((height * width) / max_pixels)
        h_bar = floor_by_factor(height / beta, factor)
        w_bar = floor_by_factor(width / beta, factor)
    elif h_bar * w_bar < min_pixels:
        beta = math.sqrt(min_pixels / (height * width))
        h_bar = ceil_by_factor(height * beta, factor)
        w_bar = ceil_by_factor(width * beta, factor)
    return h_bar, w_bar

with IMAGE_FACTOR = 28, MIN_PIXELS = 4 * 28 * 28 and MAX_PIXELS = envs.SGLANG_IMAGE_MAX_PIXELS.get() (python/sglang/srt/multimodal/processors/qwen_vl.py:L47-L50), whose default is 16384 * 28 * 28 = 12,845,056 pixels (python/sglang/srt/environ.py:L1159). Since $28^2 = 784$ is exactly the pixel area of one merged vision token, that cap says plainly: at most 16,384 tokens per image. So:

where $H, W$ are the input dimensions, $H', W'$ the resized ones, and $\lceil\lfloor \cdot \rceil\rfloor$ denotes round-to-nearest. Work it for real screen sizes:

That last row is the practical headline. One un-downscaled 4K screenshot costs 10,549 tokens and 1.29 GiB of KV — more than five copies of a 2,000-token system prompt, and enough to overflow an 8,192-token context on its own. Eight 1024×1024 images in a single conversation come to 10,952 tokens and 1,369 MiB. A "few images" is not a few hundred tokens; it is a long document.

Why the encoder is the wrong shape for a decode engine

Take an encoder of $P_{\text{enc}}$ parameters in bf16 running one image of $N_p$ patches. A forward pass reads each weight once: $2 P_{\text{enc}}$ bytes of HBM traffic, and roughly $2 P_{\text{enc}} N_p$ FLOPs of GEMM work. Arithmetic intensity is therefore

For the 1024×1024 image, $N_p = 5{,}476$. Against the H100's ridge point $I^{*} = 295$ (§0.4), that is 18.6× past the ridge at batch size one. The encoder is compute-bound before you batch anything. Decode, by the same argument, has intensity $\approx B$ and needs a batch of ~295 to reach the ridge (§1.1). Two workloads, opposite regimes, one GPU — which is exactly why the scheduler needs a separate encoder budget rather than folding encoder work into the token budget.

The placeholder ids are never embedded in any meaningful sense — the text embedding table is applied to the whole sequence and then those rows are clobbered. That is why the count must match exactly, and why the placeholder id itself is free to be repurposed, as SGLang does in §5.

The same image sent twice must not be encoded twice. Both engines cache encoder outputs, separately from the KV pool, and they make opposite choices about what a "budget" means.

vLLM: refcounted slots, deferred eviction

EncoderCacheManager tracks three structures: cached (hash → set of referencing request ids), freeable (an OrderedDict of hash → slot count for entries nobody references), and freed (hashes actually evicted, drained by the scheduler to tell the worker what to drop). A lookup both tests and un-frees:

        mm_hash = request.mm_features[input_id].identifier
        # Not cached at all
        if mm_hash not in self.cached:
            return False

        # Cached but currently not referenced by any request
        if not self.cached[mm_hash]:
            num_encoder_embeds = self.freeable.pop(mm_hash)
            self.num_freeable_slots -= num_encoder_embeds

        self.cached[mm_hash].add(request.request_id)

Admission is where the compute budget and the space budget meet. Note that can_allocate checks the compute budget first, and evicts LRU-style from freeable only if the free slots alone are insufficient:

        num_embeds = request.get_num_encoder_embeds(input_id)

        # Not enough compute budget
        if num_embeds > encoder_compute_budget:
            return False

        num_embeds += num_embeds_to_schedule

        # Enough free slots
        if num_embeds <= self.num_free_slots:
            return True

        # Not enough reclaimable slots
        if num_embeds > self.num_freeable_slots:
            return False

        # Not enough free slots but enough reclaimable slots
        # NOTE: Eviction takes place here, but physical memory is not freed
        # until model runner is notified by the scheduler output.
        while num_embeds > self.num_free_slots:
            mm_hash, num_free_embeds = self.freeable.popitem(last=False)
            del self.cached[mm_hash]

The split between "logical eviction now, physical free later" matters: the manager runs in the scheduler process, and the tensors live in the worker's self.encoder_cache dict. The scheduler ships free_encoder_mm_hashes in its output and the runner pops them at the top of the step. Both budgets come from one function:

    encoder_compute_budget = max(
        scheduler_config.max_num_encoder_input_tokens, max_tokens_per_mm_item
    )
    encoder_cache_size = max(
        scheduler_config.encoder_cache_size, max_tokens_per_mm_item
    )

The max with max_tokens_per_mm_item is a safety floor: whatever the worst image the model accepts costs, one of them must always fit, or no image could ever be scheduled. For a Qwen2-VL configured with a 16,384-token cap, that floor is the budget on a default 2,048-token batch config — the encoder cache then holds exactly one worst-case image.

SGLang: a byte budget and a plain LRU

SGLang's MultiModalStaticCache is a module-level singleton, sized in bytes rather than slots, with no reference counting at all:

    def set(
        self,
        mm_hash: int,
        embedding: EmbeddingResult,
        loc: Optional[torch.Tensor] = None,
    ) -> bool:
        assert isinstance(embedding, EmbeddingResult), embedding
        if mm_hash in self.mm_cache:
            self.mm_cache.move_to_end(mm_hash)
            return True
        data_size = _get_tensor_size(embedding.embedding)
        while self.current_size + data_size > self.max_size:
            if not self.mm_cache:
                return False
            lru_hash, lru_embedding = self.mm_cache.popitem(last=False)
            self.current_size -= _get_tensor_size(lru_embedding.embedding)

        self.mm_cache[mm_hash] = embedding
        self.current_size += data_size
        return True

Its capacity is SGLANG_VLM_CACHE_SIZE_MB, default 100 (python/sglang/srt/environ.py:L1158), converted at KV-pool build time (python/sglang/srt/mem_cache/kv_cache_builder.py:L374-L375). At 11.2 MB per 1,369-token image that is nine images.

Why the designs differ. vLLM's scheduler must decide, before the forward pass, whether the step is feasible; a slot budget in the same unit as the token budget lets it do that arithmetic without knowing $d_{\text{model}}$ or the dtype. Refcounting is then mandatory, because evicting an entry a scheduled request still needs would be a hard crash (RuntimeError: Encoder cache miss). SGLang caches inside the forward path instead: the lookup happens in mm_schedule during embedding assembly, after the batch is already committed, so eviction can never strand a scheduled request and a plain LRU suffices. The cost is that SGLang's scheduler has no encoder-cache signal to admission-control on; the cost on vLLM's side is a budget expressed in a unit that hides the actual byte cost.

SGLang keys two ways, and which one you get is decided per request inside mm_embedding_chunk. A request whose every item occupies a single contiguous span (is_per_image = all(len(item.offsets) == 1 ...), python/sglang/srt/managers/mm_schedule.py:L512-L529) takes the per-image path on CUDA — that is the default for ordinary image requests. The fallback, for items with several offsets and for EVS video results, hashes the whole request's item list together:

    item_hashes = [item.hash for item in embedding_items_per_req]
    embedding_items_hash = MultiModalStaticCache.combine_hashes(item_hashes)
    embedding_per_req = embedding_cache.get(item_hashes)

    if embedding_per_req is None:
        # ...
        embedding = data_embedding_func(embedding_items_per_req)
        # ...
        embedding_cache.set(embedding_items_hash, embedding_per_req)

A combined key means a request that lands on that fallback with images [A, B] gets no reuse from a cached [A]. The per-image path (_batch_encode_per_image_misses, python/sglang/srt/managers/mm_schedule.py:L285-L355) does not have that problem: it uses get_single(item.hash) per item, collects the misses across every request in the batch, deduplicates by hash, and issues one ViT call for all of them. That is the batching win vLLM gets through group_and_batch_mm_kwargs. Note the platform split — HIP, NPU and XPU skip the cross-request batch and run a per-request variant instead (_get_chunked_embedding_by_item), with a source comment blaming a ROCm CI regression on "one large cross-request ViT batch". So the same request shape encodes three different ways depending on item layout and platform.

Here is the sharp question. The KV blocks covering the 1,369 image-token positions are ordinary KV blocks. A naive block hash over token ids would find them identical for any two images, because every image expands to the same repeated placeholder id. Serve that cache hit and you answer about the wrong picture.

This is not a hypothetical you can dodge by leaving prefix caching off, because it is on in both engines out of the box: vLLM's enable_prefix_caching defaults to True for generative decoder models (vllm/config/cache.py:L107, resolved via ModelConfig.is_prefix_caching_supported), and SGLang's radix cache is behind disable_radix_cache, default False (python/sglang/srt/server_args.py:L952-L954). Whatever each engine does about image identity, it is doing on every request you send it.

vLLM's fix is the extra-keys mechanism established in §2.3. Each block that overlaps a multimodal item contributes a (identifier, offset) pair to the block's extra keys:

        if end_token_idx > offset:
            if start_token_idx >= offset + length:
                # This block has passed the current mm input.
                curr_mm_idx += 1
                continue

            # The block contains the current mm input. Include its offset
            # relative to the start of the block so prefix-cache keys stay
            # distinct when the same MM item appears at different positions
            # within otherwise-identical placeholder blocks.
            extra_keys.append((mm_feature.identifier, offset - start_token_idx))

Two things are in the key: which image, and where in the block it starts. The second half is not decoration. Consider two requests whose prompts differ by one text token before the image. The image is the same item, but it lands at a different in-block offset, so the placeholder blocks are genuinely different KV and must not alias. Those keys are concatenated into the block hash alongside LoRA id, cache salt, and prompt-embeds keys:

    extra_keys: list[Any] = (
        lora_extra_keys + mm_extra_keys + cache_salt_keys + prompt_embeds_keys
    )

What computes the identifier, and what breaks it

                mm_hashes[modality] = [
                    hasher.hash_kwargs(
                        hash_algorithm,
                        model_id=model_id,
                        **{modality: item},
                        **hf_processor_mm_kwargs,
                    )
                    for item in data_items.get_all_items_for_hash()
                ]

The digest (blake3 by default; sha256/sha512 for FIPS — vllm/multimodal/hasher.py:L22-L47) covers the model id, the item, and the per-request processor kwargs. The last term is the trap: pass {"max_pixels": ...} on one request and not the next and you get two identifiers for one image, two encoder-cache entries, and no prefix reuse. Serialisation of a PIL image goes through mode plus raw pixel array (vllm/multimodal/hasher.py:L62-L76), so re-encoding a JPEG at a different quality changes the hash even when the picture looks identical — but note the escape hatch above it: an EXIF ImageID UUID short-circuits the whole thing, which is how a caller supplies a stable identity for an image it knows it will send repeatedly.

SGLang: rewrite the placeholder ids themselves

SGLang rewrites placeholder IDs to image-derived sentinels. Since these can exceed the vocabulary, every path to an embedding lookup must mask, replace or otherwise safely handle them before lookup; overwriting embeddings afterward does not retroactively make an invalid lookup safe. Trace the selected model's preprocessing and speculative path:

# Constant used as the base offset for MM (multimodal) pad values.
# This ensures pad_values don't overlap with valid text token IDs.
MM_PAD_SHIFT_VALUE = 1_000_000
_MM_HASH_MASK = (1 << 64) - 1
# ...
def _compute_pad_value(hash: int) -> int:
    """Compute pad value from hash."""
    return MM_PAD_SHIFT_VALUE + (hash % (1 << 30))

and the padding pass substitutes that value across the whole span:

            num_tokens = end_idx - start_idx - 1
            pad_value = pad_values[data_idx]
            padded_ids.extend([pad_value] * num_tokens)

            last_idx = end_idx

The radix tree sees these sentinels as ordinary sequence elements, but a 30-bit hash suffix is not a unique image identity. Under uniform hashing, n distinct items have collision probability approximately 1 - exp(-n*(n-1)/(2*2^30)); at 10,000 it is about 4.55%. This does not by itself prove wrong reuse in the complete engine: inspect additional identity fields, namespaces and collision checks. Do not claim the sentinel alone guarantees distinct images.

The cited SGLang identity combines preprocessed tensor bytes with a processor fingerprint; vLLM's key includes raw-input identity and processing arguments. Neither is universally immune to configuration errors. Changed pixel limits can produce identical processed data, while fingerprint changes can still invalidate an entry. Correct identity must include every choice that changes embeddings, including model/processor revision and trusted supplied IDs.

The item-level scheduler treats a vision encoding as a unit. That does not prevent the encoder's attention kernel from tiling queries and keys. Unlike causal prefix extension, an independently finalized patch prefix would miss later bidirectional context. Once the full embeddings exist, language-model prefill can consume them in chunks.

vLLM's default permits a chunk boundary inside the placeholder span: the encoder runs whole, the embeddings sit in the encoder cache, and successive chunks slice them (get_embeds_indices_in_range, used in both _try_schedule_encoder_inputs and _gather_mm_embeddings). Setting --disable-chunked-mm-input instead forces the chunk to stop short of the image:

            # If no encoder input chunking is allowed, we do not want to
            # partially schedule a multimodal item. If the scheduled range would
            # only cover part of the mm input, roll back to before the mm item.
            if (
                self.scheduler_config.disable_chunked_mm_input
                and num_computed_tokens < start_pos
                and (num_computed_tokens + num_new_tokens)
                < (start_pos + num_encoder_tokens)
            ):
                # Account for EAGLE shift when rolling back to avoid
                # encoder cache miss. This ensures the scheduled range
                # stops before start_pos even with the shift.
                num_new_tokens = max(
                    0, start_pos - (num_computed_tokens + shift_computed_tokens)
                )
                break

That flag has a hard interaction with the token budget, enforced at startup: compute_mm_encoder_budget raises if max_tokens_per_mm_item > max_num_batched_tokens while chunking is disabled (vllm/v1/core/encoder_cache_manager.py:L310-L320) — an unsplittable item that never fits in a batch would deadlock the scheduler. With a 16,384-token image cap you need --max-num-batched-tokens 16384 at minimum, which is a real cost on the text side.

SGLang does not schedule encoder work at all. mm_schedule runs inside the forward pass: it encodes whichever items overlap the current chunk, caches the full embedding, and slices out the chunk with get_embedding_chunk (python/sglang/srt/managers/mm_schedule.py:L28-L70). Chunk one pays the entire ViT cost and chunks two through $k$ hit the cache. Simpler, and no scheduler deadlock is possible — but the ViT cost lands unbudgeted inside one step, so a batch that happens to contain eight cache-missing images produces one very long step and a visible TPOT spike for every decode request riding along with it.

vLLM's counter-move on the compute side is CUDA graphs for the encoder. Because ViT input shapes vary per image, EncoderCudaGraphManager captures graphs at power-of-two token budgets and pads into the smallest one that fits:

    @staticmethod
    def _generate_budgets(min_budget: int, max_budget: int) -> list[int]:
        """Generate power-of-2 token budgets from min_budget to max_budget."""
        budgets: list[int] = []
        b = min_budget
        while b <= max_budget:
            budgets.append(b)
            b *= 2
        # Always include max_budget if it's not already a power-of-2 boundary
        if not budgets or budgets[-1] < max_budget:
            budgets.append(max_budget)
        return budgets

The floor for Qwen2.5-VL is 64 tokens — a 224×224 image, per the comment at vllm/model_executor/models/qwen2_5_vl.py:L1758-L1762. Padding waste is bounded at 2× by the power-of-two ladder, which is the usual bucket-vs-launch-overhead trade. It is off by default: the gate is cudagraph_mm_encoder: bool = False (vllm/config/compilation.py:L543), and the runner only constructs an encoder_cudagraph_manager when that flag is set and the model takes multimodal input (vllm/v1/worker/gpu_model_runner.py:L6741); with the manager None, the execute path falls straight through to model.embed_multimodal(**mm_kwargs_batch). Do not read the empty encoder_cudagraph_token_budgets list as the off switch — its docstring says empty means auto-infer the ladder from the model architecture (vllm/config/compilation.py:L548-L556). SGLang's equivalent is off by default too: SGLANG_VIT_ENABLE_CUDA_GRAPH = EnvBool(False) (python/sglang/srt/environ.py:L1163). For large images the ViT launch overhead is negligible against the GEMMs, so neither project pays the capture cost by default.

P/D disaggregation (§1.6) splits a request at the prefill/decode boundary and ships KV. Encode-prefill disaggregation splits it one stage earlier and ships embeddings. SGLang implements it in python/sglang/srt/disaggregation/encoder/ — 8,131 lines across six modules at 7d89325 (grpc_server.py, http_server.py, preprocessor.py, receiver.py, runtime.py, server.py).

What actually crosses the wire in each direction is worth being precise about, because it is the whole economic argument.

Outbound: URL inputs send references and move fetching/preprocessing to the encoder. Base64 inputs carry the media bytes themselves with encoding overhead; they are not only a small reference. Measure both directions and preprocessing CPU cost for the actual payload type.

Inbound: an EmbeddingData carrying req_id, num_parts, part_idx, grid_dim, modality, dtype/shape, and the tensor itself (python/sglang/srt/disaggregation/encoder/receiver.py:L397-L432). It lands in a preallocated GPU pool; the shown allocator's free-list/first-fit policy is not a ring:

class EmbeddingPool:
    """Persistent GPU buffer pool for received multimodal embeddings.

    Allocator: first-fit on a free-segment list with 256-byte alignment.
    `alloc()` blocks on a Condition when the pool is full and resumes once
    a peer `release()`s a slot; `try_alloc()` is the non-blocking variant
    for callers that re-poll (the zmq_to_scheduler tick). Each successful
    alloc returns a slot_id that must be passed back to release() when the
    consumer is done with the buffer. With `engine` set (mooncake), the
    buffer is registered once so encoder RDMA writes land in pool slots.
    """

Sizing it, and why the economics differ from P/D

For a language model with $L$ layers, $h_{kv}$ KV heads and head dim $d_h$ in bf16, one token's KV is $4 L h_{kv} d_h$ bytes and one token's embedding is $2 d$ bytes. The ratio is

For Llama-3-8B ($L=32$, $h_{kv}=8$, $d_h=128$, $d=4096$) that is $2 \cdot 32 \cdot 8 \cdot 128 / 4096 = 16$. EPD ships one-sixteenth the bytes that P/D ships for the same span. One 1024×1024 image is 10.7 MiB (11,214,848 B) of embeddings against 171.1 MiB (179,437,568 B) of KV; on a 25 GB/s effective link that is 0.45 ms against 7.2 ms (derived: bytes ÷ $25 \times 10^{9}$). The transfer is small enough that plain ZMQ over TCP is the default backend, with RDMA reserved for a specific case:

def resolve_encoder_transfer_backend(
    backend: str, model_arch: str, tp_size: int
) -> str:
    if backend != "auto":
        return backend
    if model_arch == "KimiK3ForConditionalGeneration" and tp_size > 1:
        return "zmq_to_tokenizer"
    return "zmq_to_scheduler"

The second difference is hardware. A P/D prefill node and a decode node run the same weights and need comparable memory. An encoder pool runs only the vision tower — no KV cache, no long context — so it fits on cheaper, smaller cards while the language side keeps its H100s entirely for KV. The third is fan-out: the receiver splits one request's items across several encoder URLs (MMReceiverHTTP.encode, python/sglang/srt/disaggregation/encoder/receiver.py:L2430-L2470), so a multi-image request can distribute independent items and reassemble by part index. This differs from sequence/context parallelism, which can also distribute prefill work across devices; P/D is not inherently one sequence on one node. Encoders register their URL at bootstrap, default port 8997 (python/sglang/srt/server_args.py:L3232-L3241), and --enable-prefix-mm-cache (which the arg validator requires be paired with --encoder-only, python/sglang/srt/server_args.py:L7809-L7812) gives the pool a cross-request, cross-node embedding cache the single-process design cannot offer.

vLLM has the same idea, earlier: ECConnectorBase (vllm/distributed/ec_transfer/ec_connector/base.py:L1-L24) defines a scheduler-side / worker-side connector pair for "Distributed Encoder Cache & P2P Encoder cache communication", and the scheduler already consults it — has_cache_item(item_identifier) routes an item to external_load_encoder_input instead of local compute (vllm/v1/core/sched/scheduler.py:L1683-L1689). That branch is dead unless you configure one: the guard is self.ec_connector is not None, and no connector is built without an explicit config. The shipped implementation is a CPU/shared-region connector (vllm/distributed/ec_transfer/ec_connector/cpu/ec_shared_region.py), not a remote encoder pool.

One request, "<image> what is this?", 1024×1024 PNG, Qwen2-VL, chunked prefill on with max_num_batched_tokens=8192.

1. Front end. BaseMultiModalProcessor.apply (vllm/multimodal/processing/processor.py:L1839-L1870) runs the HF processor on text and image together, producing pixel_values of shape [5476, 1176]. Then _maybe_apply_prompt_updates replaces the single <image> token with 1,369 placeholder ids and records a PlaceholderRange(offset=6, length=1369).
2. Hashing. ProcessorInputs.get_mm_hashes (vllm/multimodal/processing/inputs.py:L67-L75) computes the blake3 identifier over model_id, the image and the processor kwargs. A CPU-side LRU keyed on that hash can skip step 1 entirely on a repeat. Which LRU depends on topology, and the naming is easy to get wrong: MultiModalRegistry._get_cache_type (vllm/multimodal/registry.py:L275-L299) returns None when mm_processor_cache_gb <= 0, "processor_only" when IPC caching is unavailable (more than one API-server process, or DP without an external load balancer), and otherwise mm_processor_cache_type, whose default is "lru" (vllm/config/multimodal.py:L164). A plain single-process vllm serve therefore gets MultiModalProcessorSenderCache (vllm/multimodal/cache.py:L444-L458), which stores only item metadata on P0 so the eviction policy mirrors the engine-side cache without duplicating tensors, and clears the payload from the input to avoid re-sending it over IPC. MultiModalProcessorOnlyCache (vllm/multimodal/cache.py:L392-L410), which does keep the processed tensors, is the no-IPC path. Both are sized by mm_processor_cache_gb, default 4 (vllm/config/multimodal.py:L152).
3. Scheduler, step $t$. Scheduler.schedule reaches the encoder hook (vllm/v1/core/sched/scheduler.py:L592-L607) and calls _try_schedule_encoder_inputs. get_mm_features_in_window finds the item overlapping [0, 8192); check_and_update_cache misses; can_allocate compares 1,369 against the compute budget and the free slots and returns True. The item id is appended to encoder_inputs_to_schedule, and back in schedule() the manager reserves the slots (vllm/v1/core/sched/scheduler.py:L730-L738).
4. Worker, encode. _execute_mm_encoder (vllm/v1/worker/gpu_model_runner.py:L3077-L3080) batches by modality via group_and_batch_mm_kwargs, optionally replays a captured encoder CUDA graph, calls model.embed_multimodal(**mm_kwargs_batch), then stores by hash:
   vllm/v1/worker/gpu_model_runner.py:L3283-L3291 vLLM

           # Cache the encoder outputs by mm_hash
           for mm_hash, output in zip(mm_hashes, encoder_outputs):
               self._cache_encoder_output(
                   mm_hash,
                   output,
                   scheduler_output.ec_manager_metadata,
                   scheduler_output.free_encoder_mm_hashes,
               )
               logger.debug("Finish execute for mm hash %s", mm_hash)

5. Worker, gather. _gather_mm_embeddings walks the batch, clips each item's range to the scheduled window, and builds a pinned CPU boolean mask:
   vllm/v1/worker/gpu_model_runner.py:L3352-L3363 vLLM

                   mm_hash = mm_feature.identifier
                   encoder_output = self._get_encoder_output_from_cache(mm_hash)
                   if encoder_output is None:
                       # A feature starting at/after the processed boundary is only
                       # reached via the drafter's +1 look-ahead and might not be
                       # encoded yet; fall back to the token embedding for drafting.
                       if (
                           start_pos
                           >= req_state.num_computed_tokens + num_scheduled_tokens
                       ):
                           continue
                       raise RuntimeError(f"Encoder cache miss for {mm_hash}.")

6. Splice. SupportsMultiModal.embed_input_ids (vllm/model_executor/models/interfaces.py:L453-L487) embeds the text ids, then calls _merge_multimodal_embeddings — the masked index-put shown in §1. The pinned-CPU mask is deliberate: indexing with a CPU boolean avoids a D2H sync in the hot path.
7. Prefill and block hashing. The language model runs on inputs_embeds. As blocks fill, hash_request_tokens calls generate_block_hash_extra_keys, which threads curr_mm_idx through the block loop so each block picks up the (identifier, offset) pairs for the items it covers (vllm/v1/core/kv_cache_utils.py:L745-L757).
8. Release. When the request's last token past the image is computed, the scheduler calls free_encoder_input; the entry moves to freeable but its tensor stays resident. Only a later can_allocate that needs the space calls popitem(last=False) and appends the hash to freed, which reaches the worker as free_encoder_mm_hashes and pops the tensor (vllm/v1/worker/gpu_model_runner.py:L1238-L1244). Until then a second request with the same image gets a free hit.

Irregular batch shapes. vLLM's group_and_batch_mm_kwargs only groups contiguous runs of the same modality, with an explicit apology in the source: "FIXME (ywang96): This is a hacky way to deal with multiple modalities in the same batch... The proper solution should be reordering the encoder outputs" (vllm/v1/worker/gpu_model_runner.py:L3124-L3130). Interleaving image and audio items in one request fragments the batch into many small ViT calls.

Memory spike from many large images. The pixel tensor for one 1024×1024 image is $5476 \times 1176 \times 4 = 25.8$ MB in fp32 — more than twice its embedding output. Eight arriving in one step is ~206 MB of transient activation on top of the ViT's own, none of it in the KV budget. vLLM is candid about the general problem: "Temporary hack to limit peak memory usage when processing multimodal data... Scheduler uses pruned vision tokens count to compare it versus compute budget which is incorrect" (vllm/v1/worker/gpu_model_runner.py:L3211-L3224). The encoder compute budget counts output embeddings; peak memory scales with input pixels. Only one of those is budgeted.

Separate processor-output, encoder-embedding and language-KV caches. Only the last depends on the complete causal prefix; a language-worker embedding hit avoids a transfer that an encoder-worker hit does not. For URLs, constrain schemes/hosts and redirect destinations, enforce decoded size/frame/audio-duration limits, and isolate media decoding. Treat caller-supplied IDs as trusted claims only within an authenticated namespace. The birthday calculation below is a hash-space check, not a demonstrated engine exploit.

from math import exp

buckets = 2**30
items = 10_000
probability = 1 - exp(-items*(items-1)/(2*buckets))
assert 0.045 < probability < 0.046
def kv_identity(prefix, image_id, model_revision, processor_revision):
    return (tuple(prefix), image_id, model_revision, processor_revision)
a = kv_identity([1, 2], "image-A", "model-v1", "processor-v1")
b = kv_identity([9, 2], "image-A", "model-v1", "processor-v1")
assert a != b
assert kv_identity([1, 2], "image-A", "model-v2", "processor-v1") != a
print("30-bit collision approximation:", probability)
print("Matching image identity alone is insufficient for causal KV reuse.")

Confirm the token expansion for yourself without a GPU — the arithmetic is pure Python:

python3 -c '
import math
F, P, M = 28, 14, 2
MAXPX = 16384 * 28 * 28
def rbf(n, f): return round(n / f) * f
def flr(n, f): return math.floor(n / f) * f
def tokens(h, w):
    hb, wb = max(F, rbf(h, F)), max(F, rbf(w, F))
    if hb * wb > MAXPX:
        beta = math.sqrt((h * w) / MAXPX)
        hb, wb = flr(h / beta, F), flr(w / beta, F)
    return (hb // P) * (wb // P) // (M * M), (hb, wb)
for hw in [(336,336),(512,512),(1024,1024),(1080,1920),(2160,3840)]:
    n, sz = tokens(*hw)
    print(f"{hw[1]}x{hw[0]} -> resized {sz[1]}x{sz[0]}  {n:6d} tokens  "
          f"{n*128/1024:8.1f} MiB KV @128KiB/token")'

Then flip the two flags that matter, on a real server, and watch the scheduler:

# baseline: encoder cache = max(8192, max_tokens_per_mm_item) slots.
# Read the resolved value off the startup log, do not assume 8192.
vllm serve Qwen/Qwen2-VL-7B-Instruct --max-num-batched-tokens 8192 \
    --limit-mm-per-prompt '{"image": 4}'

# forbid splitting an image across chunks; needs a batch big enough for the worst image
vllm serve Qwen/Qwen2-VL-7B-Instruct --max-num-batched-tokens 16384 \
    --disable-chunked-mm-input

# SGLang: shrink the embedding cache to one image and watch the ViT re-run
SGLANG_VLM_CACHE_SIZE_MB=12 python -m sglang.launch_server \
    --model-path Qwen/Qwen2-VL-7B-Instruct --chunked-prefill-size 8192

Send repeated images and log the exact cache tier's hit/miss, encoder invocations and transfer bytes, not just TTFT. Cache lifetimes differ. Change processing arguments, then inspect both processed output and identity: different max-pixel limits need not change an already-small image, and not every backend keys arguments identically. Run servers sequentially or isolate GPUs.

• vLLM PR #6613 — the original V1 encoder cache manager and multimodal scheduling in the V1 engine.
• vLLM issue #18334 — cited in vllm/multimodal/hasher.py:L35; why blake3 is the default and sha256/sha512 exist as FIPS alternatives for the multimodal hash.
• SGLang disaggregation/encoder/ — read server.py (MMEncoder), receiver.py (EmbeddingPool, MMReceiverBase) and runtime.py (EncoderScheduler, DPDispatcher) in that order.
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution — the naive dynamic resolution scheme whose smart_resize both engines reimplement.
• §2.3 for the extra-key contract this chapter extends, and §1.6 for the P/D economics that §7 compares against. The public API surface for image inputs is §9.1.