Offload, hierarchical cache, and budgeting VRAM

A Llama-3-8B server has been up for eleven minutes on a single H100. It started fine — the log said Available KV cache memory: 54.4 GiB, the health check passed, traffic ramped. Then:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.16 GiB.
GPU 0 has a total capacity of 79.65 GiB of which 812.00 MiB is free.
Of the allocated memory 76.12 GiB is allocated by PyTorch, and 1.94 GiB
is reserved by PyTorch but unallocated.

Nothing in the workload changed, no request was larger than the ones flowing for ten minutes, and the engine had explicitly profiled its own peak at startup. Yet it died — not at startup, where a wrong budget is caught, but deep into a run, where it costs you a page.

Whoever set this up had raised gpu_memory_utilization to 0.98 because the startup log showed 6 GiB apparently idle. That 6 GiB was the margin between what the profiler could measure and what the process would eventually touch. This chapter is about what lives in that margin, how to read it out of the engine's own logs instead of guessing, and what to do with a tier of memory 50× slower than HBM and 20× larger.

The KV pool is a residual, not an allocation. You do not ask for it; you get whatever survives a subtraction. Every engine performs the same subtraction, and the only real difference between vLLM and SGLang is which terms they measure and which they estimate. Terms that are measured shrink to fit reality. Terms that are estimated must be over-estimated, or the process dies later.

Four of those five non-KV terms are measured by a real forward pass. The fifth — the slack above the cap — is the only thing standing between you and the OOM in §1, and it is the one people delete.

Start from FORMULAS:

with $C_{\text{total}}$ the device capacity, $u$ the utilisation fraction, $W$ the weight bytes, $A_{\text{peak}}$ the transient activation peak, $O$ the non-torch allocations (CUDA context, NCCL communication buffers, attention-backend workspaces allocated outside the caching allocator), and $G$ the CUDA-graph memory pool. Work it for Llama-3-8B in bf16 on one H100 SXM 80 GB.

$C_{\text{total}}$. The device reports 79.6 GiB, not 80 — get_memory_info returns what the driver will hand out, which excludes ECC and driver reservations. Use the reported number.

$u$. As of vLLM a556f3f the default is 0.92, not the 0.90 that most blog posts quote:

    gpu_memory_utilization: float = Field(default=0.92, gt=0, le=1)
    """The fraction of GPU memory to be used for the model executor, which can
    range from 0 to 1. For example, a value of 0.5 would imply 50% GPU memory
    utilization. If unspecified, will use the default value of 0.92. This is a
    per-instance limit, and only applies to the current vLLM instance. It does
    not matter if you have another vLLM instance running on the same GPU. For
    example, if you have two vLLM instances running on the same GPU, you can
    set the GPU memory utilization to 0.5 for each instance."""

Note the semantics carefully: $u$ multiplies total_memory, not free memory. It is a planning target, not an enforced allocator ceiling or a guarantee that two instances at 0.5 coexist safely. So $73.2 = 79.6 \times 0.92$ GiB is the whole budget for weights, activations, overhead and KV together.

def request_memory(init_snapshot: MemorySnapshot, cache_config: CacheConfig) -> int:
    """
    Calculate the amount of memory required by vLLM, then validate
    that the current amount of free memory is sufficient for that.
    """
    requested_memory = math.ceil(
        init_snapshot.total_memory * cache_config.gpu_memory_utilization
    )

    if init_snapshot.free_memory < requested_memory:
        raise ValueError(
            f"Free memory on device {init_snapshot.device_} "
            f"({format_gib(init_snapshot.free_memory)}/"
            f"{format_gib(init_snapshot.total_memory)} GiB) on startup "
            f"is less than desired GPU memory utilization "
            f"({cache_config.gpu_memory_utilization}, "
            f"{format_gib(requested_memory)} GiB). Decrease GPU memory "
            f"utilization or reduce GPU memory used by other processes."

$W$. 8.03 × 10⁹ parameters at 2 bytes = 16.06 × 10⁹ B = 14.96 GiB. This term is not estimated either — it is handed to the profiler as model_memory_usage, recorded by the loader.

$A_{\text{peak}}$. Derivable only to an order of magnitude. At max_num_batched_tokens = 8192 the resident layer working set is hidden states $8192 \times 4096 \times 2 = 67$ MB, MLP intermediate $8192 \times 14336 \times 2 = 235$ MB, and a handful of those alive concurrently — call it 1 GiB. But the true peak also includes the attention backend's scratch workspace, the sampler's logits buffer, and any all-gather staging, none of which follows from model shape. This is exactly why the engines profile instead of computing.

vLLM measures

Worker.determine_available_memory runs a synthetic forward pass at the largest shape the scheduler can ever produce, inside a context manager that brackets it with two memory snapshots:

        # Execute a forward pass with dummy inputs to profile the memory usage
        # of the model.
        with memory_profiling(
            self.init_snapshot,
            weights_memory=int(self.model_runner.model_memory_usage),
        ) as profile_result:
            self.model_runner.profile_run()

The dummy run is self._dummy_run(self.max_num_tokens, is_profile=True) at vllm/v1/worker/gpu_model_runner.py:L6612-L6615 plus a dummy sampler or pooler run — the is_profile flag exists specifically "to pre-allocate communication buffers" — and, for multimodal models, an encoder pass on maximum-feature-size items. The accounting is the interesting part: total_consumed comes from mem_get_info, not PyTorch's reserved counter.

    # Measure total consumption via mem_get_info() instead of
    # memory_reserved(), which goes negative when pluggable allocators
    # (e.g. cumem) bypass PyTorch's tracking.
    result.total_consumed = (
        result.before_create.free_memory - result.after_profile.free_memory
    )

    # total_consumed already covers persistent torch allocations; add only the
    # transient peak headroom to avoid double-counting.
    result.transient_peak_headroom = (
        result.after_profile.torch_peak - result.after_profile.torch_allocated
    )
    result.non_kv_cache_memory = result.total_consumed + result.transient_peak_headroom

total_consumed is everything the process took from the driver, torch and non-torch alike; transient_peak_headroom is the spike above what stayed resident, which the allocator needs again on every real forward pass. Their sum is what must be held back. CUDA graphs — replayable recordings of a whole decode step's kernel launches, which reserve their own memory pool (§8.1) — are profiled separately, by capturing every graph descriptor into a throwaway pool against a minimal KV cache (vllm/v1/worker/gpu_model_runner.py:L6775-L6797). The final subtraction:

        self.available_kv_cache_memory_bytes = (
            self.requested_memory
            - profile_result.non_kv_cache_memory
            - cudagraph_memory_estimate_applied
        )

Bytes become blocks by a plain division at vllm/v1/core/kv_cache_utils.py:L1023-L1041: num_blocks = int(available_memory // page_size // num_layers). With $b_{tok} = 131{,}072$ B for Llama-3-8B (from §2.1), 54.4 GiB is $5.841 \times 10^{10}$ bytes; the two floor divisions give $5.841\times10^{10} \mathbin{//} 65{,}536 \mathbin{//} 32 = \mathbf{27{,}852}$ blocks of 16, i.e. 445,632 token slots, or 54 concurrent sequences at 8k context. Derived.

SGLang estimates, then measures what is left

SGLang's mem_fraction_static means something structurally different. It is not a ceiling on consumption; it is the complement of a reserve. The default is computed from a closed-form heuristic before anything runs:

                # Tokens the activation working set scales with (per serving mode).
                if self.disaggregation_mode == "decode":
                    running_requests = (
                        self.max_running_requests
                        or decode_cuda_graph_config.max_bs
                        or 1
                    )
                    draft_tokens = self.speculative_num_draft_tokens or 1
                    activation_tokens = max(running_requests * draft_tokens, 2048)
                elif self.chunked_prefill_size > 0:
                    activation_tokens = max(self.chunked_prefill_size, 2048)
                else:
                    activation_tokens = max(self.max_prefill_tokens, 2048)
                # Constant meta data (e.g., from attention backend) + activation slack.
                reserved_mem = 512
                reserved_mem += activation_tokens * 1.5
                # Some adjustments for large parallel size
                reserved_mem += self.tp_size * self.pp_size / 8 * 1024
                reserved_mem += self.reserve_for_graph_mb()
                if gpu_mem is not None and gpu_mem > 60 * 1024:
                    reserved_mem = max(reserved_mem, 10 * 1024)

The docstring is unusually candid about this: "The coefficient 1.5 is a heuristic value, in the future, we can do better estimation by looking at the model types, hidden sizes or even do a dummy run" (python/sglang/srt/server_args.py:L4839-L4862). On any GPU above 60 GB the reserve is floored at 10 GB. On our 80 GB card with chunked_prefill_size = 8192 the formula gives $512 + 12288 + 128 + G$ MB, already above the floor, so $u_{\text{static}} = (81920 - \text{reserved})/81920$.

Then, crucially, the actual pool size is not mem_fraction_static × capacity. It is measured free memory minus a slack proportional to the complement:

    def _profile_available_bytes(self, pre_model_load_memory: int) -> int:
        # KV pool budget = currently-free GPU memory minus the non-static runtime
        # slack (pre_model_load_memory * (1 - mem_fraction_static)). Whatever is
        # already resident (model weights, etc.) is thus charged against it.
        available_gpu_memory = get_available_gpu_memory(
            self.device,
            self.gpu_id,
            distributed=get_world_group().world_size > 1,
            cpu_group=get_world_group().cpu_group,
        )

        slack_gb = pre_model_load_memory * (1 - get_schedule().mem_fraction_static)

So SGLang measures capacity and weights but estimates activations and graphs; vLLM measures all four. The tradeoff runs both ways: vLLM's profile costs a max-shape forward pass plus a full graph capture at every startup and asserts that no other process on the card moves during it, while SGLang starts faster but hands you a formula to tune by hand for an unusual model. A newer opt-in path (SGLANG_ENABLE_POST_CAPTURE_KV_SIZING, gated at python/sglang/srt/server_args.py:L5043-L5067) sizes the pool after graph capture instead, dropping the reserve to a 1536 MB floor — but it excludes MLA, DCP, fp4 KV and memory-saver mode, so most DeepSeek deployments still use the heuristic.

The profile is a path-specific estimate of the peak, and the engine knows it — when suggesting an explicit --kv-cache-memory, vLLM subtracts an admitted fudge factor:

            # empirically observed that the memory profiling may
            # slightly underestimate the memory consumption.
            # So leave a small buffer (=150MiB) to avoid OOM.
            redundancy_buffer_memory = 150 * (1 << 20)

            non_kv_cache_memory = (
                self.total_consumed
                + self.peak_activation_memory
                + cuda_graph_memory_bytes
            )

Four things live in the gap between profile and reality, and none of them are visible at startup:

At $u = 0.92$ you hold 6.4 GiB back; at $u = 0.98$, 1.6 GiB, buying 4.8 GiB of KV — a 9% larger pool, roughly 5 more concurrent 8k sequences. That is the trade: 9% more concurrency against every one of the four failure modes above. Take it on a dedicated card with a fixed workload and a tested maximum prompt length; never on shared hardware.

Four distinct failures produce four distinct strings. Matching the string to the cause saves an hour.

A host tier smaller than the GPU tier reduces the potential cache window but is not necessarily useless. Selective backup, different eviction orders, and expensive infrequently reused prefixes can still produce useful hits. Measure useful-byte hit rate and avoided recomputation against transfer and host-memory cost.

Now the second half. Below HBM sit three more tiers, each roughly an order of magnitude larger and slower. §1.4 derived the crossover for a preemption swap, which pays PCIe twice — out and back: $R^{*} = B_{\text{pcie}} / (2 b_{tok})$. A hierarchical cache read is different: the write happened minutes ago, off the critical path, and only the inbound leg blocks the request. So the threshold is twice as permissive:

Fetching beats recomputing whenever the engine's achieved prefill rate $R$ is below $R^{*}_{\text{read}}$. Using $R \approx 22{,}000$ tok/s for Llama-3-8B at 400 TFLOP/s achieved (§1.4, derived) and $b_{tok} = 131{,}072$ B:

Bandwidth provenance: HBM3 at 3.35 TB/s is the H100 SXM datasheet figure fixed in §0.4. PCIe Gen5 x16 at 64 GB/s per direction is the PCI-SIG link rate, as used in §1.4. 14 GB/s is the vendor sequential-read spec for a PCIe Gen5 x4 enterprise NVMe (e.g. Kioxia CM7-R class). 12.5 and 1.25 GB/s are 100 GbE and 10 GbE line rates. All are optimistic ceilings; real pinned-copy and filesystem throughput lands lower, which moves each tier left — toward the recompute line, never away from it.

For the assumed 8B model and recomputation rate, the bandwidth-only calculation favors PCIe over recomputation and disfavors 10 GbE. These are model-specific comparisons, not universal rules: link startup, CPU/NUMA contention, shared bandwidth, prefix size, and faster or slower prefill can reverse the result.

Most writing about SGLang's hierarchical cache describes a class called HiRadixCache. At 7d89325 that class is never constructed:

$ grep -rn "HiRadixCache(" python/sglang/srt/
python/sglang/srt/mem_cache/hiradix_cache.py:77:class HiRadixCache(RadixCache):

One hit, and it is the class statement. The live path runs through a factory instead. Every branch of default_radix_cache_factory returns a ChunkCache variant, RadixCacheCpp (behind SGLANG_EXPERIMENTAL_CPP_RADIX_TREE), PureSWARadixCache, LMCRadixCache or the FlexKV factory — and the fall-through, which is what a normal server gets, is _create_unified_radix_cache (python/sglang/srt/mem_cache/registry.py:L80-L143). HiCache is then bolted onto that object by a second call:

    cache = UnifiedRadixCache(params)
    if (
        ctx.enable_hierarchical_cache
        or get_disagg().disaggregation_decode_retraction_backup == "host_pool"
    ):
        cache.init_hicache(server_args, params)
        ctx.tp_worker.register_hicache_layer_transfer_counter(
            cache.cache_controller.layer_done_counter
        )
    return cache

That is the shape of the whole subsystem now. UnifiedRadixCache is a plain prefix cache whose HiCache fields are declared inert at construction — "HiCache D↔H defaults (overridden by init_hicache)" and "Owns the storage backend lifecycle; built by init_hicache" (python/sglang/srt/mem_cache/unified_radix_cache.py:L230-L235) — and init_hicache attaches the host pools, controller and storage backend after the fact. Tiering is a layer, not a subclass.

The convergence is deliberate and documented: the per-model radix variants "(radix_cache.py, swa_radix_cache.py, mamba_radix_cache.py, hiradix_cache.py, chunk_cache.py) are converging onto the Unified Radix Cache" (python/sglang/srt/mem_cache/README.md:L38-L45). Two independently-grown prefix caches merged into one tree with a component model, and the hierarchy became something you attach to it. hiradix_cache.py is still in the repo and is often the clearest statement of an idea, so this chapter quotes it where that holds — always labelled. You cannot reach it with a flag.

With that settled: the unified tree gives every node a second address — host_value (host-pool indices) alongside its device value, plus a hash_value used as the key in an external store. The code and metrics call these L1, L2 and L3.

Sizing the host tier

        self.dtype = device_pool.store_dtype
        self.size_per_token = self.get_size_per_token()
        if host_size > 0:
            self.size = sync_fixed_hicache_size(
                int(host_size * 1e9 // self.size_per_token), host_size
            )
        else:
            self.size = int(device_pool.size * host_to_device_ratio)
        # Align up the host memory pool size to the page size
        self.page_num = self.size // self.page_size + 1
        self.size = self.page_num * self.page_size

--hicache-size in GB wins if set; otherwise --hicache-ratio multiplies the device pool. Defaults as of 7d89325: 2.0 in cache mode, 1.2 in buffer_only mode, and unset for a decode-role server, where kv_cache_builder resolves it against the retraction-backup backend (python/sglang/srt/server_args.py:L7426-L7440). With our 445,632-token GPU pool, the default ratio 2.0 buys 891,264 host tokens = 109 GiB of pinned host RAM — a number worth knowing before you enable the flag on a 128 GB box.

Write policy: through, selective, or back

Three policies, validated in the controller (python/sglang/srt/managers/cache_controller.py:L334-L339) and resolved into two pieces of tree state by init_hicache:

        # State initialization
        self.write_through_threshold = (
            1 if server_args.hicache_write_policy == "write_through" else 2
        )
        self.is_write_back = (
            self.cache_controller is not None
            and self.cache_controller.write_policy == "write_back"
        )

Those two flow down to the tree core, where the whole policy reduces to one predicate evaluated on every prefix hit:

    def _inc_hit_count_and_check(
        self, node: UnifiedTreeNode, chunked: bool = False
    ) -> bool:
        """Increment hit count; check whether a write backup should be fired."""
        if node.evicted or chunked:
            return False
        if self.is_write_back:
            return False
        node.hit_count += 1
        return (
            self.enable_hicache
            and not node.backuped
            and node.hit_count >= self.write_through_threshold
        )

So: write_through (threshold 1) copies every node to host on its first prefix hit; write_through_selective (threshold 2) waits for a second hit, filtering out one-shot prefixes that would only pollute the host pool; write_back short-circuits the predicate entirely and instead stages nodes to host at eviction time, in evict_device_leaf, which evicts "one device leaf (demote if backuped, delete if write-through); for an unbacked write-back node, the result carries the BackupKV for the cache to execute and then demote" (python/sglang/srt/mem_cache/unified_cache/unified_tree_core.py:L1244-L1250). The default is write_through, and write-back is on its way out: the earlier implementation already carried "note this path will be deprecated in the future" over its write-back eviction (python/sglang/srt/mem_cache/hiradix_cache.py:L1231-L1233 — earlier implementation; it reads as foreshadowing). The reason is structural: write-back must do a host allocation and a D2H copy on the eviction path, inside the allocator's critical section.

The backup invariant, and where the merge improved it

Backed-up nodes must form a contiguous prefix from the root — a host tier with holes never serves anything, since a prefix match is only usable if it reaches the root. Both implementations enforce that and disagree about how, which is the clearest single example of what the convergence bought. The earlier HiRadixCache enforced it by refusing: if the parent is not backed up, return 0.

    def write_backup(self, node: TreeNode, write_back=False) -> int:
        # Backup invariant (for write-through mode): backed-up nodes must form a
        # contiguous prefix from root — no gaps.  Skip if parent isn't backed
        # up yet;
        if not write_back and (
            node.parent != self.root_node and not node.parent.backuped
        ):
            return 0

The unified tree enforces the same invariant by repairing: it walks up collecting unbacked ancestors and reverses the chain so they are written first.

    def _build_backup_kv_action(
        self, node: UnifiedTreeNode, write_back: bool = False
    ) -> BackupKV:
        """Build the backup action for a node and its unbacked ancestors."""
        chain = [node]
        if not write_back:
            ancestor = node.parent
            while (
                ancestor is not None
                and ancestor is not self.root_node
                and not ancestor.backuped
            ):
                chain.append(ancestor)
                ancestor = ancestor.parent
            # write_through: Ancestors first to preserve backup invariant
            chain.reverse()
        return BackupKV([target.id for target in chain])

Same invariant, opposite failure mode. Under the old rule a hot node reached only through a cold ancestor could stall indefinitely; under the new one, its first hit drags the whole unbacked ancestry to host in a single action. If you are reading numbers from a HiCache write-up aimed at the old class, this is one reason they may not reproduce.

The storage tier and its backends

L3 is behind one ABC, HiCacheStorage (python/sglang/srt/mem_cache/hicache_storage.py:L150-L155), whose interesting methods are the _v2 family: batch_exists_v2 performs a longest-prefix existence probe across multiple pools before any data moves, and batch_get_v2/batch_set_v2 move pages directly into and out of registered host-pool memory. Ten backends are registered at 7d89325:

# Register built-in storage backends
StorageBackendFactory.register_backend(
    "file", "sglang.srt.mem_cache.hicache_storage", "HiCacheFile"
)
# ... sim, nixl, mooncake, hf3fs, aibrix, eic, simm ...
StorageBackendFactory.register_backend(
    "mori",
    "sglang.srt.mem_cache.storage.umbp.umbp_store",
    "UMBPStore",
)

StorageBackendFactory.register_backend(
    "shm",
    "sglang.srt.mem_cache.storage.shm",
    "HiCacheShm",
)

Mapped onto Figure 2's tiers: file and hf3fs are filesystem (NVMe class), shm and simm are shared host memory, mooncake, aibrix and eic are remote KV stores, nixl and mori are RDMA transports, and sim is a test simulator. A dynamic pseudo-backend loads an out-of-tree class by module path, so the list is a floor. The lmcache and flexkv directories under storage/ are not registered here — they integrate as alternative radix caches (lmc_radix_cache.py, flexkv_radix_cache.py), not L3 backends.

Follow one request whose 10,240-token prompt hits 2,048 tokens in the GPU tree and needs 8,192 more.

1. prefetch_from_storage (python/sglang/srt/mem_cache/unified_radix_cache.py:L1587-L1620) builds a page-aligned key and bails immediately if the tail is shorter than prefetch_threshold (256 tokens by default, overridable through hicache_storage_backend_extra_config), if the controller is rate-limited, or if a fetch for this request is already in flight — "overwriting would leak its staging slots". Nothing is allocated yet: the probe comes first.
2. The existence probe. The storage thread calls batch_exists_v2, which returns a PoolTransferResult whose kv_hit_pages is the usable prefix length across every registered pool — the minimum, so a missing auxiliary page (Mamba state, or DeepSeek-V3.2's DSA sparse-attention indexer, §7.2) shrinks the KV prefix too (python/sglang/srt/mem_cache/hicache_storage.py:L165-L196).
3. Second threshold check, then allocate. Back on the scheduler thread the drain revokes the prefetch when operation.storage_hit_count < self.prefetch_threshold — a "below-threshold hit … (not enough benefit)" (python/sglang/srt/mem_cache/unified_radix_cache.py:L2185-L2191) — and only then allocates host pages for exactly the hit length. Under host pressure it does not give up outright: it retries with "a shorter page-aligned prefix", and abandons only if even that falls below the threshold (python/sglang/srt/mem_cache/unified_radix_cache.py:L2135-L2154).
4. Fetch and graft. check_prefetch_progress reduces the completed token count to a minimum across attention groups via _sync_and_check_hybrid_prefetch_result so every rank agrees on one usable prefix, then tree_core.insert_host grafts the fetched pages onto the tree as host_value (python/sglang/srt/mem_cache/unified_radix_cache.py:L1754-L1810). Anything beyond the agreed prefix goes on the host-release queue. The graft respects the same invariant as a write-through backup: a refill under an un-backed-up node is dropped rather than creating a hole, and says so — HiCache prefetch dropped %d-token refill under un-backed-up node %d (python/sglang/srt/mem_cache/unified_cache/unified_tree_core.py:L1786-L1794).
5. Load back. load_back refuses runs shorter than load_back_threshold = 10 tokens or larger than the caller's memory quota (python/sglang/srt/mem_cache/unified_radix_cache.py:L1396-L1405), then hands the transfer to the L2 engine — which is unchanged from the earlier implementation. The H2D copy is issued per layer:

        with device_module.stream(self.host_to_device_stream):
            start_event.wait(self.host_to_device_stream)
            ack_start.record()
            for layer_id in range(layer_num):
                for transfer in transfers:
                    local_layer_id = (
                        transfer.layer_mapper(layer_id)
                        if transfer.layer_mapper is not None
                        else layer_id
                    )
                    # ...
                    transfer.host_pool.load_to_device_per_layer(
                        transfer.device_pool,
                        transfer.host_indices,
                        transfer.device_indices,
                        local_layer_id,
                        self.io_backend,
                        is_draft=transfer.is_draft,
                    )
                if on_layer_done is not None:
                    on_layer_done(layer_id)

That loop, plus the LayerDoneCounter it drives, is what turns a blocking PCIe copy — the trace's 8,192 tokens are exactly 1 GiB of KV, 16.8 ms at Figure 2's 64 GB/s — into something the forward pass can consume incrementally: layer 0's KV lands while layer 0 is still computing. The whole point of the per-layer decomposition is to hide the transfer behind the model.

First, the verification promised by §1.4. At a556f3f, grep -rn "swap_space" vllm/ returns zero hits — the field is gone from CacheConfig entirely. grep -rn "swap" vllm/v1/ returns 47 hits, and every one is either an unrelated identifier (swap_row in the block table, swap_states in logits processors) or belongs to vllm/v1/kv_offload/cpu/, where swap_blocks_batch is the name of a Triton kernel for GPU↔CPU block copies. There is no preemption-swap path in vLLM V1. Host memory is not a fallback for eviction; it is only ever a cache.

What exists instead is two separate offload stacks, reachable through the same KVConnectorBase_V1 factory that §1.6 describes for P/D:

vllm/v1/kv_offload/ — the general stack

An OffloadingManager runs in the scheduler and tracks addresses only; a separate OffloadingWorker moves data. Its primitive set is a cache protocol, not a swap protocol — lookup, prepare_load, touch, complete_load, prepare_store, complete_store (vllm/v1/kv_offload/base.py:L163-L184). Keys are make_offload_key(block_hash, group_idx) — the same content hashes §2.3 derives — so the host tier shares a namespace with the GPU prefix cache without sharing an allocator. Two specs are registered:

# Register various specs here.
OffloadingSpecFactory.register_spec(
    "CPUOffloadingSpec", "vllm.v1.kv_offload.cpu.spec", "CPUOffloadingSpec"
)
OffloadingSpecFactory.register_spec(
    "TieringOffloadingSpec",
    "vllm.v1.kv_offload.tiering.spec",
    "TieringOffloadingSpec",
)

CPUOffloadingSpec is a single host tier backed by a SharedOffloadRegion: an mmap'd, page-aligned region sized by a mandatory cpu_bytes_to_use, with lru or arc eviction (vllm/v1/kv_offload/cpu/policies/factory.py:L84-L90). Under TP, ranks take disjoint slots — except for a pure-MLA model where every rank's bytes are identical, in which case all share slot 0, dividing host footprint by the TP degree (vllm/v1/kv_offload/cpu/spec.py:L90-L110).

TieringOffloadingSpec subclasses it and adds L3 under the constraint its docstring states: "The CPU primary tier has direct GPU access and serves as the gateway for all GPU↔offload operations. Secondary tiers cannot directly access GPU memory and must transfer data through the primary tier" (vllm/v1/kv_offload/tiering/spec.py:L84-L95). Four secondary tiers are registered at vllm/v1/kv_offload/tiering/factory.py:L103-L125 — example, fs (thread-pooled filesystem), p2p (NIXL data plane, ZMQ control plane) and obj (object store) — plus an out-of-tree escape hatch taking a module_path.

vllm/v1/simple_kv_offload/ — the specialised stack

The simple stack answers a different question: what if the host tier were the same data structure as the GPU pool? SimpleCPUOffloadScheduler derives a CPU KVCacheConfig from the GPU one and builds a second full KVCacheCoordinator and BlockPool over host memory (vllm/v1/simple_kv_offload/manager.py:L94-L142), so host prefix matching uses the same find_longest_cache_hit, hash table and LRU free queue as the GPU pool (§2.2). Backends are exactly two, ("cpu", "disk"), both moving blocks with cuMemcpyBatchAsync; the disk backend uses O_DIRECT against 4096-aligned pinned staging buffers with separate store and load threads so "loads (latency-critical) never block behind stores (background work)" (vllm/v1/simple_kv_offload/disk_backend.py:L3-L8).

Its most instructive feature is a write policy SGLang lacks: lazy offload, which writes blocks to host only when they are near eviction, by walking the GPU free queue with a persistent cursor.

    def _prepare_lazy_store_specs(
        self,
    ) -> tuple[list[int], list[int], list[str]]:
        """Single-pass cursor walk: offload cached GPU blocks near eviction.

        Walks the GPU free queue from the cursor, counting blocks that are
        free-or-offloaded (safe for the allocator to evict). Stops when
        target_free blocks are covered or CPU capacity is reached.
        """
        gpu_pool = self._gpu_block_pool
        if gpu_pool is None or self._target_free <= 0:
            return [], [], []

        free_queue = gpu_pool.free_block_queue
        cpu_pool = self.cpu_block_pool
        num_cpu_free = cpu_pool.get_num_free_blocks()

This is SGLang's write_back without the drawback that deprecated it: the walk is amortised across steps by the cursor and bounded by _target_free, so it never runs synchronously inside an eviction. The honest comparison: vLLM's general stack is the more extensible — pluggable specs, tiers and eviction policies, all loadable out-of-tree — while SGLang's HiCache is the more integrated, because the host tier is a second address on every radix node rather than a parallel key-value namespace, which is what makes the per-layer overlapped load-back in Figure 3 possible at all. Enabling vLLM's is one pair of flags: kv_offloading_size in GiB and kv_offloading_backend, "native" or "lmcache" (vllm/config/cache.py:L210-L219).

Bandwidth says the host tier is 22× cheaper than recompute. That is necessary, not sufficient — the tier only earns its keep if requests actually hit in it, and hit rate is a residency question. Model a tier as LRU with capacity $C$ tokens fed by unique insertions at $\lambda$ tokens/s. An entry survives for roughly

Under a simplified unique-insertion LRU approximation, a 445632-token GPU tier at 2000 tokens/s gives 222.8 s. A 2x-capacity inclusive write-through host tier contains copies of many GPU entries, so its unique window is about 891264/2000=445.6 s, not (445632+891264)/2000=668.4 s. The additive result applies to an ideal exclusive hierarchy with disjoint contents. Real reuse distances and pinned entries determine hit rates.

So the host tier pays for exactly one thing: inter-turn gaps between 3.7 and 7.4 minutes under the inclusive write-through approximation. Below 3.7 minutes the GPU cache already hits and the host tier is pure write amplification; above 7.4 minutes in that simplified model both miss and you need L3 or nothing. That window is the shape of human multi-turn chat — read a 400-word answer, think, type, come back four minutes later — which is why multi-turn chat is the canonical win. The workloads it does not help:

The prefetch threshold encodes exactly this: SGLang refuses to fetch fewer than 256 tokens because the round-trip's fixed cost exceeds re-prefilling them. Both bail-outs in Figure 3 are that idea.

Read your own budget instead of trusting Figure 1. Start vLLM at debug level and grep four lines:

VLLM_LOGGING_LEVEL=DEBUG vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --max-model-len 8192 --max-num-batched-tokens 8192 2>&1 \
  | grep -E "init memory snapshot|Memory profiling takes|Available KV cache|kv-cache-memory="

Line one gives $O_{\text{non-torch}}$ and $C_{\text{total}}$; line two is MemoryProfilingResult.__repr__ — weights, torch peak increase, total consumed; line three is the residual; line four is the suggested explicit byte count. Then flip one thing at a time and watch which term moves: --enforce-eager should zero the graph term, VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 should leave the graph pool unaccounted (and inflate the KV pool by the same amount), and doubling --max-num-batched-tokens should move the activation peak and potentially graph/workspace/backend allocation terms too.

For SGLang, the equivalent is a single startup line — max_total_num_tokens=..., chunked_prefill_size=..., max_running_requests=... (python/sglang/srt/managers/scheduler.py:L1094-L1099) — plus, with --enable-hierarchical-cache, the host-pool allocation line Allocating kv hierarchical KV host pool: N tokens, X GB host memory. Compare max_total_num_tokens against vLLM's block count × 16 on the same model and card; a large gap is the estimate-versus-profile difference made visible.

• vLLM PR #11743, "Memory profiling rework" — the origin of the MemorySnapshot / memory_profiling split and the torch-versus-non-torch category model quoted in §4.
• vLLM RFC #19854, "KV cache offloading framework" — the design discussion behind vllm/v1/kv_offload/'s manager/worker/spec split. Read it before writing an out-of-tree OffloadingSpec.
• SGLang PR #2693, "Hierarchical caching for SGLang" — the original HiRadixCache, including the write-policy debate that produced write_through_selective. Read it as history, then read SGLang issue #20415, the Unified Radix Cache tracker named in python/sglang/srt/mem_cache/README.md:L38-L45, for the merge that absorbed it.
• SGLang HF3FS backend notes and the Mooncake paper (arXiv:2407.00079) for what a production L3 actually has to solve — Mooncake is the reference design for a disaggregated KV store and is a registered backend in both engines.
• LMCache (repo) — the third-party KV layer that both engines expose as a first-class backend (kv_offloading_backend="lmcache" in vLLM, lmc_radix_cache.py in SGLang). Useful as a contrast: it owns the whole hierarchy rather than plugging into the engine's.
• NVIDIA H100 datasheet for the 3.35 TB/s HBM3 figure, and the PCI-SIG PCIe 5.0 base specification for the 64 GB/s-per-direction x16 link rate. Both are ceilings; treat any number derived from them as optimistic.