17 min read15 sections

The Inference Engineering Course

Outline

Every chapter in the book, with the source files it is expected to cite and its current status. This is an authored imported catalog in content/courses/inference/html/OUTLINE.html. Source-pinned labels describe citation provenance, not executed code or GPU measurements; see the evidence policy.

Chapters

Parts

Labs

Verified

Draft

§1

Part 0 — Foundations

The physics of the problem: what the machine is actually doing, what it is bounded by, and in what number format.

Ch.	Title	Summary	Primary sources	Status
`00-01`	Inference is not training	The shape of the problem: no backward pass, no gradients, a latency SLO instead of a loss curve, and a batch you do not control. Why every training intuition about GPU utilisation misleads you here.	`vllm/entrypoints/llm.py` `python/sglang/srt/entrypoints/engine.py` `docs/design/`	SOURCE PINNED
`00-02`	The decode loop and why the KV cache exists	The autoregressive loop written out; the derivation that makes the cache mandatory (per step, attention drops O(n²)→O(n) but the projections drop O(n)→O(1), which is where the real saving is); and exactly what is and is not cacheable.	`vllm/v1/worker/gpu_model_runner.py` `python/sglang/srt/model_executor/forward_batch_info.py`	SOURCE PINNED
`00-03`	GPU architecture for inference engineers	SMs, warps, tensor cores, HBM vs L2 vs SRAM, occupancy, and kernel-launch overhead — only the parts that change serving decisions.	`csrc/libtorch_stable/attention/merge_attn_states.cu` `csrc/cuda_compat.h` `csrc/cuda_utils.h`	SOURCE PINNED
`00-04`	Arithmetic intensity and the roofline	Bytes moved vs FLOPs done. Derive the intensity of a prefill GEMM and a decode GEMV, place both on an H100 roofline, and predict which optimisations can possibly help.	`benchmarks/kernels/` `python/sglang/benchmark/one_batch.py`	SOURCE PINNED
`00-05`	Numerics: FP32 to INT4, and why bf16 won	Dynamic range vs precision, accumulation dtype, FP8 E4M3/E5M2, INT8/INT4, MXFP4. Where each format is legal and where it silently corrupts.	`vllm/model_executor/layers/quantization/` `python/sglang/srt/layers/quantization/`	SOURCE PINNED

§2

Part 1 — The core serving loop

Two workloads sharing one GPU, and the scheduler that decides who runs each step.

Ch.	Title	Summary	Primary sources	Status
`01-01`	Prefill and decode are two different computers	One is a compute-bound GEMM, the other a memory-bound GEMV. Every scheduling decision in both engines follows from this split.	`vllm/v1/core/sched/scheduler.py` `python/sglang/srt/managers/scheduler.py`	SOURCE PINNED
`01-02`	TTFT, TPOT, ITL, E2E, goodput	Precise definitions, the percentile you should actually alert on, and how to write an SLO that a capacity plan can be built from.	`vllm/benchmarks/serve.py` `python/sglang/benchmark/serving.py` `vllm/v1/metrics/`	SOURCE PINNED
`01-03`	Static, dynamic, and continuous batching	Why per-request padding wastes most of the GPU, what Orca changed, and how iteration-level scheduling is implemented in both engines.	`vllm/v1/core/sched/scheduler.py` `python/sglang/srt/managers/schedule_batch.py`	SOURCE PINNED
`01-04`	The scheduler: states, admission, preemption	The request state machine, the waiting/running queues, the watermark that triggers preemption, and recompute vs swap as a cost decision.	`vllm/v1/core/sched/scheduler.py` `vllm/v1/core/sched/request_queue.py` `vllm/v1/request.py` `python/sglang/srt/managers/schedule_policy.py`	SOURCE PINNED
`01-05`	Chunked prefill	Slicing a long prefill across iterations so decoding requests are not starved behind it; the token-budget arithmetic, and the ITL/TTFT trade it makes.	`vllm/v1/core/sched/scheduler.py` `vllm/config/scheduler.py` `python/sglang/srt/managers/schedule_policy.py`	SOURCE PINNED
`01-06`	Prefill–decode disaggregation	Splitting prefill and decode onto separate machines: why the two phases want different hardware, the handshake and request state machine, and the KV transport backends (Mooncake, NIXL, MORI) that move the cache between them.	`python/sglang/srt/disaggregation/prefill.py` `python/sglang/srt/disaggregation/decode.py` `python/sglang/srt/disaggregation/mooncake/` `python/sglang/srt/disaggregation/nixl/` `vllm/distributed/kv_transfer/` `vllm/v1/core/sched/scheduler.py`	SOURCE PINNED

§3

Part 2 — Memory and the KV cache

The KV cache is the scarce resource. Everything about capacity, and most things about latency, reduce to how you allocate it.

Ch.	Title	Summary	Primary sources	Status
`02-01`	The KV cache size formula	Derive it from layer shapes, then work it for Llama-3-8B, Llama-3-70B, Qwen, and DeepSeek-V3. How many concurrent 8k-token requests fit on one H100?	`vllm/v1/kv_cache_interface.py` `vllm/v1/core/kv_cache_utils.py` `python/sglang/srt/mem_cache/memory_pool.py`	SOURCE PINNED
`02-02`	PagedAttention: fragmentation, blocks, block tables	Why contiguous allocation wastes 60–80%, how paging fixes it, and the block pool / block table / copy-on-write machinery in code.	`vllm/v1/core/block_pool.py` `vllm/v1/core/kv_cache_manager.py` `vllm/v1/worker/block_table.py` `python/sglang/srt/mem_cache/allocator/` `python/sglang/srt/mem_cache/memory_pool.py`	SOURCE PINNED
`02-03`	Prefix caching by hash (vLLM)	Block hashing, the cached-block lookup, what invalidates a hit, and why the granularity is a block rather than a token.	`vllm/v1/core/kv_cache_utils.py` `vllm/v1/core/block_pool.py` `benchmarks/benchmark_prefix_caching.py`	SOURCE PINNED
`02-04`	RadixAttention: the tree (SGLang)	The radix tree, prefix matching, LRU eviction over a tree, and why a tree beats a hash map for multi-turn and few-shot workloads — the central design divergence between the two engines.	`python/sglang/srt/mem_cache/radix_cache.py` `python/sglang/srt/mem_cache/base_prefix_cache.py` `python/sglang/srt/managers/schedule_policy.py`	SOURCE PINNED
`02-05`	KV cache quantization	FP8 and INT8 KV, per-tensor vs per-head vs per-token scales, where dequant happens inside the attention kernel, and the newer rotated schemes (TurboQuant's Hadamard-rotated Lloyd-Max, INT4 per-token-head). Leads with the point that KV quantization is a capacity lever, not a latency one.	`vllm/model_executor/layers/quantization/kv_cache.py` `vllm/model_executor/layers/quantization/turboquant/` `vllm/v1/attention/backends/turboquant_attn.py` `vllm/config/cache.py` `python/sglang/srt/mem_cache/kv_cache_dtype.py`	SOURCE PINNED
`02-06`	Offload, hierarchical cache, and budgeting VRAM	CPU/disk tiers (HiCache, kv_offload), when a PCIe round-trip beats recompute, and how to actually reason about gpu_memory_utilization.	`vllm/v1/kv_offload/` `python/sglang/srt/mem_cache/hiradix_cache.py` `python/sglang/srt/mem_cache/hicache_storage.py`	SOURCE PINNED

§4

Part 3 — Attention kernels

Where the GPU work happens. From the online-softmax recurrence to the kernel signature you can read in the repo.

Ch.	Title	Summary	Primary sources	Status
`03-01`	Naive attention and the online softmax	Count the HBM traffic of textbook attention, then derive the running-max/running-sum recurrence and prove it equals standard softmax.	`vllm/v1/attention/backends/flash_attn.py` `python/sglang/srt/layers/attention/`	SOURCE PINNED
`03-02`	FlashAttention 1 → 2 → 3	Tiling, recomputation, work partitioning, warp specialisation, and Hopper async/FP8. Why doing more FLOPs is faster.	`vllm/v1/attention/backends/flash_attn.py` `vllm/vllm_flash_attn/` `python/sglang/srt/layers/attention/`	SOURCE PINNED
`03-03`	Decode attention: FlashDecoding, split-K, paged kernels	Batch-1 attention has no query-axis parallelism to exploit. Split the KV instead and combine partial softmaxes — plus why vLLM deleted its bespoke PagedAttention kernel (PR #47361) and now dispatches to general paged/varlen kernels.	`vllm/v1/attention/backends/triton_attn.py` `csrc/rocm/attention.cu` `csrc/libtorch_stable/attention/merge_attn_states.cu` `python/sglang/srt/layers/attention/`	SOURCE PINNED
`03-04`	FlashInfer and the attention-backend abstraction	What a backend must provide, how metadata is built once per step and reused, how each engine selects a backend at startup — including FlashAttention 4, which defaults on Blackwell — and why vLLM degrades where SGLang refuses.	`vllm/v1/attention/backend.py` `vllm/v1/attention/selector.py` `vllm/v1/attention/backends/flashinfer.py` `python/sglang/srt/layers/attention/`	SOURCE PINNED
`03-05`	MHA → MQA → GQA → MLA	The KV footprint of each, with the arithmetic. Why GQA is the default and why MLA is a different kind of trade.	`vllm/v1/attention/backends/mla/` `vllm/model_executor/layers/` `python/sglang/srt/layers/radix_attention.py`	SOURCE PINNED
`03-06`	RoPE, scaling, sliding window, attention sinks	How position enters at inference, what RoPE scaling and YaRN change at serving time, and how windowed attention interacts with the block allocator.	`vllm/model_executor/layers/rotary_embedding/` `python/sglang/srt/layers/rotary_embedding/`	SOURCE PINNED

§5

Part 4 — Quantization

Fewer bits per weight is not automatically faster. When it is, why, and what it costs in accuracy.

Ch.	Title	Summary	Primary sources	Status
`04-01`	The design space: PTQ/QAT, weight-only, W+A, KV	The axes that matter, what each one buys, and the decision tree for picking a scheme from a workload description.	`vllm/model_executor/layers/quantization/` `python/sglang/srt/layers/quantization/`	SOURCE PINNED
`04-02`	GPTQ, AWQ, SmoothQuant, and calibration	What each algorithm actually optimises, why activation outliers are the hard part, and how calibration data choice shows up in the eval.	`vllm/model_executor/layers/quantization/` `csrc/quantization/`	SOURCE PINNED
`04-03`	Formats and kernels: FP8, INT8, INT4, Marlin, Machete	The hardware instructions behind each format and the GEMM kernels that make low-bit weights fast instead of slow. Marlin and Machete are kernel backends selected inside AutoGPTQConfig/AutoAWQConfig, not separate methods; bitsandbytes and gguf are gone from vLLM and live only in SGLang.	`csrc/libtorch_stable/quantization/marlin/` `csrc/libtorch_stable/quantization/machete/` `vllm/model_executor/kernels/linear/` `python/sglang/srt/layers/quantization/fp8_utils.py`	SOURCE PINNED
`04-04`	When quantization pays — and when it does not	A latency win at low batch, a capacity win at high batch, a loss in the middle. Plus how to measure degradation without fooling yourself.	`benchmarks/kernels/` `python/sglang/benchmark/one_batch.py`	SOURCE PINNED

§6

Part 5 — Parallelism and distributed inference

When the model does not fit, or one GPU is not fast enough. Which axis to split, and what each split costs in collectives.

Ch.	Title	Summary	Primary sources	Status
`05-01`	Tensor parallelism	Column-parallel then row-parallel, why that ordering puts one all-reduce per projection pair, usually two per Transformer block, and how heads are split for attention.	`vllm/distributed/` `vllm/model_executor/layers/linear.py` `python/sglang/srt/layers/linear.py`	SOURCE PINNED
`05-02`	Pipeline parallelism and bubbles	Stage partitioning, micro-batching, where the bubble comes from in a decode-heavy workload, and when PP is the wrong tool.	`vllm/distributed/` `python/sglang/srt/managers/scheduler_pp_mixin.py`	SOURCE PINNED
`05-03`	Data parallelism, attention-DP, expert parallelism	Replica routing, DP attention with TP MoE, expert placement, and the load-balancing problem that EPLB exists to solve.	`vllm/v1/engine/coordinator.py` `python/sglang/srt/layers/dp_attention.py` `python/sglang/srt/eplb/`	SOURCE PINNED
`05-04`	Collectives, topology, custom all-reduce	NVLink vs PCIe vs InfiniBand, why NCCL is not always the fastest path for small messages, and how comm overlaps compute.	`vllm/distributed/device_communicators/` `csrc/custom_all_reduce.cuh` `python/sglang/srt/distributed/`	SOURCE PINNED
`05-05`	Worker orchestration and multi-node serving	How each engine launches ranks, broadcasts work, and survives a dead worker. Multiproc vs Ray in vLLM; the SGLang process model.	`vllm/v1/executor/multiproc_executor.py` `vllm/v1/executor/ray_executor.py` `python/sglang/srt/managers/tp_worker.py` `python/sglang/srt/managers/data_parallel_controller.py`	SOURCE PINNED

§7

Part 6 — Decoding algorithms

What happens after the logits. Sampling, speculation, and constraints — all of it batched, all of it on-GPU.

Ch.	Title	Summary	Primary sources	Status
`06-01`	Sampling on the GPU	Temperature, top-k, top-p, min-p, penalties — done for a heterogeneous batch without a host round-trip. And why beam search left serving.	`vllm/v1/sample/sampler.py` `vllm/v1/sample/ops/` `python/sglang/srt/layers/sampler.py`	SOURCE PINNED
`06-02`	Speculative decoding and the correctness proof	Draft k, verify in one forward pass, accept with the rejection-sampling rule. The proof that the output distribution is unchanged.	`vllm/v1/sample/rejection_sampler.py` `vllm/v1/spec_decode/` `python/sglang/srt/speculative/`	SOURCE PINNED
`06-03`	Draft sources: draft models, n-gram, suffix decoding	Where a cheap guess comes from when you do not want a second model. Prompt lookup, n-gram, and suffix automata.	`vllm/v1/spec_decode/ngram_proposer.py` `vllm/v1/spec_decode/suffix_decoding.py` `python/sglang/srt/speculative/ngram_worker.py`	SOURCE PINNED
`06-04`	Medusa, EAGLE-1/2/3, MTP, and tree attention	Drafting from the target model's own hidden states, and verifying a tree instead of a chain — including the finding that vLLM's EAGLE still produces a chain at this SHA, so tree attention is SGLang-only in practice.	`vllm/v1/spec_decode/llm_base_proposer.py` `vllm/model_executor/models/llama_eagle3.py` `vllm/model_executor/models/medusa.py` `python/sglang/srt/speculative/eagle_utils.py` `python/sglang/kernels/ops/speculative/spec_tree.py`	SOURCE PINNED
`06-05`	Structured and constrained decoding	FSMs and CFGs compiled over the token vocabulary, xgrammar/outlines/llguidance, the host-side cost that actually dominates, and jump-forward decoding — taught in full and then shown to be dead code at both pinned SHAs.	`vllm/v1/structured_output/` `python/sglang/srt/constrained/` `python/sglang/srt/constrained/outlines_jump_forward.py`	SOURCE PINNED
`06-06`	The speculative decoding zoo	Everything past EAGLE that ships in the pinned trees: DFlash, DSpark, frozen-KV MTP, multi-layer EAGLE drafts, standalone draft workers, suffix/ngram CPU proposers, and adaptive speculation that tunes draft length at runtime. What each changes and when to reach for it.	`python/sglang/srt/speculative/dflash_worker_v2.py` `python/sglang/srt/speculative/dspark_components/` `python/sglang/srt/speculative/frozen_kv_mtp_worker_v2.py` `python/sglang/srt/speculative/multi_layer_eagle_worker_v2.py` `python/sglang/srt/speculative/adaptive_spec_params.py` `python/sglang/srt/speculative/spec_registry.py` `vllm/v1/spec_decode/dflash.py` `vllm/v1/spec_decode/draft_model.py`	SOURCE PINNED

§8

Part 7 — Architectures that change the inference story

Model designs whose serving characteristics differ enough that the engine has to be built differently.

Ch.	Title	Summary	Primary sources	Status
`07-01`	Mixture of Experts	Routing, grouped GEMM, capacity factor, load imbalance, and why MoE turns a compute problem into a memory-and-communication problem.	`vllm/model_executor/layers/fused_moe/` `csrc/libtorch_stable/moe/moe_align_sum_kernels.cu` `python/sglang/srt/layers/moe/` `python/sglang/srt/models/deepseek_v2.py`	SOURCE PINNED
`07-02`	MLA in full detail	Low-rank KV compression, the absorbed-weight trick, why decode and prefill want different formulations, and what it does to the cache budget.	`vllm/v1/attention/backends/mla/` `python/sglang/srt/layers/attention/`	SOURCE PINNED
`07-03`	Hybrid SSM / Mamba models and non-KV state	Constant-size recurrent state instead of a growing cache. What that breaks: prefix caching, preemption, and the block allocator.	`vllm/v1/attention/backends/mamba2_attn.py` `vllm/v1/worker/mamba_utils.py` `python/sglang/srt/mem_cache/mamba_radix_cache.py`	SOURCE PINNED
`07-04`	Multimodal serving	The vision encoder as a second model, image-token expansion, encoder output caching, and what a prefix cache means when the prefix is an image.	`vllm/multimodal/` `vllm/v1/core/encoder_cache_manager.py` `python/sglang/srt/multimodal/` `python/sglang/srt/mem_cache/multimodal_cache.py`	SOURCE PINNED
`07-05`	LoRA serving, embeddings, rerankers, pooling	Batching requests that use different adapters in one kernel, and the non-generative models a serving stack still has to host.	`vllm/lora/` `vllm/v1/pool/` `python/sglang/srt/lora/` `python/sglang/srt/layers/pooler.py`	SOURCE PINNED

§9

Part 8 — Compilation and runtime

Getting from a PyTorch module to a launch sequence that does not waste the GPU between kernels.

Ch.	Title	Summary	Primary sources	Status
`08-01`	CUDA graphs and kernel-launch overhead	Why a decode step can be dominated by launch latency, what a graph captures, and the constraints capture imposes on the model runner.	`vllm/v1/cudagraph_dispatcher.py` `vllm/compilation/` `python/sglang/srt/model_executor/cuda_graph_config.py`	SOURCE PINNED
`08-02`	torch.compile, Inductor, piecewise capture	What gets fused, what forces a graph break, and why both engines compile in pieces rather than end to end.	`vllm/compilation/` `python/sglang/srt/compilation/`	SOURCE PINNED
`08-03`	Writing a Triton kernel; CUTLASS at a glance	Write, tune, and benchmark a real fused kernel; then read enough CUTLASS to understand what the FP8 GEMMs in csrc are doing.	`vllm/model_executor/layers/` `csrc/cutlass_extensions/` `python/sglang/kernels/`	SOURCE PINNED
`08-04`	Weight loading, sharded loaders, adding a model	How a checkpoint becomes sharded device tensors, and a full walkthrough of adding a model to vLLM by reading a real model file.	`vllm/model_executor/model_loader/` `vllm/models/` `python/sglang/srt/model_loader/` `python/sglang/srt/models/`	SOURCE PINNED

§10

Part 9 — The serving system around the engine

The engine is maybe half the product. This is the other half: the API, the stream, the router, and the pager.

Ch.	Title	Summary	Primary sources	Status
`09-01`	The OpenAI-compatible API surface	What the endpoints actually accept, where the compatibility is approximate, and how sampling params map onto engine internals.	`vllm/entrypoints/launchers/api_server/routers.py` `vllm/entrypoints/openai/chat_completion/serving.py` `python/sglang/srt/entrypoints/http_server.py` `python/sglang/srt/entrypoints/openai/`	SOURCE PINNED
`09-02`	Tokenization and incremental detokenization	Streaming detokenisation, the partial-UTF-8 problem, stop-string matching across chunk boundaries, and why this is a top source of user-visible bugs.	`vllm/v1/engine/detokenizer.py` `vllm/tokenizers/detokenizer_utils.py` `vllm/tokenizers/hf.py` `python/sglang/srt/managers/detokenizer_manager.py` `python/sglang/srt/managers/async_dynamic_batch_tokenizer.py`	SOURCE PINNED
`09-03`	Streaming and the end-to-end request lifecycle	HTTP in, tokens out: every hop, every queue, every process boundary, in both engines, with the file and function at each step.	`vllm/entrypoints/launchers/api_server/entry.py` `vllm/entrypoints/openai/chat_completion/serving.py` `vllm/v1/engine/async_llm.py` `vllm/v1/engine/core_client.py` `python/sglang/srt/managers/tokenizer_manager.py` `python/sglang/srt/managers/scheduler.py`	SOURCE PINNED
`09-04`	Routers, KV-aware balancing, session affinity	Round-robin throws away your prefix cache. Cache-aware routing, the SGLang Rust gateway, and autoscaling with cold starts.	`sgl-model-gateway/` `python/sglang/srt/managers/disagg_service.py`	SOURCE PINNED
`09-05`	Observability, failure modes, recovery	The Prometheus metrics worth alerting on, what a NCCL hang and an OOM look like in the logs, and how each engine recovers a dead worker.	`vllm/v1/metrics/` `vllm/v1/fault_tolerance/` `python/sglang/srt/observability/`	SOURCE PINNED

§11

Part 10 — Benchmarking and performance engineering

Most published inference numbers are not comparable. This part is about producing numbers that are.

Ch.	Title	Summary	Primary sources	Status
`10-01`	Workload characterization	Input/output length distributions, arrival process, prefix-sharing rate. These dominate every result; measure them before tuning anything.	`benchmarks/` `python/sglang/benchmark/serving.py`	SOURCE PINNED
`10-02`	The benchmark harnesses in both repos	What vllm/benchmarks/serve.py and python/sglang/benchmark/serving.py actually do (the old bench_serving entry points are now shims), the datasets they ship, and each dataset's bias.	`vllm/benchmarks/serve.py` `vllm/benchmarks/throughput.py` `python/sglang/benchmark/serving.py` `python/sglang/benchmark/one_batch.py`	SOURCE PINNED
`10-03`	Running a benchmark that is not a lie	Warmup, saturation, open vs closed loop, request-rate control, and the reporting discipline that makes two benchmark runs actually comparable.	`vllm/benchmarks/serve.py` `vllm/benchmarks/sweep/server.py` `vllm/v1/worker/gpu_worker.py` `python/sglang/benchmark/serving.py` `python/sglang/srt/managers/schedule_policy.py`	SOURCE PINNED
`10-04`	Batch invariance and reproducible inference	The same prompt yields different tokens depending on what else is in the batch. Where the nondeterminism comes from (reduction order, split-K, atomics, kernel selection), what it costs to eliminate, and the batch-invariant op sets both repos now ship.	`benchmarks/benchmark_batch_invariance.py` `python/sglang/srt/batch_invariant_ops/` `vllm/model_executor/layers/batch_invariant.py`	SOURCE PINNED
`10-05`	Profiling, roofline of a decode step, capacity planning	torch.profiler and Nsight on a live server, reading the trace, placing the measured step on the roofline, then converting it to cost per million tokens.	`vllm/profiler/` `python/sglang/profiler.py`	SOURCE PINNED

§12

Part 11 — vLLM deep dive

Enough of the repo to navigate it without a map, change it, and defend the change in review.

Ch.	Title	Summary	Primary sources	Status
`11-01`	Repo map, and what V1 changed	Directory by directory, then the rewrites: V0 to V1 as history (no V0 code remains), and the V1 to V2 model-runner rewrite — which is already the default for every dense model, not the opt-in its README implies.	`vllm/` `vllm/v1/` `docs/design/`	SOURCE PINNED
`11-02`	AsyncLLM → EngineCore → Executor → Worker	The process boundaries, the ZMQ hops, and who owns which piece of state.	`vllm/v1/engine/async_llm.py` `vllm/v1/engine/core.py` `vllm/v1/engine/core_client.py` `vllm/v1/executor/abstract.py`	SOURCE PINNED
`11-03`	The V1 scheduler, in code	A line-by-line read of schedule(), the token budget, the KV allocation call, and how SchedulerOutput reaches the worker.	`vllm/v1/core/sched/scheduler.py` `vllm/v1/core/sched/output.py` `vllm/v1/core/kv_cache_manager.py`	SOURCE PINNED
`11-04`	GPUModelRunner, the input batch, the attention backend	How a SchedulerOutput becomes tensors, where persistent batch state lives, and how attention metadata is built once per step — across BOTH the legacy 8k-line gpu_model_runner.py and the 2k-line Model Runner V2, already the default for dense models under vllm/v1/worker/gpu/.	`vllm/v1/worker/gpu_model_runner.py` `vllm/v1/worker/gpu/model_runner.py` `vllm/v1/worker/gpu/README.md` `vllm/v1/worker/gpu_input_batch.py` `vllm/v1/attention/backend.py`	SOURCE PINNED
`11-05`	Extension points	Adding a model, a kernel, a sampling parameter, a scheduler policy, or a plugin — with the tests you are expected to write.	`vllm/plugins/` `vllm/model_executor/models/registry.py` `vllm/sampling_params.py`	SOURCE PINNED

§13

Part 12 — SGLang deep dive

The same treatment for the other engine, with attention to where its choices diverge and why.

Ch.	Title	Summary	Primary sources	Status
`12-01`	Repo map and process architecture	python/sglang/srt directory by directory, the multi-process layout and its ZMQ topology, and the parallelism axes the outline did not anticipate — moe_dp_size, attn_cp_size, and the dynamic-weight DP path under layers/moe/dwdp/.	`python/sglang/srt/` `python/sglang/srt/entrypoints/engine.py` `python/sglang/srt/server_args.py`	SOURCE PINNED
`12-02`	TokenizerManager → Scheduler → TpModelWorker → ModelRunner	The full call path with file and line at each hop, including the overlap between tokenisation and the model step.	`python/sglang/srt/managers/tokenizer_manager.py` `python/sglang/srt/managers/scheduler.py` `python/sglang/srt/managers/tp_worker.py` `python/sglang/srt/model_executor/model_runner.py`	SOURCE PINNED
`12-03`	RadixAttention and the memory pools, in code	The tree node, match_prefix, insert, evict, and how the pools underneath hand out token slots.	`python/sglang/srt/mem_cache/radix_cache.py` `python/sglang/srt/mem_cache/memory_pool.py` `python/sglang/srt/mem_cache/allocator/`	SOURCE PINNED
`12-04`	Frontend DSL, grammar backends, router, extension points	The programming model SGLang is named after, how constrained generation plugs into the scheduler, and where to hook your own code.	`python/sglang/lang/` `python/sglang/srt/constrained/` `sgl-model-gateway/`	SOURCE PINNED

§14

Part 13 — Comparison, frontier, and practice

Putting the two engines side by side, then looking at what is still unsolved.

Ch.	Title	Summary	Primary sources	Status
`13-01`	Head-to-head design comparison	One row per design decision; columns for vLLM, SGLang, TensorRT-LLM, TGI, llama.cpp; a final column for why each chose what it chose.	`vllm/` `python/sglang/srt/`	SOURCE PINNED
`13-02`	Choosing an engine for a workload	A decision procedure driven by workload shape, model family, hardware, and team capacity — not by benchmark screenshots.	—	SOURCE PINNED
`13-03`	Open problems	Disaggregation at scale, the KV cache as a distributed system, heterogeneous hardware, ultra-long context, on-device.	—	SOURCE PINNED
`13-04`	What to build next	A project ladder from a weekend to a quarter, and a curated reading list of the papers, PRs, and talks that repay the time.	—	SOURCE PINNED

§15

Labs

Runnable labs. Each lab has a README.html and a run.py, and is referenced from the chapter it belongs to.

Lab	What you measure	Chapter	Status
`01-measure-your-gpu-roofline` Measure your GPU's roofline	Sweep matmul shapes to find achieved TFLOP/s and HBM GB/s, then plot the ridge point for your actual card.	`00-04`	DRAFT
`02-kv-cache-sizing` KV cache sizing calculator	Compute the cache footprint for a model config, then verify it against the number the engine reports at startup.	`02-01`	DRAFT
`03-continuous-batching-effects` Continuous batching under load	Sweep concurrency and watch TTFT, TPOT, and throughput trade against each other; find the knee.	`01-03`	DRAFT
`04-prefix-cache-hit-rate` Prefix cache hit rate	Run a shared-prefix workload against both engines and read the hit-rate metric; break the cache and watch it fall.	`02-03`	DRAFT
`05-chunked-prefill` Chunked prefill and ITL jitter	Measure inter-token-latency spikes with and without chunked prefill under a mixed long/short workload.	`01-05`	DRAFT
`06-quantization-tradeoff` Quantization: latency, capacity, accuracy	Same model in bf16 and FP8/INT4: latency at batch 1, max concurrency, and a small eval to price the accuracy.	`04-04`	DRAFT
`07-tp-scaling` Tensor-parallel scaling	TP=1,2,4,8 on one node: measure the speedup, then attribute the gap to collectives with a profile.	`05-01`	DRAFT
`08-spec-decode-acceptance` Speculative decoding acceptance rate	Measure acceptance length across workloads and find the point where speculation becomes a net loss.	`06-02`	DRAFT
`09-structured-decoding-overhead` Structured decoding overhead	Price the grammar mask: throughput with and without a JSON schema, and how it scales with batch size.	`06-05`	DRAFT
`10-profile-a-decode-step` Profile a decode step	Capture a torch.profiler trace of one decode iteration and account for every microsecond in it.	`10-05`	DRAFT
`11-end-to-end-benchmark` An end-to-end benchmark you can defend	Run both engines on identical hardware and workload, with warmup, saturation, and open-loop arrivals; report goodput.	`10-03`	DRAFT
`12-read-a-request-through-the-code` Trace one request through the source	Instrument both engines with logging at every hop and watch a single request traverse the whole stack.	`09-03`	DRAFT
`13-batch-invariance` Reproducibility under batch invariance	Send one prompt at several batch sizes with temperature 0 and diff the outputs; then re-run with the determinism flag and measure what it costs. Uses the engines' own correctness harnesses, not the performance benchmark.	`10-04`	DRAFT

Outline

Part 0 — Foundations

Part 1 — The core serving loop

Part 2 — Memory and the KV cache

Part 3 — Attention kernels

Part 4 — Quantization

Part 5 — Parallelism and distributed inference

Part 6 — Decoding algorithms

Part 7 — Architectures that change the inference story

Part 8 — Compilation and runtime

Part 9 — The serving system around the engine

Part 10 — Benchmarking and performance engineering

Part 11 — vLLM deep dive

Part 12 — SGLang deep dive

Part 13 — Comparison, frontier, and practice

Labs

Explore the library

Reading preferences