Front door

The source-reading standard is to ground implementation claims in code at the stated SHA. A source-pinned label identifies that provenance; it is not a hardware-tested or fully recertified status. The standard is:

• Path and line range, always. vllm/v1/core/sched/scheduler.py:L412-L458, never "the scheduler module".
• Real snippets. Quoted source, trimmed, with elisions marked # .... Anything invented is labelled PSEUDOCODE in the code header.
• Traces, not summaries. Each major subsystem gets a call path: entry point, every hop, and the point where GPU work is launched — with the function names in order, so you can follow along in an editor.
• Gaps are flagged, not filled. Where the implementation could not be located, the chapter says so in an Unverified callout and names the likely location. A flagged gap is a feature; an invented API poisons the book.
• Numbers have provenance. Every latency, throughput, or memory figure is either cited to an external source, derived from stated assumptions, illustrative, or locally measured. These categories are not interchangeable; CPU arithmetic checks do not establish GPU throughput.
• Volatile behaviour is version-stamped. Flag names, defaults, and file layouts move weekly in both projects. Anything that could rot is written as "as of <sha>…".

The book is linear. Each part assumes the ones before it, and there are no forward references to undefined concepts. If you read it in order, nothing will be used before it is explained.

The full chapter list, with the source files each chapter is expected to cite and its current drafting status, is on the Outline page.

What you need to know

• Python, comfortably. You will read a lot of it.
• Transformers at the level of "I have read The Illustrated Transformer and implemented attention once." Chapter 00-02 re-derives what it needs.
• Enough linear algebra to follow a shape argument. No proofs beyond one page.
• No CUDA. Part 0 teaches the GPU model from scratch, and Part 3 teaches kernels from there.

Choose a route and verify prerequisites

Review linear algebra, PyTorch, and the Transformer course as needed. Before Part 2, derive bytes/token and distinguish a sampled token from a cached token. Before Part 6, normalize a categorical distribution and explain conditional probability. Before Part 10, distinguish mean latency, a percentile and offered load.

• Reading: no engine or GPU is required.
• CPU arithmetic: download the reference checks; Python 3.11+ is sufficient. These checks are not GPU benchmarks.
• Single GPU: verify model access, exact model/tokenizer commit, trained context limit, disk space, weight/KV/activation budget, driver, CUDA and kernel support.
• Multiple GPUs: additionally record topology, rank/device placement, negotiated link bandwidth, ports and process cleanup.

For an engine run, record the full engine SHA, Python/PyTorch/CUDA/driver versions, dependency lock, GPU model/count, backend and model/tokenizer revisions. Run the engine's own installation instructions from that checkout, inspect its --help, then poll readiness and complete one short request before a sweep. Gated weights require an authorized account; use synthetic prompts and avoid logging credentials. A model-name alias or engine SHA alone is not a reproducible environment.

What you need to run the labs

Roughly half the labs run on any CUDA GPU with 24 GB, using a 7–8B model. The rest want 80 GB (H100/A100) or two of them for the tensor-parallel and disaggregation labs. Every lab states its hardware floor at the top and degrades to a smaller model where it can.

mkdir -p ~/Documents/other_git_repos && cd ~/Documents/other_git_repos

git clone https://github.com/vllm-project/vllm.git
git -C vllm checkout a556f3fccb701e5618d84d547ff454c56a1bfdfb

git clone https://github.com/sgl-project/sglang.git
git -C sglang checkout 7d893255c359bb8ab74d2870c8ac865fb57230d6

You do not need to build either project to read the book — every citation is a path into the source tree. Cloning does not install a compatible engine. GPU experiments require a separately built environment for the pinned SHA and its dependencies; these snapshots have not been installed or GPU-tested in this checkout. Use the preflight below before interpreting any command as a validated recipe.

Every chapter follows the same shape, so you can skim to the part you need:

Reference pages

• Outline — every chapter, its sources, its status.
• Formulas — KV cache size, arithmetic intensity, roofline, acceptance rate, memory budget. One page, all symbols defined.
• Glossary — every term of art, one line each, linked to the chapter that defines it.
• Labs — the runnable exercises, with hardware requirements.
• Sources — pinned SHAs, clone dates, bibliography.
• Progress — the session log, open questions, and every outstanding Unverified flag.

This repository imports complete inference HTML pages. Edit content/courses/inference/html/, not the generated _site/ tree. The shared site builder supplies the reader navigation, search, diagram controls and math rendering. The original project's fragment builder is not shipped here.

python3 -m pip install -r site/requirements.txt
python3 site/build.py
python3 site/test_build.py
python3 site/test_presentation.py
python3 content/courses/inference/html/labs/cpu_checks.py