Labs
Thirteen runnable labs. Each is a directory under labs/ with a README page
and a run.py, and each is referenced from the chapter it belongs to. The point of a
lab is to replace a claim in the book with a number you measured yourself.
Labs 1–6, 9–10 and 13 run on a single 24 GB GPU with a 7–8B model. Lab 7 measures TP on 2–8 GPUs (TP=1 is its baseline); PCIe is a valid contrasting topology. Lab 11 can use one 80 GB GPU for its stated workload. Lab 12 permits a smaller model on one GPU and has offline trace modes. Lab 8 is split: measuring acceptance length needs only 24 GB, but finding the batch size where speculation stops paying needs 80 GB, because the break-even sits near 150 concurrent sequences and that is about 44 GB of KV. Where the hardware is unavailable, the lab README says exactly which measurement is being substituted with a published figure, and the number is labelled cited rather than measured — see measurement provenance.
Download CPU reference checks for formula limits, prefix-cache arithmetic and lab parser fixtures. Run with Python 3.11+. This checks educational arithmetic and synthetic fixtures only, not real GPU timings or engine compatibility. Each lab directory also contains a downloadable run.py; check its help and the environment preflight.
Lab index
| # | Lab | What you measure | Chapter | Status |
|---|---|---|---|---|
| 01 | 01-measure-your-gpu-roofline |
Sweep matmul shapes to find achieved TFLOP/s and HBM GB/s on your actual card, then plot the ridge point. Every later prediction in the book is checked against these two numbers. | 00-04 | DRAFT |
| 02 | 02-kv-cache-sizing |
Compute the KV footprint from a model config, then compare it against what the engine reports at startup. Reconcile any gap — the gap is the lesson. | 02-01 | DRAFT |
| 03 | 03-continuous-batching-effects |
Sweep concurrency and watch TTFT, TPOT, and throughput trade against each other. Find the knee, then explain where it comes from. | 01-03 | DRAFT |
| 04 | 04-prefix-cache-hit-rate |
Run a shared-prefix workload against both engines, read the hit-rate metric, then deliberately break the cache and watch it collapse. | 02-03 | DRAFT |
| 05 | 05-chunked-prefill |
Measure inter-token-latency spikes with and without chunked prefill, under a workload mixing one long prompt with many short ones. | 01-05 | DRAFT |
| 06 | 06-quantization-tradeoff |
The same model in bf16 and in FP8/INT4: latency at batch 1, maximum concurrency, and a small eval to price the accuracy loss. | 04-04 | DRAFT |
| 07 | 07-tp-scaling |
TP = 1, 2, 4, 8 on one node. Measure the speedup, then attribute the shortfall to collectives with a profile rather than guessing. | 05-01 | DRAFT |
| 08 | 08-spec-decode-acceptance |
Measure acceptance length across workload types, then find the batch size at which speculation stops paying for itself. | 06-02 | DRAFT |
| 09 | 09-structured-decoding-overhead |
Price the grammar mask: throughput with and without a JSON schema, and how the gap scales with batch size and schema complexity. | 06-05 | DRAFT |
| 10 | 10-profile-a-decode-step |
Capture one decode iteration with torch.profiler and account for every microsecond in it — including the gaps between kernels. |
10-05 | DRAFT |
| 11 | 11-end-to-end-benchmark |
Both engines, identical hardware and workload, with warmup, saturation, and open-loop arrivals. Report goodput, not throughput. | 10-03 | DRAFT |
| 12 | 12-read-a-request-through-the-code |
Instrument both engines at every hop and watch a single request traverse the whole stack, matching each log line to a file and function. | 09-03 | DRAFT |
| 13 | 13-batch-invariance |
Send one prompt at several batch sizes with temperature 0 and diff the outputs; then re-run with the determinism flag and measure what it costs. Uses the engines' own correctness harnesses (tests/v1/determinism/, python/sglang/test/test_deterministic.py) — not the performance benchmark, which never compares output text. |
10-04 | DRAFT |
Lab conventions
Every lab directory has the same shape, so you can run any of them without reading the whole README first:
labs/01-measure-your-gpu-roofline/
├── README.html what it measures, hardware floor, exact commands, what to expect
└── run.py self-contained; --help lists every knob; prints a result table- Hardware floor is stated first. If your card cannot run it, the README says so before you install anything.
- Every run prints its environment. GPU, driver, torch version, engine version — so a result pasted into an issue is reproducible.
- Results are compared against a prediction. Each lab asks you to predict the number from a formula in FORMULAS before running, then explains the residual.
- No lab writes to the repos. The pinned source trees stay clean; labs read from them and write only into their own directory.
CPU reference route
Inference arithmetic and lab-parser checks cover cache sizing, speculative correction, request timing, prefix rounding, JSON Schema validity, trace unions and ULP ordering. Quantization and parallelism checks cover byte packing, GPTQ correction, scaling, paired evaluation, TP algebra, PP scheduling and remote expert routing.
Run either file with Python 3.11 or later. The core checks need only the standard library; adjacent lab-parser checks require their downloaded lab files, and the schema checks additionally use Lab 09's requirements. These are synthetic mathematical and parser checks, not replayed real traces or GPU performance measurements.