18 min read6 sections

Lab 06 · for chapter 04-04

Quantization: latency, capacity, accuracy

Same model in bf16 and FP8/INT4: latency at batch 1, max concurrency, and a small eval to price the accuracy.

Download lab script · CPU reference checks · Environment preflight.

Quantization is sold as a free 2×. It is not free and it is usually not 2×. This lab runs the same model three ways — bf16, FP8, and a 4-bit checkpoint — and puts a number on each of the three things you actually trade: latency at batch 1, how many concurrent requests fit, and how much accuracy you gave up to get them.

Hardware

The memory arithmetic runs anywhere. The latency and concurrency measurements need one GPU that holds an 8B model in bf16 — 24 GB is the floor. The FP8 compute arm additionally needs SM89 or newer (Ada L40S / RTX 4090, or Hopper H100), because vLLM's CUTLASS FP8 GEMM is gated on device capability, not just on the flag parsing. On older silicon you still get the memory win and you do not get the compute win, and that is worth knowing before you draw conclusions from a latency number. The accuracy arm runs on any card that can serve the model at all — it is the same eval on the same server, so its hardware floor is whatever the largest of your three configurations needs.

csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_entry.cu:L145-L159 vLLM

bool cutlass_scaled_mm_supports_fp8(int64_t cuda_device_capability) {
  // CUTLASS FP8 kernels need at least
  //   CUDA 12.0 on SM90 systems (Hopper)
  //   CUDA 12.4 on SM89 systems (Lovelace)

#if defined CUDA_VERSION
  if (cuda_device_capability >= 90) {
    return CUDA_VERSION >= 12000;
  } else if (cuda_device_capability >= 89) {
    return CUDA_VERSION >= 12040;
  }
#endif

  return false;
}

Not executed here

run.py was written against the CLI surface, log lines, and eval harnesses cited below, and its argument handling and arithmetic were exercised, but it has not been run against a live engine or a GPU. Every accuracy cell in this README is a placeholder you fill in. There is no measured accuracy number anywhere on this page, and if you find one, it is a defect — report it. Memory and capacity figures are derived arithmetic and labelled as such. Latency is not predicted at all, because it cannot be.

§1

What you measure

Three axes, and they move independently. That independence is the whole reason to run the lab rather than read a blog post.

axis 1

Latency at batch 1

Memory-bound. Halving the weight bytes should nearly halve the decode step, because at batch 1 you are reading the whole weight matrix to produce one token. Should. Whether it does depends on whether your card has the kernel.

axis 2

Maximum concurrency

Pure arithmetic, and the most reliable win. Weights you do not store are KV cache you can store. This one you can predict to within a few percent before running anything.

axis 3

Accuracy

The bill. It is task-dependent, checkpoint-dependent, and it is the only one of the three that a benchmark table cannot be borrowed for.

Predict axis 2 first, on paper. Llama-3-8B has 8.03 B parameters; in bf16 that is 14.96 GiB of weights, in FP8 7.48 GiB. Its per-token KV footprint across all 32 layers at $h_{kv}=8$, $d_h=128$, bf16 is $2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072$ bytes = 128 KiB (the derivation is §2.1; lab 02 is the calculator). On a 24 GiB card at vLLM's default utilisation:

vllm/config/cache.py:L80 vLLM

    gpu_memory_utilization: float = Field(default=0.92, gt=0, le=1)

Derived — arithmetic for Llama-3-8B on a 24 GiB card at util 0.92, ignoring activation and CUDA-graph reservations. Not measured. Your engine will report less; see lab 02 for why.

Configuration	Weights	KV budget	KV bytes/token	Token capacity	vs. bf16
bf16 weights, bf16 KV	14.96 GiB	7.12 GiB	131,072	58,351	1.00×
FP8 weights, bf16 KV	7.48 GiB	14.60 GiB	131,072	119,615	2.05×
FP8 weights, FP8 KV	7.48 GiB	14.60 GiB	65,536	239,230	4.10×

That 4.10× is the headline the marketing uses, and notice what produced it: the weight saving contributed 2.05× and the KV dtype contributed the other 2×. They are separate flags with separate accuracy consequences, and conflating them is how people end up blaming FP8 weights for damage done by an FP8 KV cache. Run them as three configurations, not two.

§2

The flags that actually exist

Both engines take a --quantization flag whose accepted values are a fixed list. Read the list rather than guessing; several names that circulate on the internet are not in it. vLLM's, at this SHA:

vllm/model_executor/layers/quantization/__init__.py:L12-L46 vLLM

QuantizationMethods = Literal[
    "awq",
    "auto_awq",
    "fp8",
    "fbgemm_fp8",
    "fp_quant",
    "modelopt",
    "modelopt_fp4",
    "modelopt_mxfp8",
    "modelopt_mixed",
    "auto_gptq",
    "gptq",
    "gptq_marlin",
    "awq_marlin",
    "humming",
    "compressed-tensors",
    "experts_int8",
    "quark",
    "moe_wna16",
    "torchao",
    "inc",
    "mxfp4",
    "gpt_oss_mxfp4",
    "deepseek_v4_fp8",
    "online",
    # Below are online quant shorthand names (see vllm.config.quantization).
    # Listed here as strings to avoid a circular import; kept in sync with
    # _ONLINE_SHORTHANDS by the assertion in get_quantization_config().
    "fp8_per_tensor",
    "fp8_per_block",
    "fp8_per_channel",
    "int8_per_channel_weight_only",
    "nvfp4_per_token",
    "mxfp8",
]

The last six are online shorthands: they quantize a bf16 checkpoint at load time, so you can run the FP8 arm against exactly the same weight files as the bf16 arm. That is the only way to make the comparison clean, and it is why this lab uses them:

vllm/config/quantization.py:L116-L130 vLLM

_ONLINE_SHORTHANDS: dict[str, QuantizationConfigArgs] = {
    "fp8_per_tensor": QuantizationConfigArgs(
        linear=QuantSpec(weight=kFp8StaticTensorSym),
        moe=QuantSpec(weight=kFp8StaticTensorSym),
    ),
    "fp8_per_block": QuantizationConfigArgs(
        linear=QuantSpec(weight=kFp8Static128BlockSym),
        moe=QuantSpec(weight=kFp8Static128BlockSym),
    ),
    # Per-output-channel weight scale + dynamic per-token activation.
    # Same shape as llmcompressor's FP8_DYNAMIC recipe.
    "fp8_per_channel": QuantizationConfigArgs(
        linear=QuantSpec(weight=kFp8StaticChannelSym),
        moe=QuantSpec(weight=kFp8StaticChannelSym),
    ),

Plain --quantization fp8 also works on a bf16 checkpoint, because Fp8Config defaults to "this checkpoint is not FP8-serialized" and quantizes on load:

vllm/model_executor/layers/quantization/fp8.py:L95-L106 vLLM

    def __init__(
        self,
        is_checkpoint_fp8_serialized: bool = False,
        activation_scheme: str = "dynamic",
        ignored_layers: list[str] | None = None,
        weight_block_size: list[int] | None = None,
        store_dtype: str | None = None,
    ) -> None:
        super().__init__()

        self.is_checkpoint_fp8_serialized = is_checkpoint_fp8_serialized

SGLang's list is separate and differently spelled — w8a8_fp8 and w8a8_int8 have no vLLM equivalent by that name, and marlin appears as a bare entry:

python/sglang/srt/server_args.py:L143-L164 SGLang

QUANTIZATION_CHOICES = [
    "awq",
    "fp8",  # MOE + linear online quantization.
    "mxfp8",  # MOE + linear online quantization.
    "gptq",
    "marlin",
    "gptq_marlin",
    "awq_marlin",
    "bitsandbytes",
    "gguf",
    # Modelopt has some online quantization support through ModelOptModelLoader.
    "modelopt",
    "modelopt_fp8",
    "modelopt_fp4",
    "nvfp4_online",
    "modelopt_mixed",
    "petit_nvfp4",
    "w8a8_int8",  # mentioned in quantization.md documentation, supporting compressed-tensors quant_method.
    "w8a8_fp8",  # mentioned in quantization.md documentation, supporting compressed-tensors quant_method.
    "moe_wna16",  # custom loading logic for gptq/awq checkpoints (likely untested/unused)
    "w4afp8",
    "mxfp4",  # MOE-only.

The KV cache dtype is a separate flag on both, and this is the single most common methodological error in quantization comparisons — quantizing the cache and reporting the result as "FP8 model":

vllm/engine/arg_utils.py:L1221, L1228 and vllm/config/cache.py:L88-L90 vLLM

        cache_group.add_argument("--block-size", **cache_kwargs["block_size"])
# ...
        cache_group.add_argument("--kv-cache-dtype", **cache_kwargs["cache_dtype"])
    cache_dtype: CacheDType = "auto"
    """Data type for kv cache storage. If "auto", will use model data type.
    CUDA 11.8+ supports fp8 (=fp8_e4m3) and fp8_e5m2. ROCm (AMD GPU) supports

python/sglang/srt/server_args.py:L690-L701 SGLang

    kv_cache_dtype: A[
        str,
        Arg(
            help=(
                'Data type for kv cache storage. "auto" will use model data type. '
                '"bf16" or "bfloat16" for BF16 KV cache. "fp8_e5m2" and '
                '"fp8_e4m3" are supported for CUDA 11.8+. "mxfp8" is supported '
                'by the FA4 backend. "nvfp4" selects '
                'the NVFP4 FP4 E2M1 KV cache recipe; "fp4_mx_block16" '
                "selects the MX-style block-size-16 FP4 E2M1 KV cache "
                "recipe. Both require CUDA 12.8+ and PyTorch 2.8.0+"
            ),

4-bit is not the same experiment

There is no online INT4 path in either list. A 4-bit arm means loading a different artifact — an AWQ or GPTQ checkpoint someone else produced with their own calibration set. You are then comparing three weights files, not one model at three precisions, and any accuracy difference you measure includes whatever that calibration did. Say so in your write-up. SGLang's own AWQ test is explicit about the flag pairing:

test/registered/quant/test_awq.py:L56-L64 SGLang

    def setUpClass(cls):
        cls.model = "QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ"
        cls.base_url = DEFAULT_URL_FOR_TEST
        cls.process = popen_launch_server(
            cls.model,
            cls.base_url,
            timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
            other_args=["--dtype", "bfloat16", "--quantization", "awq_marlin"],
        )

§3

Running it

Step 0 — predict, with no GPU

shell — labs/06-quantization-tradeoff shell

$ python3 run.py plan --params 8.03e9 --layers 32 --kv-heads 8 --head-dim 128 --vram 24
$ python3 run.py plan --config ./config.json --vram 80 --context-len 8192
$ python3 run.py --help          # every knob, all three subcommands

Step 1 — three servers, one at a time

shell — vLLM, three configurations shell

$ vllm serve meta-llama/Meta-Llama-3-8B-Instruct --port 8000

$ vllm serve meta-llama/Meta-Llama-3-8B-Instruct --port 8000 \
    --quantization fp8

$ vllm serve meta-llama/Meta-Llama-3-8B-Instruct --port 8000 \
    --quantization fp8 --kv-cache-dtype fp8

shell — SGLang, the same three shell

$ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000

$ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000 \
    --quantization fp8

$ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000 \
    --quantization fp8 --kv-cache-dtype fp8_e4m3

Before believing any of it, confirm the quantization actually happened. Both engines say so at startup. SGLang names the method and counts the layers it touched:

python/sglang/srt/model_executor/model_runner_components/load_model_utils.py:L156-L164 SGLang

    if (
        server_args.quantization is not None
        and isinstance(quantized_layers, tuple)
        and len(quantized_layers) == 2
    ):
        layer_types, quantized_layers_count = quantized_layers
        logger.info(
            f"Online {server_args.quantization} quantization: quantized {quantized_layers_count} layers of types: {layer_types}"
        )

and then reports the weight footprint you predicted in step 0:

python/sglang/srt/model_executor/model_runner.py:L1158-L1166 SGLang

        if self.startup_weight_load is None:
            logger.info(
                f"Load weight end. "
                f"elapsed={self.weight_load_time:.2f} s, "
                f"type={type(self.model).__name__}, "
                f"{quant_str + ', ' if quant_str else ''}"
                f"avail mem={after_avail_memory:.2f} GB, "
                f"mem usage={self.weight_load_mem_usage:.2f} GB."
            )

vLLM's equivalent pair is the weight footprint and, later, the concurrency the cache supports — which is axis 2, handed to you for free:

vllm/v1/worker/gpu_model_runner.py:L5510-L5514 vLLM

        logger.info_once(
            "Model loading took %s GiB memory and %.6f seconds",
            format_gib(self.model_memory_usage),
            time_after_load - time_before_load,
        )

vllm/v1/core/kv_cache_utils.py:L1925-L1931 vLLM

    logger.info_once(
        "GPU KV cache size: %s tokens, "
        "Maximum concurrency for %s tokens per request: %.2fx",
        f"{num_tokens:,}",
        f"{max_model_len:,}",
        max_concurrency,
    )

SGLang prints the same information as raw counts on rank 0 only, and you divide:

python/sglang/srt/managers/scheduler.py:L1094-L1100 SGLang

        if self.ps.tp_rank == 0:
            logger.info(
                f"max_total_num_tokens={self.max_total_num_tokens}, "
                f"chunked_prefill_size={get_schedule().chunked_prefill_size}, "
                f"max_prefill_tokens={self.max_prefill_tokens}, "
                f"max_running_requests={self.max_running_requests}, "
                f"context_len={self.model_config.context_len}, "

Step 2 — latency at batch 1

Use the engines' own single-batch benchmarks, not a stopwatch around curl. vLLM's takes the batch size as a flag and does its own warmup:

vllm/benchmarks/latency.py:L35-L53 vLLM

    parser.add_argument("--input-len", type=int, default=32)
    parser.add_argument("--output-len", type=int, default=128)
    parser.add_argument("--batch-size", type=int, default=8)
    parser.add_argument(
        "--n",
        type=int,
        default=1,
        help="Number of generated sequences per prompt.",
    )
    parser.add_argument("--use-beam-search", action="store_true")
    parser.add_argument(
        "--num-iters-warmup",
        type=int,
        default=10,
        help="Number of iterations to run for warmup.",
    )
    parser.add_argument(
        "--num-iters", type=int, default=30, help="Number of iterations to run."
    )

SGLang's takes a list of batch sizes and can attach to a server you already started:

python/sglang/benchmark/one_batch_server.py:L1-L11 SGLang

"""
Benchmark the latency of running a single batch with a server.

This script launches a server and uses the HTTP interface.
It accepts server arguments (the same as launch_server.py) and benchmark arguments (e.g., batch size, input lengths).

Usage:
python3 -m sglang.benchmark.one_batch_server --model meta-llama/Meta-Llama-3.1-8B --batch-size 1 16 64 --input-len 1024 --output-len 8

python3 -m sglang.benchmark.one_batch_server --model None --base-url http://localhost:30000 --batch-size 16 --input-len 1024 --output-len 8

shell — batch-1 latency, three ways shell

$ vllm bench latency --model meta-llama/Meta-Llama-3-8B-Instruct \
    --batch-size 1 --input-len 1024 --output-len 128 --output-json bf16.json

$ python3 -m sglang.benchmark.one_batch_server --model None \
    --base-url http://localhost:30000 --batch-size 1 --input-len 1024 --output-len 128

# or the lab's own client, which drives either engine and saves a row you can merge
$ python3 run.py latency --engine vllm --base-url http://127.0.0.1:8000 \
    --label bf16 --save bf16.json

Step 3 — price the accuracy

Do not write your own eval. Both engines ship one that already talks to a running server, and using theirs means your number is comparable to their CI. vLLM's:

tests/evals/gsm8k/README.md:L16-L22 vLLM

```bash
# Start vLLM server first
vllm serve Qwen/Qwen2.5-1.5B-Instruct --port 8000

# Run evaluation
python tests/evals/gsm8k/gsm8k_eval.py --port 8000
```

tests/evals/gsm8k/gsm8k_eval.py:L342-L361 vLLM

    parser.add_argument(
        "--num-shots", type=int, default=5, help="Number of few-shot examples"
    )
    parser.add_argument(
        "--num-questions",
        type=int,
        default=1319,
        help="Number of questions to evaluate",
    )
    parser.add_argument(
        "--max-tokens", type=int, default=256, help="Max tokens for generation"
    )
    parser.add_argument("--host", type=str, default="http://127.0.0.1", help="Host URL")
    parser.add_argument("--port", type=int, default=8000, help="Port number")
    parser.add_argument(
        "--temperature", type=float, default=0.0, help="Temperature for generation"
    )
    parser.add_argument(
        "--seed", type=int, default=42, help="Random seed for reproducibility"
    )

SGLang's unified runner covers the same task and several more; the deprecated sglang.test.few_shot_gsm8k entry point still works but points here:

python/sglang/test/run_eval.py:L1-L4 SGLang

"""
Usage:
python3 -m sglang.test.run_eval --port 30000 --eval-name mmlu --num-examples 10
"""

shell — the same eval against each server shell

$ python3 tests/evals/gsm8k/gsm8k_eval.py --port 8000 --num-questions 1319 \
    --num-shots 5 --temperature 0 --save-results bf16-gsm8k.json

$ python3 -m sglang.test.run_eval --port 30000 --eval-name gsm8k --num-examples 1319

Use the full 1319 questions for the run you report, and the same seed and temperature for every arm. A 200-question run has a standard error around three percentage points on its own, which is larger than the effect you are trying to measure — you would be reporting noise.

§4

What to expect

Axis 2 is the one you can predict

Compare the engine's reported capacity against the plan output. If it comes out within a few percent, you understand the allocator; if not, lab 02 has the list of causes. FP8 weights should roughly double capacity on a 24 GiB card with an 8B model, and adding an FP8 KV cache should double it again. Both are arithmetic, both are checkable.

Axis 1 has three possible outcomes, and all three are informative

How to read a batch-1 latency result. No numbers here are predictions — the point is what each outcome means.

Observation	Most likely cause	How to confirm
FP8 clearly faster at batch 1	Working FP8 GEMM; you are weight-bandwidth-bound and halved the weights.	The gain should shrink toward nothing as you raise the batch size, because you stop being bandwidth-bound. Sweep it.
FP8 the same speed, memory clearly lower	No FP8 kernel for your card — weights stored small, dequantized before a bf16 matmul.	Check compute capability against the CUTLASS gate quoted at the top of this page.
FP8 slower	Per-token activation quantization or dequantization overhead dominating at batch 1, or a fallback kernel.	Profile one decode step — lab 10. Look for quant/dequant kernels between the GEMMs.

Axis 3: what a defensible accuracy table looks like

This is the table you produce. Every cell marked fill in is a number you measured on your own hardware with the commands above. It is deliberately empty. Do not copy accuracy figures from anywhere — including from this book.

Template. Empty on purpose. GSM8K, 1319 questions, 5-shot, temperature 0, one server per row.

Configuration	Weights (GiB)	Capacity (tok)	Batch-1 TPOT	GSM8K	Δ vs. bf16
bf16 / bf16 KV	fill in	fill in	fill in	fill in	— (baseline)
FP8 / bf16 KV	fill in	fill in	fill in	fill in	fill in
FP8 / FP8 KV	fill in	fill in	fill in	fill in	fill in
AWQ / GPTQ 4-bit	fill in	fill in	fill in	fill in	fill in

run.py report prints exactly this table, and it prints FILL IN in any accuracy cell you did not pass a measured value for. That is deliberate: the script cannot be made to emit an accuracy number it was not given.

The only accuracy figures on this page are vLLM's own CI floors

For a sense of scale — not for your write-up — vLLM's GSM8K suite carries matched configurations for a model in bf16 and the same model in FP8. These are the thresholds the maintainers expect the model to clear, on 1319 questions at 5 shots:

tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-DEP2.yaml:L1-L9 and tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-FP8-DEP2.yaml:L1-L10 vLLM

model_name: "Qwen/Qwen3.5-35B-A3B"
accuracy_threshold: 0.84
tolerance: 0.03
num_questions: 1319
num_fewshot: 5
server_args: >-
  --max-model-len 4096
  --data-parallel-size 2
  --enable-expert-parallel
model_name: "Qwen/Qwen3.5-35B-A3B-FP8"
accuracy_threshold: 0.79
tolerance: 0.03
num_questions: 1319
num_fewshot: 5
server_args: >-
  --max-model-len 4096
  --data-parallel-size 2
  --enable-expert-parallel
  --kv-cache-dtype fp8

Read that carefully, because it is easy to misread. These are floors, not measurements — the assertion is one-sided:

tests/evals/gsm8k/test_gsm8k_correctness.py:L189-L192 vLLM

        assert measured_metric >= expected_metric - tol, (
            f"GSM8K metric too low: {measured_metric:.4f} < "
            f"{expected_metric:.4f} - {tol:.4f} = {expected_metric - tol:.4f}"
        )

So the honest statement is: vLLM's maintainers set the FP8 variant's floor five points below the bf16 variant's, for a configuration that quantizes both weights and KV cache, on a MoE model, on GSM8K. That is a data point about their risk tolerance and the shape of the effect. It is not "FP8 costs five points", and quoting it that way would be wrong in at least four directions: different model, different task, weights-and-cache conflated, and a floor is not a mean. Cited; thresholds, not measurements.

§5

Exercises

Predict the token capacity for Llama-3-70B in bf16 and in FP8 on 2×80 GiB at --tensor-parallel-size 2. One of the two answers is qualitatively different from a "2×" story. Which, and why?
Run the FP8 arm with --quantization fp8 and again with --quantization fp8_per_channel. Predict which is faster at batch 1 and which is more accurate, then measure both. Explain any surprise in terms of where the scale factors live.
Serve bf16 with --kv-cache-dtype fp8 and nothing else quantized. Measure capacity, batch-1 latency, and GSM8K. Which of the three moved, and does that isolate the cost of the cache dtype from the cost of the weight dtype?
Sweep batch size 1, 4, 16, 64 for bf16 and FP8. Plot the FP8 speedup against batch size. Predict the shape of that curve before you look, from the arithmetic-intensity argument in §0.4.
Read vllm/model_executor/layers/quantization/utils/w8a8_utils.py and answer: on a card that fails the CUTLASS FP8 gate, does vLLM refuse to start, silently fall back, or warn? Then decide which of those three you would want in production, and why the answer is not obvious.

Answers

70.6 B parameters is 131.5 GiB in bf16 and 65.7 GiB in FP8. Across two 80 GiB cards at util 0.92 the budget is 147.2 GiB. bf16 leaves 15.7 GiB for KV; FP8 leaves 81.5 GiB. Per-token KV at $L=80$, $h_{kv}=8$, $d_h=128$, bf16 is 320 KiB, so capacity goes from about 51,000 tokens to about 267,000 — a 5.2× jump, not 2×. The reason is that capacity is a residual: when weights dominate the budget, halving them does far more than double what is left. This is also why the same experiment on a 1 B model shows almost no capacity gain at all. (Derived; arithmetic.)
Both store FP8 weights, but scale metadata, alignment and backend workspaces can differ. The difference is scale granularity: fp8 with dynamic activation scaling versus a per-output-channel weight scale. The per-channel variant has strictly more scale factors, so it represents outlier channels with less within-channel range pressure; this does not guarantee better task accuracy. Whether it is faster depends on the workload and on whether your card has a kernel that fuses the per-channel scale into the epilogue or has to do a separate pass. Measure it — and note that "more scales, same speed" is the outcome to hope for and not the outcome to assume.
Capacity roughly doubles; batch-1 latency barely moves (you did not change the weights, and at batch 1 the KV read is small next to the weight read); GSM8K may move, and if it does, that movement is attributable to the cache dtype alone. That isolation is the entire point of running it as a separate arm, and it is what the four-row template above is structured to give you.
A decreasing curve is one hypothesis, not a guarantee. At batch 1 the decode step reads the whole weight matrix to produce one token, so time is weight bytes over bandwidth and halving the bytes nearly halves the time. As batch grows, the same weight read is amortised over more tokens, arithmetic intensity rises, and you cross the ridge into compute-bound territory where FP8 only helps if the FP8 tensor cores are actually faster than the bf16 ones. Kernel dispatch, calibration, KV traffic and achievable FP8 compute can instead produce plateaus or nonmonotonic changes. Report the measured curve and its uncertainty rather than forcing it toward 1×.
It falls back, in two places and silently. Fp8Config calls cutlass_fp8_supported() while choosing its activation quantization key and picks per-tensor instead of per-token when the answer is no (vllm/model_executor/layers/quantization/fp8.py:L286-L292); separately, the CUTLASS scaled-mm kernel's is_supported returns (False, "...not supported") so the dispatcher chooses a different kernel (vllm/model_executor/kernels/linear/scaled_mm/cutlass.py:L287-L295). Neither raises. The production argument cuts both ways — a silent fallback keeps you serving on a mixed fleet, and it also means a capacity-planning spreadsheet built on "FP8 is 1.8× faster" can be quietly wrong on a third of your nodes. The defensible answer is: fall back, but assert the capability in your deployment check, not at 3 a.m.

§6

Key takeaways

Weight quantization and KV-cache quantization are separate flags with separate accuracy consequences. Run three arms, not two, or you will attribute the cache's damage to the weights.
Capacity is a residual, so the capacity gain from halving weights is larger the more the weights dominated the budget — 2× on an 8B model at 24 GiB, over 5× on a 70B at 2×80 GiB. Compute it, do not assume it.
FP8 latency at batch 1 depends on a compute-capability gate in a CUDA file, not on the flag being accepted. A card below SM89 gives you the memory win and no compute win, and the flag will not tell you.
Use online quantization for the FP8 arm so all three arms share one weights file. A 4-bit arm cannot do that, and its accuracy result therefore includes somebody else's calibration set.
Use the engines' own eval harnesses at the full question count. A 200-question GSM8K run has more sampling error than the effect you are measuring.
The accuracy table is the one artefact you cannot borrow. Everything else in this lab can be predicted or cited; accuracy has to be measured on your model, your task, and your checkpoint.