The design space: PTQ/QAT, weight-only, W+A, KV

Here is the warning, at the bottom of SGLang's config validation:

            if self.quantization not in optimized_quantization_methods:
                # Don't warn for MXFP4/MXFP8 on SM100 since they have optimized kernels
                if not (
                    self.quantization in ["mxfp4", "mxfp8"] and is_sm100_supported()
                ):
                    logger.warning(
                        "%s quantization is not fully "
                        "optimized yet. The speed can be slower than "
                        "non-quantized models.",
                        self.quantization,
                    )

Of the 31 real method names SGLang accepts on --quantization — 32 choices, one of which is the unquant escape hatch — 12 are absent from the optimized_quantization_methods allow-list, including awq, gptq, bitsandbytes, gguf and mxfp4 (python/sglang/srt/server_args.py:L143-L179, python/sglang/srt/configs/model_config.py:L1473-L1496). They load fine and may run slower than bf16. And even inside the allow-list, "optimized" means "there is a fast kernel for this format", not "this will make your deployment faster" — that depends on your batch size, and no engine can see your batch size at startup.

The second failure mode is quieter. Take Llama-3-8B to 4-bit weights and you expect 16.06 GB to become 4.0 GB. It becomes 5.73 GB, because 13% of the parameters are the embedding and the LM head and almost nobody quantizes those, and because a group-128 scale plus a zero-point is 0.156 extra bits on every weight. 2.80×, not 4×. Nothing warned you. The arithmetic is in §4.1.3.

This chapter is the map. It gives you the axes, the metadata cost of each, and a decision procedure that starts from a workload description and ends at a scheme. §4.2 does the algorithms that pick the scales, §4.3 does the kernels that consume them, and §4.4 does the measurement methodology. Number formats themselves — bit layouts, dynamic range, the affine map $x \approx s(q - z)$, accumulation dtype — belong to §0.5 and are not re-derived here.

A quantization scheme is a point in a six-dimensional space. Five of the dimensions are independent of each other; the sixth (accumulation) is independent of all of them and is the one people forget.

vLLM encodes exactly this in one frozen dataclass. If you read nothing else in the quantization tree, read QuantKey:

@dataclass(frozen=True)
class ScaleDesc:
    """
    Class for describing a single quantization scaling factor.
    dtype: data type of the scale
    static: static scale if True, dynamic if False
    group_shape: group shape of the scale
    """

    dtype: torch.dtype
    static: bool
    group_shape: GroupShape
# ...

@dataclass(frozen=True)
class QuantKey:
    """
    Class for identifying the type of quantization.
    dtype: quantized data type
    scale: scale descriptor
    scale2: second-level scale descriptor
    symmetric: symmetric if True, asymmetric if False
    """
# ...
    dtype: torch.dtype | ScalarType
    scale: ScaleDesc
    scale2: ScaleDesc | None = None
    symmetric: bool = True

Storage dtype, granularity (group_shape), static-vs-dynamic (static), symmetry (symmetric), and a second-level scale for two-tier formats like NVFP4. That is axes 1–5 minus "which tensor", and "which tensor" is supplied by where the key is used: weight_quant_key or activation_quant_key.

The figure is the whole argument for why the axes are not interchangeable. At batch 1 with a 2k context, weights are 98.1% of the bytes a decode step moves, the KV cache is 1.9%, and activations are 0.002%. Quantizing activations at batch 1 saves you nothing, because there are no activation bytes to save. Quantizing weights is the only lever that touches the 98%.

Granularity: metadata is a real tax

Let $W$ be a weight matrix of $N$ output channels by $K$ input channels, $b$ the bits per quantized element, $b_s$ the bits per scale, and $g$ the number of weights sharing one scale. The effective bits per weight is

Granularity in vLLM is a two-integer object, not a string. A negative entry means "the whole extent of that dimension":

# Use proxy as NamedTuple direct subclasses cannot have static members
class _GroupShape(NamedTuple):
    row: int
    col: int


class GroupShape(_GroupShape):
    """
    This class describes the quantization group shape.
    It includes static members for common shapes (per-tensor, per-token).
    """
# ...
    def is_per_tensor(self) -> bool:
        return self.row == -1 and self.col == -1

    def is_per_token(self) -> bool:
        return self.row == 1 and self.col == -1

    def is_per_channel(self) -> bool:
        return self.row == -1 and self.col == 1

    def is_per_group(self) -> bool:
        return self.row == 1 and self.col >= 1


GroupShape.PER_TENSOR = GroupShape(-1, -1)
GroupShape.PER_TOKEN = GroupShape(1, -1)
GroupShape.PER_CHANNEL = GroupShape(-1, 1)

PER_TOKEN and PER_CHANNEL are the same shape family applied to different operands: one scale per row of the activation matrix, one scale per output channel of the weight matrix. That symmetry is why W8A8 with per-channel weights and per-token activations is the natural pairing — both scale vectors factor cleanly out of the dot product, and the GEMM epilogue applies them as a rank-1 outer product.

Now the whole-model arithmetic that produced the 2.80× in §4.1.1. Llama-3-8B has $8.030\times10^{9}$ parameters, of which $6.979\times10^{9}$ ($86.9\%$) sit in the 224 linear matrices of Figure 1 and $1.051\times10^{9}$ ($13.1\%$) in the untied embedding and LM head.

2.80×, not 4×. Flipping lm_head_quantized on — AWQ and GPTQ both expose it (vllm/model_executor/layers/quantization/auto_awq.py:L183-L199) — gets you to 3.24× and puts 4-bit noise directly on the logits. Most published checkpoints leave it off.

Symmetry: why weights are almost always symmetric

An asymmetric quantizer stores $x \approx s(q-z)$ with a per-group integer zero-point $z$; a symmetric one fixes $z=0$. For weights, three things push towards symmetric. First, weight distributions in trained transformers are close to zero-mean, so the zero-point buys little. Second, $z \neq 0$ adds a cross-term to the GEMM: $\sum_k w_k x_k$ becomes $s_w s_x \sum_k (q^w_k - z_w)(q^x_k - z_x)$, whose expansion needs a column-sum correction term that either costs a pre-pass or bloats the epilogue. Third — and this is the one that actually decides it — the low-precision tensor-core instructions take signed integers, so symmetric maps onto the hardware and asymmetric does not.

vLLM's compressed-tensors predicates say this out loud, in a comment repeated verbatim across every W8A8 and W4A8 detector:

    def _is_dynamic_token_w8a8(
        weight_quant: QuantizationArgs, input_quant: QuantizationArgs
    ) -> bool:
        is_8_bits = weight_quant.num_bits == input_quant.num_bits == 8
        weight_strategy = (
            weight_quant.strategy == QuantizationStrategy.TENSOR.value
            or weight_quant.strategy == QuantizationStrategy.CHANNEL.value
        )
        is_token = (
            weight_strategy and input_quant.strategy == QuantizationStrategy.TOKEN.value
        )
        is_dynamic = not weight_quant.dynamic and input_quant.dynamic

        # Both symmetric and asymmetric input quantization supported.
        # Only symmetric weight quantization supported.
        return is_8_bits and is_token and weight_quant.symmetric and is_dynamic

The exception is 4-bit weight-only, where the dequantised weight goes into a bf16 tensor-core GEMM anyway, so the zero-point costs nothing at the instruction level and buys real accuracy on skewed groups. That is why AWQ carries zero_point as a first-class constructor argument (auto_awq.py:L183-L195) and GPTQ carries is_sym (auto_gptq.py:L106-L131).

Static vs dynamic activation scales

Weights are known at load time, so their scales are always static. Activations are not, and you get two choices. Static: run a calibration set through the model, record the observed range per tensor, freeze it into the checkpoint. Costs one fp32 per tensor, zero runtime work, and clips anything the calibration set did not see. Dynamic: compute $s = \max|x| / q_{\max}$ over each token's row inside the kernel, every forward pass. Costs a reduction over the activation and an extra scale array. A finite exact max-based scale covers that sampled row, but overflow, scale underflow/rounding, saturation, nonfinite values and implementation choices still need explicit handling.

The line in vLLM's FP8 linear method that picks between them is the clearest statement of the axis anywhere in either tree:

        self.weight_block_size = self.quant_config.weight_block_size
        self.block_quant = self.weight_block_size is not None
        self.act_q_static = self.quant_config.activation_scheme == "static"

        if self.block_quant:
            assert not self.act_q_static
            assert self.weight_block_size is not None

            self.activation_quant_key = create_fp8_quant_key(
                static=self.act_q_static,
                group_shape=GroupShape(1, self.weight_block_size[0]),
            )
            self.weight_quant_key = create_fp8_quant_key(
                static=True, group_shape=GroupShape(*self.weight_block_size)
            )
        else:
            self.weight_quant_key = kFp8StaticTensorSym
            # Use per-token quantization for better perf if dynamic and cutlass
            if self.act_q_static:
                self.activation_quant_key = kFp8StaticTensorSym
            elif cutlass_fp8_supported():
                self.activation_quant_key = kFp8DynamicTokenSym
            else:
                self.activation_quant_key = kFp8DynamicTensorSym
        

Note the assert not self.act_q_static under block quantization: the two axes are not fully independent in practice, because a 128×128 weight block only pairs with a 1×128 activation group, and a static scale over a 128-element slice of an activation row is not a thing anyone calibrates. The config constructor rejects the combination with an explicit message (fp8.py:L115-L132). Everywhere else in the space, dynamic per-token is the a common choice because it adapts to the current row. Fusing the reduction and conversion can hide some overhead, but neither higher accuracy nor zero cost is universal.

PTQ vs QAT, and why serving only sees one of them

Quantization-aware training inserts fake-quant nodes into the forward pass and lets gradient descent adapt the weights to the rounding. It produces better low-bit models, and it costs a training run. Post-training quantization takes a finished checkpoint, runs a few hundred calibration sequences through it, and solves for scales — minutes to hours on one GPU.

Serving engines are entirely agnostic. Nothing in either registry, either QuantizationConfig base class, or any from_config reads a field saying how the checkpoint was produced: a QAT and a PTQ checkpoint at the same (format, granularity, symmetry) load through identical code. The axis decides which checkpoint you download, and PTQ dominates serving because the people who serve models are not the people who trained them. §4.2 covers the PTQ algorithms.

Accumulation

Independent of every axis above. INT4 weights dequantised to bf16 accumulate in fp32; INT8 GEMMs accumulate in int32; FP8 GEMMs accumulate in fp32 on H100 tensor cores. Storage dtype constrains bandwidth; accumulate dtype constrains numerical drift over $K$ terms. Both matter and they are separate. §0.5 owns this and works the error bound.

The roofline decides the first branch and nothing else does. From §0.4: a decode step with batch $B$ reads each weight once and does $2B$ FLOP with it, so the arithmetic intensity of the weight-stationary GEMM is

where $b_w$ is bytes per stored weight and $I^{*} = \pi/\beta$ is the ridge point. On an H100 SXM, $\pi_{\text{bf16}} = 989.4$ TFLOP/s and $\beta = 3.35$ TB/s, so $I^{*}_{\text{bf16}} = 295$; fp8 tensor cores run at $\pi_{\text{fp8}} = 1979$ TFLOP/s (§0.4, table of H100 constants), so $I^{*}_{\text{fp8}} = 591$.

Read the last two rows against each other, because that is the entire argument. W4A16 divides the memory-bound step time by four but leaves the compute ceiling at bf16, so it runs out of runway at $B \approx 74$ and past that converges to bf16 performance minus the dequantisation overhead. W8A8-FP8 divides step time by only two in the memory-bound region — but it also doubles the ceiling, so its crossover sits at the same $B^{*} = 295$ as bf16 and it stays 2× ahead on both sides of the ridge. Weight-only wins at low batch; W+A wins at high batch; their ideal pairwise crossover is around $B=\pi_{bf16}/(2\beta)\approx148$. W4's own ridge near 74 is a different boundary: W4 remains faster than W8A8 between 74 and 148 in this model. Actual kernel and mixed-layer costs can shift the crossing.

That derived boundary agrees with the largest published sweep: the Red Hat / Neural Magic study (arXiv:2411.02355, >500,000 evaluations over the Llama-3.1 family) recommends W4A16 for synchronous, low-concurrency deployments and W8A8 for asynchronous continuous batching. Same conclusion, arrived at by measurement rather than by roofline.

Three things the tree deliberately does not do. It does not branch on model size — size changes whether you must quantize, not which scheme wins. It does not branch on context length except through $N$, because context length only enters through the KV axis. And it puts the accuracy gate last, as a veto rather than a selector, because accuracy is the one axis you cannot predict from the workload description and must measure.

Both engines keep a single flat registry mapping a checkpoint's quant_method string to a QuantizationConfig subclass. vLLM's is a Literal plus a dict built lazily inside get_quantization_config (vllm/model_executor/layers/quantization/__init__.py:L12-L47 and L139-L180); SGLang's is a module-level dict, patched per platform (python/sglang/srt/layers/quantization/__init__.py:L71-L143).

Three structural facts the table makes visible.

The KV axis is not in the registry at all. vLLM's KV formats live in CacheDType in vllm/config/cache.py:L19-L37 — seventeen values including four TurboQuant modes and int4_per_token_head — and SGLang's in --kv-cache-dtype's nine choices at python/sglang/srt/server_args.py:L690-L713. A weight scheme and a KV scheme are selected by different flags and validated by different code, but compatibility still depends on model architecture, kernel, device and checkpoint metadata. The one place they meet is compressed-tensors, whose checkpoints can carry a kv_cache_scheme — and vLLM restricts what it will accept there to a single point in the space:

        type_ = kv_cache_scheme.get("type")
        num_bits = kv_cache_scheme.get("num_bits")

        if type_ != "float" or num_bits != 8:
            raise NotImplementedError(
                "Currently supported kv cache quantization is "
                "num_bits=8, type=float, however "
                f"received num_bits={num_bits}, type={type_}"
            )

vLLM has an online path; SGLang's equivalent is narrower. vLLM's _ONLINE_SHORTHANDS desugars a CLI string into a full QuantizationConfigArgs with separate linear and moe specs, each holding a weight QuantKey and an optional activation QuantKey:

# CLI shorthands accepted by `--quantization`. Each desugars to a full
# QuantizationConfigArgs; activation overrides go through quantization_config.
_ONLINE_SHORTHANDS: dict[str, QuantizationConfigArgs] = {
    "fp8_per_tensor": QuantizationConfigArgs(
        linear=QuantSpec(weight=kFp8StaticTensorSym),
        moe=QuantSpec(weight=kFp8StaticTensorSym),
    ),
    "fp8_per_block": QuantizationConfigArgs(
        linear=QuantSpec(weight=kFp8Static128BlockSym),
        moe=QuantSpec(weight=kFp8Static128BlockSym),
    ),
    # Per-output-channel weight scale + dynamic per-token activation.
    # Same shape as llmcompressor's FP8_DYNAMIC recipe.
    "fp8_per_channel": QuantizationConfigArgs(
        linear=QuantSpec(weight=kFp8StaticChannelSym),
        moe=QuantSpec(weight=kFp8StaticChannelSym),
    ),
# ...
    # INT8 weight-only on MoE; linear stays unquantized (no `linear` field).
    "int8_per_channel_weight_only": QuantizationConfigArgs(
        moe=QuantSpec(weight=kInt8StaticChannelSym),
    ),

This is the design space made addressable: QuantSpec(weight=..., activation=...) per layer kind, so you can quantize the MoE experts and leave attention alone — which is exactly what int8_per_channel_weight_only does. SGLang reaches the same place differently, with a REQUANTIZATION_METHODS set that allows a CLI method to override a checkpoint's declared method and requantize at load (python/sglang/srt/configs/model_config.py:L1546-L1587), logging a warning that "requantization may incur a loss in accuracy".

Neither list is stable. bitsandbytes and gguf are in SGLang's registry and have no entry in vLLM's QuantizationMethods literal or any file in vllm/model_executor/layers/quantization/ at a556f3f. Meanwhile vLLM has collapsed awq/awq_marlin onto one AutoAWQConfig and gptq/gptq_marlin/auto_gptq onto one AutoGPTQConfig, with the backend now chosen inside the config rather than by the method name. Version-stamp anything you copy from the table.

A quantized checkpoint declares itself with one dict in config.json: quantization_config, whose quant_method key is the registry lookup string. Everything else in the dict is method-specific and gets parsed by that method's from_config.

The probe loop is the interesting part. A checkpoint that says quant_method: "gptq" may be better served by the Marlin kernels, or by the moe_wna16 path if it is an MoE — so every config class gets a chance to claim it, and the order is hard-coded because several of them would claim the same checkpoint:

        quant_cfg = self.model_arch_config.quantization_config

        if quant_cfg is not None:
            quant_method = quant_cfg["quant_method"]
            # Quantization methods which are overrides (i.e. they have a
            # `override_quantization_method` method) must be checked in order
            # of preference (this is particularly important for GPTQ).
            overrides = [
                "auto_gptq",
                "gptq",
                "gptq_marlin",
                "auto_awq",
                "awq",
                "awq_marlin",
                "inc",
                "moe_wna16",
                "modelopt",
# ...
            ]
# ...
            quantization_methods = [
                q for q in supported_quantization if q not in overrides
            ]
            # Any custom overrides will be in quantization_methods so we place
            # them at the start of the list so custom overrides have preference
            # over the built-in ones.
            quantization_methods = quantization_methods + overrides

If no override fires and the user also passed --quantization, the two must agree exactly or the engine refuses to start:

            quant_method = quant_method if quant_method != "" else None
            # Verify quantization configurations.
            if self.quantization is None:
                self.quantization = quant_method
            elif self.quantization != quant_method:
                raise ValueError(
                    "Quantization method specified in the model config "
                    f"({quant_method}) does not match the quantization "
                    f"method specified in the `quantization` argument "
                    f"({self.quantization})."
                )

SGLang runs the same shape with one extra escape hatch: a compatible_quantization_methods table lets a CLI value stand in for a checkpoint value it is known to subsume — modelopt_fp4 over a checkpoint declaring modelopt, w8a8_int8 over one declaring compressed-tensors (python/sglang/srt/configs/model_config.py:L1497-L1508). vLLM has no such table; it routes every equivalent case through override_quantization_method instead. The tradeoff is legibility versus locality: vLLM's rule lives in each config class and needs the hard-coded ordering above to stay deterministic; SGLang's lives in one dict you can read at a glance but has to be updated by hand for each new pair.

Then the per-layer step. compressed-tensors is the most general scheme because its checkpoint carries config_groups: a list of (targets, weights, input_activations) triples, each target a layer name, a regex, or a module class name. One checkpoint can therefore be W4A16 on attention, W8A8 on the MLP, and unquantized on lm_head. _quantization_scheme_map_from_config flattens that into a target-to-scheme dict (compressed_tensors.py:L299-L369), and at layer construction get_scheme_dict matches the layer's prefix against the targets (compressed_tensors.py:L934-L968). If nothing matches, or the layer is in ignore, the method returns None and the layer falls back to UnquantizedLinearMethod — the "this layer stays bf16" leaf in Figure 4. If something does match, _get_scheme_from_parts dispatches through an ordered chain of predicates — NVFP4, MXFP4, MXFP8, W4A8-FP8, WNA8O8, WNA8, WNA4, WNA16, then the FP8 W8A8/W8A16 pair and the INT8 static-tensor, dynamic-token and W4A8 cases (compressed_tensors.py:L722-L883) — and returns one of the twelve scheme modules in compressed_tensors/schemes/.

Quantization is lossy and the loss is not uniformly distributed across tasks. Three published results, none of them mine, none of them measurable on the hardware I have:

Aggregate accuracy hides the damage. Accuracy is Not All You Need (arXiv:2407.09141, NeurIPS 2024) evaluated six quantization schemes and found aggregate benchmark accuracy within 2% of baseline in nearly every case — while a large fraction of individual answers changed. They call these flips, and report a Spearman correlation of 0.981 between flip rate and KL divergence from the baseline model's output distribution on MMLU. A quantized model can score the same and behave differently on the requests you actually serve.

Reasoning and long-generation tasks degrade first. Quantization Hurts Reasoning? (arXiv:2504.04823, COLM 2025) swept weight, KV and activation quantization across DeepSeek-R1-distilled Qwen and Llama from 1.5B to 70B plus QwQ-32B, on AIME, MATH-500, GPQA and LiveCodeBench. W8A8 and W4A16 are near-lossless; below that, degradation is severe and task-dependent, with mathematical reasoning hit hardest. The mechanism is compounding: a long chain of thought gives per-token error many more opportunities to change a downstream token.

The format matters less than people assume, at 8 bits. arXiv:2411.02355 reports FP8 W8A8 as effectively lossless at every model scale in the Llama-3.1 family, well-tuned INT8 W8A8 at 1–3% degradation, and INT4 weight-only more competitive than expected.

Neither this book nor either repository will hand you an accuracy table for your model. The closest thing that ships with the code is vLLM's CI thresholds under tests/evals/, which are lower bounds a maintainer picked, not measurements.

Enumerate both registries yourself and diff them — this is the table above, regenerated at whatever SHA you are on:

python -c "from vllm.model_executor.layers.quantization import QUANTIZATION_METHODS as M; print(len(M)); print('\n'.join(sorted(M)))"

python -c "from sglang.srt.layers.quantization import QUANTIZATION_METHODS as M; print(len(M)); print('\n'.join(sorted(M)))"

python -c "from vllm.config.cache import CacheDType; from typing import get_args; print(get_args(CacheDType))"

Then check what a checkpoint actually declares before you download 40 GB of it:

huggingface-cli download <repo> config.json --local-dir /tmp/cfg
python -c "import json; c=json.load(open('/tmp/cfg/config.json'))['quantization_config']; print(c.get('quant_method')); print(json.dumps({k:v for k,v in c.items() if k!='config_groups'}, indent=2)); [print(g['targets'], g['weights'].get('num_bits'), g['weights'].get('strategy'), g['weights'].get('group_size'), g['weights'].get('symmetric'), 'act:', (g.get('input_activations') or {}).get('strategy'), (g.get('input_activations') or {}).get('dynamic')) for g in c.get('config_groups', {}).values()]"

That last line prints one row per config group: bits, strategy, group size, symmetry, and whether activations are quantized and dynamically scaled. It is the six axes of this chapter, read straight off the artefact.

• "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization — arXiv:2411.02355 (ACL 2025). Over 500,000 evaluations across the Llama-3.1 family; the source of the W4A16-for-synchronous / W8A8-for-asynchronous recommendation that the roofline in §4.1.4 reproduces from first principles.
• Accuracy is Not All You Need — arXiv:2407.09141 (NeurIPS 2024). Introduces flips and shows they correlate with KL divergence at $\rho = 0.981$ on MMLU. Read this before you trust any aggregate accuracy delta.
• Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models — arXiv:2504.04823 (COLM 2025), code at ruikangliu/Quantized-Reasoning-Models. The task-dependence result: mathematical reasoning degrades first and hardest.
• vLLM docs — docs/features/quantization/ at this SHA carries one page per method (auto_awq.md, gptqmodel.md, modelopt.md, quark.md, torchao.md, inc.md, b12x.md) plus online.md for the shorthand schema and quantized_kvcache.md for the KV axis.
• compressed-tensors — the checkpoint format that makes the design space addressable per layer. QuantizationArgs and QuantizationStrategy live in the external compressed-tensors package, not in either engine tree; vLLM's consumption of them is in vllm/model_executor/layers/quantization/compressed_tensors/.
• SGLang quantization guide — docs.sglang.io/advanced_features/quantization.html, linked from model_config.py's own requantization warning.