GPTQ, AWQ, SmoothQuant, and calibration

Quantize the weights of OPT-6.7B to INT8 per-channel and the model is fine. Quantize the activations of the same model to INT8 per-tensor and it produces garbage. That failure is not a bug in anyone's kernel; it is a property of the trained network, and Dettmers et al. measured it precisely in LLM.int8() (arXiv:2208.07339).

Their finding, stated in their numbers:

• Typical hidden-state values live in roughly [−3.5, 3.5]. Outlier features are 3–20× larger than the largest magnitude in any other feature dimension.
• They are not scattered. For a 6.7B transformer at sequence length 2048 there are about 150,000 outlier activations per sequence across the whole model — concentrated in only 6 hidden dimensions.
• They emerge with scale. "At around 6.7B parameters, a phase shift occurs, and all transformer layers and 75% of all sequence dimensions are affected by extreme magnitude features."
• They are load-bearing. Zeroing those outlier dimensions drops top-1 attention softmax probability mass by more than 20% and degrades validation perplexity by 600–1000%. Zeroing the same number of randomly chosen dimensions costs 0.3% of probability mass and about 0.1% perplexity.

SmoothQuant later reported the ratio even more starkly on OPT-scale models: activation outliers around 100× larger than most activation values, and — the fact everything in this chapter exploits — "if one channel has an outlier, it persistently appears in all tokens" (arXiv:2211.10438).

The engineering consequence is mechanical. A per-tensor INT8 quantizer sizes its step from the tensor maximum, so if one channel is 100× the rest then $\Delta = 100 \cdot \max|X_{\text{normal}}| / 127$ and every normal channel — 4090 of the 4096 in Llama-3-8B's residual stream — collapses into the bottom one or two integer codes. You spent 8 bits per value and learned almost nothing about 99.9% of them, while the outlier channel that actually matters is still quantized coarsely, because 8 bits split across a 100:1 range is 8 bits split across a 100:1 range.

Every method here starts from the same observation — weights are easy, activations are hard — and picks a different move.

The rescaling identity underneath AWQ and SmoothQuant is one line. For a linear layer $Y = XW$ with $X \in \mathbb{R}^{T \times C_{\text{in}}}$ and $W \in \mathbb{R}^{C_{\text{in}} \times C_{\text{out}}}$, pick any positive vector $s \in \mathbb{R}^{C_{\text{in}}}$. Then

exactly in real arithmetic. Runtime cost is zero only where the required transforms can be folded or fused without additional work. Nothing has been approximated. What changes is the numerical range of the two factors: $\hat{X}_{:,j} = X_{:,j}/s_j$ and $\hat{W}_{j,:} = s_j W_{j,:}$. Quantization error is a function of range, so this identity is a dial that moves error between the two operands. AWQ chooses scales to protect salient weights under group quantization. SmoothQuant trades activation range against weight range. Both can increase $s_j$ on large-activation channels; their objectives and quantized operands differ, not necessarily the numerical direction of scaling. That is the whole chapter in one equation.

GPTQ (arXiv:2210.17323) does not touch activations at all. It solves a different problem: given that you are going to round $W$ to a 3- or 4-bit grid anyway, what is the best possible rounding?

"Best" needs a definition, and GPTQ's is the layer-wise reconstruction objective. For one linear layer with weight $W$ and a calibration input batch $X \in \mathbb{R}^{C_{\text{in}} \times N}$ ($N$ = total calibration tokens):

Note what this is not: it is not $\|W - \hat{W}\|^2$. Round-to-nearest minimises that, and RTN is the baseline GPTQ beats. Minimising the output error instead means an individual weight is allowed to move far from its original value if doing so cancels error elsewhere — provided the layer's output on real data stays close.

Where the Hessian comes from

Expand the objective as a function of the perturbation $E = \hat{W} - W$. Since $\|EX\|_F^2 = \operatorname{tr}(E X X^\top E^\top)$, the objective is exactly quadratic in $E$ with Hessian

This is the entire reason GPTQ needs calibration data: $H$ is the second-moment matrix of the layer's inputs. It says which input channels are large and which are correlated. Nothing about the loss function, the labels, or backpropagation enters — the Hessian here is of the local reconstruction objective, not of the training loss, which is why one forward pass over a few hundred sequences suffices.

Because $H$ is shared across all output rows (every row of $W$ sees the same $X$), the problem decouples into $C_{\text{out}}$ independent problems over the $C_{\text{in}}$ input channels, all sharing one $H^{-1}$. That sharing is what makes GPTQ affordable.

The Optimal Brain Quantization update

GPTQ inherits its update rule from Optimal Brain Surgeon by way of Optimal Brain Quantization. Quantize weights one input-channel (column) at a time. When you quantize column $q$, you incur error $w_q - \mathrm{quant}(w_q)$; the optimal correction spreads that error over the columns not yet quantized, weighted by the inverse Hessian:

Read the second equation as: the scalar rounding residual, normalised by how "free" that coordinate is ($[H^{-1}]_{qq}$ large means the objective barely cares about $q$), smeared over the remaining coordinates in proportion to the inverse-Hessian column. After the update the remaining columns are no longer the original weights — they have absorbed a debt that they will pay off when their turn comes. This is why a column-by-column sweep beats rounding everything at once: each column gets to correct for its predecessors.

GPTQ's three practical departures from OBQ:

1. Fixed order. OBQ picks the greedily-best next weight per row, which forces a different $H^{-1}$ per row. GPTQ observes that for large models a fixed column order costs almost nothing, so all rows can be quantized in lockstep against one shared $H^{-1}$. This turns per-weight bookkeeping into batched matrix work.
2. Dampening. $XX^\top$ from a finite calibration batch is often near-singular. GPTQ adds $\lambda I$ with $\lambda$ set to 1% of the mean diagonal before inverting.
3. Cholesky. The naive loop repeatedly removes a row/column from $H^{-1}$ and re-inverts, which accumulates error until the matrix goes indefinite on a 175B model. Instead, note that the sequence of updated inverse-Hessian information can be represented by an upper Cholesky factor of the damped inverse. Its rows are not the unchanged inverse-Hessian rows: the residual is divided by the triangular diagonal before a rank-one update over remaining columns. Compute that factor once, up front, with a numerically stable routine, then just read rows off it. The update becomes a rank-1 outer product against a precomputed triangular factor — stable and fast.

Cost, with real shapes

Take Llama-3-8B ($L=32$, $d=4096$, FFN intermediate 14336). The Hessian is $C_{\text{in}} \times C_{\text{in}}$ in fp32, so it is the input dimension that hurts:

• q_proj, gate_proj, up_proj: $C_{\text{in}} = 4096 \Rightarrow$ $4096^2 \times 4 = 67$ MB per Hessian, Cholesky $\approx 4096^3/3 = 2.3\times10^{10}$ flops.
• down_proj: $C_{\text{in}} = 14336 \Rightarrow$ $14336^2 \times 4 = 822$ MB, Cholesky $\approx 9.8\times10^{11}$ flops.

Derived from the shapes, not measured — but they explain GPTQ's operational shape: it runs layer-by-layer with one transformer block resident, and the Hessians are the memory peak. The reported wall-clock for OPT-175B and BLOOM-176B is roughly 4 GPU hours at 3–4 bits (GPTQ paper, calibrating on 128 random 2048-token segments of C4).

act-order, and what it costs at inference

The --act-order heuristic (serialised as desc_act in checkpoints) quantizes columns in order of decreasing $\mathrm{diag}(H)$ — that is, decreasing calibration activation energy. The logic is scheduling: a column quantized early has thousands of remaining columns to absorb its error; a column quantized last has none. So do the high-energy, high-cost columns first. The GPTQ repository reports it as decisive on the outlier-heavy OPT-66B: WikiText-2 perplexity 9.55 → 9.34 at 4 bits and 14.16 → 9.95 at 3 bits (IST-DASLab/gptq README).

The bill arrives at serving time. With group-wise scales, the group a weight belongs to is determined by its position along the input axis. Permute the columns and the group boundaries no longer align with contiguous input channels, so the checkpoint must ship a per-input-channel group index g_idx, and the GEMM must gather scales through it. vLLM stores exactly that:

        # Quantized weights
        qweight = PackedvLLMParameter(
            data=torch.empty(
                input_size_per_partition // self.quant_config.pack_factor,
                output_size_per_partition,
                dtype=torch.int32,
            ),
            input_dim=0,
            output_dim=1,
            packed_dim=0,
            packed_factor=self.quant_config.pack_factor,
            weight_loader=weight_loader,
        )

        # Activation order
        g_idx = RowvLLMParameter(
            data=torch.empty(
                input_size_per_partition,
                dtype=torch.int32,
            ),
            input_dim=0,

and at weight-prep time sorts it so the Marlin kernel can walk groups contiguously, keeping the permutation around as a separate tensor:

def marlin_is_k_full(act_order: bool, is_row_parallel: bool) -> bool:
    return (not act_order) or (act_order and not is_row_parallel)


def marlin_repeat_scales_on_all_ranks(
    act_order: bool, group_size: int, is_row_parallel: bool
) -> bool:
    # Need to repeat scales on every rank if act_ordering or
    # channelwise and RowParallelLinear
    is_channelwise = group_size == -1
    return act_order or (is_channelwise and is_row_parallel)


def marlin_make_empty_g_idx(device: torch.device) -> torch.Tensor:
    return torch.nn.Parameter(
        torch.empty(0, dtype=torch.int, device=device), requires_grad=False
    )


def marlin_sort_g_idx(g_idx: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
    g_idx_sort_indices = torch.argsort(g_idx).to(torch.int)
    return g_idx[g_idx_sort_indices], g_idx_sort_indices

Two real costs are visible in those two predicates. marlin_repeat_scales_on_all_ranks returns True whenever act-order is on: because the permutation scrambles which input channels land on which tensor-parallel rank, every rank must hold the full scale tensor rather than its shard. And marlin_is_k_full returns False for an act-order layer inside a RowParallelLinear, telling the kernel it is only seeing a slice of the reduction axis. Turning on desc_act is therefore not free at TP>1 — it costs replicated scales and a constrained kernel path.

Which is why llm-compressor offers a third setting, actorder="weight" — order the columns by weight statistics instead, so the permutation folds back into the stored layout and no g_idx ships at all. It "can improve accuracy without added latency", and it sits next to dampening_frac — the $\lambda$ of the previous section — as the two knobs the docs single out for tuning (docs/features/quantization/llm_compressor/int4.md:L136-L138).

vLLM's compressed-tensors path only materialises g_idx for the group variant:

        self.num_bits = num_bits
        self.pack_factor = Fraction(32, num_bits)
        self.strategy = strategy
        self.symmetric = symmetric
        self.group_size = -1 if group_size is None else group_size
        self.has_g_idx = actorder == ActivationOrdering.GROUP
        self.layer_name = layer_name

AWQ (arXiv:2306.00978) starts from a diagnostic experiment rather than an objective. Quantize OPT-6.7B to INT3 with group size 128 by round-to-nearest and keep just 1% of weight channels in FP16. Which 1%?

Halving the perplexity gap by choosing 1% of channels differently is the finding, and the direction is the counter-intuitive part: a large weight is not important, a weight that multiplies a large activation is. It falls straight out of §1 — a persistent 100× activation channel means every weight in that input row contributes 100× more to the output, so its rounding error is amplified 100× too.

From mixed precision to pure INT4

Keeping 1% of channels in FP16 works but is a serving disaster: mixed dtypes inside one GEMM destroy the tiling. AWQ's contribution is achieving the same protection with a uniform INT4 weight, using the rescaling identity from §2.

Consider one weight $w$ multiplied by activation $x$. Quantize with step $\Delta$ (set by the group's max): $Q(w) = \Delta \cdot \mathrm{round}(w/\Delta)$, error bounded by $\Delta/2$, so the expected output error is $\mathbb{E}|\mathrm{RoundErr}| \cdot \Delta \cdot |x|$. Now scale: quantize $w s$ instead and divide the activation, giving $Q(ws)\cdot(x/s)$. The output error becomes

where $\Delta'$ is the group's step after scaling. The whole trick is that $\Delta$ is set by the group maximum, and scaling up one channel out of 128 usually does not change that maximum — so $\Delta' \approx \Delta$ and the error on the scaled channel falls by $1/s$. You bought protection for the salient channel out of the slack in the group's shared scale.

The catch, and the reason $s$ cannot simply be huge: if you scale enough channels enough, $\Delta'$ does grow, and every non-salient channel in the group pays. AWQ therefore does not solve for $s$ per channel. It constrains $s$ to a one-parameter family driven by the measured per-channel mean activation magnitude $s_X \in \mathbb{R}^{C_{\text{in}}}$:

and finds $\alpha^\star$ by a grid search over 20 points in $[0,1]$, per layer. $\alpha = 0$ is plain RTN; $\alpha = 1$ is full activation-proportional scaling. Each grid point costs one quantize-and- reconstruct pass over a cached calibration activation for that layer, which is why AWQ is cheap: no Hessian, no inverse, no Cholesky. It needs the calibration data only to compute one vector of per-channel activation means and to score 20 candidates.

Reported outcome, configuration stated: Llama-2-7B at INT4 group-128 reaches WikiText-2 perplexity 5.60 against RTN's 5.73; OPT-6.7B at INT3-g128 is the 23.54 → 11.39 collapse above. The paper's serving companion TinyChat reports 3.2–3.3× over an FP16 HuggingFace baseline — a different framework from the ones this book reads, so treat it as evidence that W4A16 can be made fast, not as a vLLM number.

GPTQ and AWQ both leave activations in bf16, which is fine for W4A16 where the win is weight bandwidth. It is useless if you want W8A8 — INT8 tensor cores, roughly 2× the bf16 math throughput on an H100, and half the activation traffic. For that you must quantize $X$, and §1 said that is where the model breaks.

SmoothQuant takes the identity of §2 and points it at the activations. Its choice of $s$ is a geometric interpolation between the two operands' per-channel maxima:

Substitute back and the arithmetic is clean. The smoothed activation channel maximum becomes $\max|X_j| / s_j = \max|X_j|^{1-\alpha}\max|W_j|^{\alpha}$, and the smoothed weight channel maximum becomes $s_j \max|W_j| = \max|X_j|^{\alpha}\max|W_j|^{1-\alpha}$. At $\alpha = 0.5$ these are the same number: the geometric mean. The difficulty has been split exactly in half. That is the picture in Figure 1, and it is why the 100:1 activation spread becomes 10:1.

$\alpha$ is the dial:

• $\alpha = 0$: $s_j = 1/\max|W_j|$ — all difficulty pushed onto activations. Useless here.
• $\alpha = 0.5$: the balanced point, and SmoothQuant's default across OPT and BLOOM.
• $\alpha = 0.75$: used for GLM-130B, whose activation outliers are more extreme (about 30% of channels affected), pushing more difficulty into the weights.
• $\alpha \to 1$: activations become trivially quantizable and the weights inherit the outlier structure — now the weights are the ones you cannot represent.

Why it costs nothing at runtime

$\mathrm{diag}(s)^{-1}$ is a per-input-channel rescale of $X$, and $X$ is always the output of something — an RMSNorm, or a previous linear. A per-channel scale on an RMSNorm output is exactly a rescale of its learned gain vector; on a linear output it is a rescale of that linear's output columns and bias. So SmoothQuant folds $\mathrm{diag}(s)^{-1}$ backwards into the previous operator's stored weights and $\mathrm{diag}(s)$ forwards into $W$, offline. The published checkpoint contains no smoothing factors and the engine executes exactly the same graph it always did.

This is why grep -rni smooth over vllm/model_executor/layers/quantization/ and python/sglang/srt/layers/quantization/ at the pinned SHAs returns nothing about SmoothQuant. There is no SmoothQuant runtime. There is only a W8A8 INT8 checkpoint whose activations happen to be quantizable, which lands in the ordinary INT8 schemes:

        # WEIGHT
        weight = ModelWeightParameter(
            data=torch.empty(
                sum(output_partition_sizes), input_size_per_partition, dtype=torch.int8
            ),
            input_dim=1,
            output_dim=0,
            weight_loader=weight_loader,
        )

        layer.register_parameter("weight", weight)

        # WEIGHT SCALE
        if self.strategy == QuantizationStrategy.CHANNEL:
            weight_scale = ChannelQuantScaleParameter(
                data=torch.empty((sum(output_partition_sizes), 1), dtype=torch.float32),
                output_dim=0,
                weight_loader=weight_loader,
            )
        else:
            assert self.strategy == QuantizationStrategy.TENSOR
            weight_scale = PerTensorScaleParameter(
                data=torch.empty(len(output_partition_sizes), dtype=torch.float32),
                weight_loader=weight_loader,
            )
        layer.register_parameter("weight_scale", weight_scale)

        # INPUT SCALE
        input_zero_point = None
        input_scale = None

An INT8 weight, a per-output-channel fp32 scale, and an optional input scale — present only when the checkpoint declares static activation quantization. Leave it out and the engine quantizes each token's activation vector on the fly from its own amax, which is what SGLang always does on GPU:

        x_q, x_scale = per_token_quant_int8(x)

        x_q_2d = x_q.view(-1, x_q.shape[-1])
        x_scale_2d = x_scale.view(-1, x_scale.shape[-1])
        output_shape = [*x_q.shape[:-1], layer.weight.shape[1]]

        output = int8_scaled_mm(
            x_q_2d,
            layer.weight,
            x_scale_2d,
            layer.weight_scale,
            out_dtype=x.dtype,
            bias=bias,
        )

        return output.view(output_shape)

Per-token dynamic scaling already removes the token-to-token variation. What it cannot remove is the channel-to-channel spread inside a single token — a per-token scale is one number for all 4096 channels of that token. SmoothQuant is precisely the missing half: it fixes the channel axis offline so that the per-token amax the kernel computes at runtime is a useful scale rather than a hostage to six dimensions.

The reference recipe in vLLM's own docs shows the two algorithms composed, SmoothQuant first to make the activations quantizable and GPTQ second to place the weights:

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier

# Configure the quantization algorithms
recipe = [
    SmoothQuantModifier(smoothing_strength=0.8),
    GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),
]

smoothing_strength=0.8 is $\alpha$. Note it is higher than the paper's 0.5 default — the recipe is tuned for modern Llama-family models, and $\alpha$ is the first thing to sweep when a W8A8 conversion degrades.

All three methods need data, but for different things, and confusing them leads to the wrong debugging move.

GPTQ's requirement is the heaviest in shape but the most forgiving in content: $XX^\top$ over 262,144 tokens (128 sequences × 2048, the GPTQ paper's C4 setting) is a well-averaged object. SmoothQuant's is the most fragile — a per-channel max estimated from a sample. If your deployment traffic activates a channel your calibration corpus never did, that channel's $s_j$ was computed as if it were ordinary, and it arrives at the INT8 quantizer unsmoothed.

vLLM's documented defaults are 512 sequences at length 2048 from an instruction corpus, with the model's own chat template applied:

from datasets import load_dataset

NUM_CALIBRATION_SAMPLES = 512
MAX_SEQUENCE_LENGTH = 2048

# Load and preprocess the dataset
ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))

def preprocess(example):
    return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
ds = ds.map(preprocess)

and the accompanying advice is unusually direct about the failure mode:

- Start with 512 samples for calibration data, and increase if accuracy drops
- Ensure the calibration data contains a high variety of samples to prevent overfitting towards a specific use case
- Use a sequence length of 2048 as a starting point
- Employ the chat template or instruction template that the model was trained with
- If you've fine-tuned a model, consider using a sample of your training data for calibration

Two of those five bullets are about distribution match, which is where the silent failures live. The canonical shape: calibrate on WikiText because every paper does, evaluate on WikiText perplexity, see a 0.1 delta, ship. Actual traffic is code completion and function calling — a token distribution with different activation statistics, different salient channels, and a different per-channel max. Nothing in that eval could have caught it, because the eval was drawn from the calibration distribution.

The published evidence is real, and its size depends on the method. Williams and Aletras (ACL 2024, arXiv:2311.09755) ran nine models (LLaMA/Vicuna/OPT at three sizes) across five corpora — C4, CNN-DM, RedPajama, RefinedWeb, Wikipedia — with ten independently drawn 128×2048 calibration sets each. Their headline: "we find substantial variations in downstream task performance, contrasting existing work that suggests a greater level of robustness". On LLaMA-7B with SparseGPT (2:4 pruning), accuracy ranged from 52.7% to 61.7% on RTE and 66.4% to 73.0% on BoolQ across sets drawn from the same corpus. For 4-bit quantization the spread is smaller — OPT-6.7B varied about 1.6 points with GPTQ, 0.9 with SpQR — but that is the same order as the accuracy delta most teams accept when they decide to ship.

The algorithms are offline. Nothing in vLLM or SGLang runs GPTQ or AWQ. What the engines see is a directory of tensors, and their job is to reconstruct a GEMM from it.

The W4A16 artefact, sized

Take Llama-3-8B's gate_proj: $4096 \times 14336 = 58{,}720{,}256$ weights. At INT4 with group size 128, the checkpoint holds (all figures derived from the shapes and the layouts quoted below):

The 0.16 bits of overhead is the group metadata. Halve the group size to 64 and it doubles; go channel-wise ($\texttt{group\_size} = -1$) and it nearly vanishes, at a real accuracy cost. That tradeoff is the granularity axis of §0.5, priced.

The packing, concretely

Eight INT4 values per int32. vLLM's generic packer:

def pack_quantized_values_into_int32(
    w_q: torch.Tensor, wtype: ScalarType, packed_dim: int = 0
):
    # move dim to pack to the end
    perm = (*[i for i in range(len(w_q.shape)) if i != packed_dim], packed_dim)
    inv_perm = tuple(perm.index(i) for i in range(len(perm)))
    w_q_perm = w_q.permute(perm)

    pack_factor = 32 // wtype.size_bits
    mask = (1 << wtype.size_bits) - 1

    new_shape_perm = list(w_q_perm.shape)
    assert w_q_perm.shape[-1] % pack_factor == 0
    new_shape_perm[-1] //= pack_factor

    res = torch.zeros(new_shape_perm, dtype=torch.int32, device=w_q.device)
    for i in range(pack_factor):
        res |= (w_q_perm[..., i::pack_factor] & mask) << wtype.size_bits * i

    return res.permute(inv_perm)

GPTQ and AWQ do not agree on the layout. GPTQ packs along the input axis in natural nibble order. AWQ packs along the output axis in the interleaved order [0, 4, 1, 5, 2, 6, 3, 7] — a choice made so that a dequant kernel can unpack eight nibbles into two aligned halves with fewer shuffles. That order is hard-coded in both engines' dequant paths. vLLM's Triton reference kernel reconstructs it arithmetically — it is not the default path, which is the compiled torch.ops._C.awq_dequantize; the Triton version is selected only by VLLM_USE_TRITON_AWQ=1 (vllm/_custom_ops.py:L491-L505), which makes it the readable statement of a layout both paths share:

    # Load the weights.
    iweights = tl.load(qweight_ptr + offsets, masks, 0.0)
    iweights = tl.interleave(iweights, iweights)
    iweights = tl.interleave(iweights, iweights)
    iweights = tl.interleave(iweights, iweights)

    # Create reverse AWQ order as tensor: [0, 4, 1, 5, 2, 6, 3, 7]
    # that will map given indices to the correct order.
    reverse_awq_order_tensor = (
        (tl.arange(0, 2) * 4)[None, :] + tl.arange(0, 4)[:, None]
    ).reshape(8)

    # Use this to compute a set of shifts that can be used to unpack and
    # reorder the values in iweights and zeros.
    shifts = reverse_awq_order_tensor * 4
    shifts = tl.broadcast_to(shifts[None, :], (BLOCK_SIZE_Y * BLOCK_SIZE_X, 8))
    shifts = tl.reshape(shifts, (BLOCK_SIZE_Y, BLOCK_SIZE_X * 8))

    # Unpack and reorder: shift out the correct 4-bit value and mask.
    iweights = (iweights >> shifts) & 0xF

Three tl.interleave calls turn each int32 lane into eight copies of itself; the shift vector then extracts a different nibble from each copy, in AWQ's order. The dequantization itself is one line further down — iweights = (iweights - zeros) * scales, the affine map of §0.5 — and that is the entire runtime cost of AWQ. There is no AWQ scaling at inference. The scales were folded into the weights offline, just like SmoothQuant's.

Where the two engines diverge

At the pinned SHAs the structural difference is that vLLM has collapsed the format-specific configs into two, deferring kernel choice to a selector, while SGLang keeps a config class per (format, kernel) pair:

    method_to_config: dict[str, type[QuantizationConfig]] = {
        "awq": AutoAWQConfig,
        "awq_marlin": AutoAWQConfig,
        "auto_awq": AutoAWQConfig,
        "fp8": Fp8Config,
        "fbgemm_fp8": FBGEMMFp8Config,
        "fp_quant": FPQuantConfig,
        "modelopt": ModelOptFp8Config,
        "modelopt_fp4": ModelOptNvFp4Config,
        "modelopt_mxfp8": ModelOptMxFp8Config,
        "modelopt_mixed": ModelOptMixedPrecisionConfig,
        "auto_gptq": AutoGPTQConfig,
        "gptq": AutoGPTQConfig,
        "gptq_marlin": AutoGPTQConfig,

    "awq": AWQConfig,
    "awq_marlin": AWQMarlinConfig,
    "bitsandbytes": BitsAndBytesConfig,
    "gguf": GGUFConfig,
    "gptq": GPTQConfig,
    "gptq_marlin": GPTQMarlinConfig,

The consequence is behavioural. In vLLM, --quantization awq and --quantization awq_marlin construct the same object and AutoAWQConfig.get_quant_method picks Marlin or the fallback per layer, via check_marlin_supported and check_marlin_supports_layer. In SGLang the flag picks the class, and therefore the kernel family, for the whole model. One layer Marlin cannot serve — after tile padding — degrades that layer alone in vLLM; in SGLang it means you should have chosen awq up front.

A second divergence, in the non-Marlin AWQ path. vLLM switches strategy on token count:

        qweight = layer.qweight
        scales = layer.scales
        qzeros = layer.qzeros
        pack_factor = self.quant_config.pack_factor
        out_shape = x.shape[:-1] + (qweight.shape[-1] * pack_factor,)
        reshaped_x = x.reshape(-1, x.shape[-1])

        # num_tokens >= threshold
        FP16_MATMUL_HEURISTIC_CONDITION = x.shape[:-1].numel() >= 256
        # Batch invariant mode requires torch.matmul path
        # for Triton override
        if FP16_MATMUL_HEURISTIC_CONDITION or envs.VLLM_BATCH_INVARIANT:
            out = ops.awq_dequantize(qweight, scales, qzeros, 0, 0, 0)
            out = torch.matmul(reshaped_x, out)
        else:
            out = ops.awq_gemm(reshaped_x, qweight, scales, qzeros, pack_factor)
        if bias is not None:
            out.add_(bias)
        return out.reshape(out_shape)

Below 256 tokens, fuse the dequant into a specialised GEMM — the layer is memory-bound on weights, so reading 4 bits instead of 16 is the whole win. At or above 256 tokens, materialise the full FP16 weight once and call cuBLAS, because the GEMM is now compute-bound and one dequant amortised over 256+ rows is cheaper than a hand-written kernel that cannot match cuBLAS on math throughput. This is exactly the ridge-point argument of §0.4 written as an if. SGLang's equivalent path has no such branch:

        qweight = layer.qweight
        scales = layer.scales
        qzeros = layer.qzeros
        pack_factor = self.quant_config.pack_factor
        out_shape = x.shape[:-1] + (qweight.shape[-1] * pack_factor,)
        reshaped_x = x.reshape(-1, x.shape[-1])
        out = awq_dequantize(qweight, scales, qzeros)
        out = torch.matmul(reshaped_x, out)

        if bias is not None:
            out.add_(bias)
        return out.reshape(out_shape)

Always dequantize, always torch.matmul. That is the wrong choice at batch 1 — you materialise a 117 MB fp16 tensor to multiply it by a single row — and it is why SGLang's non-Marlin AWQ path is a compatibility fallback rather than a serving path. Both engines expect Marlin to handle the real workload (§4.3).

SGLang resolves the act-order/tensor-parallel collision the other way — full scales replicated when act-order requires them, with the shuffle optimization disabled on that path:

        self.kernel.use_shuffle = True
        scale_and_zero_size = input_size // group_size
        scale_and_zero_input_dim = None
        if (
            input_size != input_size_per_partition
            and self.quant_config.group_size != -1
        ):
            if self.quant_config.desc_act:
                self.kernel.use_shuffle = False
            else:
                scale_and_zero_size = input_size_per_partition // group_size
                scale_and_zero_input_dim = 0

input_size != input_size_per_partition means a row-parallel layer whose reduction axis is sharded. Read the branch carefully: only the else arm — no desc_act — sets scale_and_zero_size = input_size_per_partition // group_size and scale_and_zero_input_dim = 0, which is what makes the scales a shard. Under desc_act both stay at their defaults (input_size // group_size and None), so SGLang also replicates the full scale and zero-point tensors on every rank — and additionally sets use_shuffle = False. Same problem, strictly worse concession: vLLM pays replicated scales and a is_k_full=False kernel path, SGLang pays replicated scales and loses the shuffle optimisation.

One gate_proj of a 4-bit AWQ Llama-3-8B, from quant_config.json to the kernel call, in vLLM at a556f3f.

1. AutoAWQConfig.from_config (auto_awq.py:L238-L253) reads w_bit/bits, q_group_size/group_size, zero_point, and modules_to_not_convert, and sets pack_factor = 32 // weight_bits = 8. Note it accepts either key spelling — AutoAWQ and llm-compressor disagree on the field names.
2. AutoAWQConfig.get_quant_method (auto_awq.py:L285-L331) checks is_layer_skipped against modules_to_not_convert, then check_marlin_supported(self.quant_type, self.group_size, self.zero_point). For uint4 with group 128 and zero points on an H100 this passes, so the layer gets AutoAWQMarlinLinearMethod. It is called with allow_tile_padding=True, so tile-misalignment alone is fixed by padding at weight prep rather than rejected; if check_marlin_supports_layer still fails it logs "Layer '%s' is not supported by AutoAWQMarlin. Falling back to unoptimized AWQ kernels." and returns AutoAWQLinearMethod instead.
3. create_weights (auto_awq.py:L453-L491) allocates the checkpoint layout, not the kernel layout: qweight is (4096, 14336//8) = (4096, 1792) int32 with packed_dim=1 — packed along the output axis, the AWQ convention. qzeros is (32, 1792), scales is (32, 14336).
4. Weights load. Nothing is transformed yet; the parameter classes only route shards.
5. process_weights_after_loading → _convert_awq_to_standard_format (auto_awq.py:L93-L168) unpacks every int32 into eight nibbles, undoes the [0,4,1,5,2,6,3,7] interleave, and repacks along the input axis so the layout matches GPTQ's:

    # --- Convert qweight: (K, N // pack) packed_dim=1 → (K // pack, N) packed_dim=0
    qw = getattr(layer, w_q_name).data
    K, N_packed = qw.shape
    N = N_packed * pack_factor

    # Unpack int32 → individual values, fix AWQ ordering
    unpacked = (qw.unsqueeze(-1) >> shifts) & mask  # (K, N_packed, pack_factor)
    unpacked = unpacked[:, :, reverse_order]
    unpacked = unpacked.reshape(K, N)  # (K, N)

    # Repack along input dim (dim 0)
    unpacked = unpacked.reshape(K // pack_factor, pack_factor, N)
    new_qw = (unpacked.to(torch.int32) << shifts[None, :, None]).sum(
        dim=1, dtype=torch.int32
    )

6. Marlin repack. MarlinLinearKernel.process_weights_after_loading (vllm/model_executor/kernels/linear/mixed_precision/marlin.py:L128-L146) calls ops.gptq_marlin_repack to shuffle into Marlin's tile order, and marlin_permute_scales for the scales. AWQ has no g_idx, so marlin_make_empty_g_idx supplies zero-length placeholders — the kernel signature requires the tensors to exist.
7. Forward. apply_weights calls apply_gptq_marlin_linear:

        c = self.config
        w_q, w_s, w_zp, w_gidx = self._get_weight_params(layer)

        # `process_weights_after_loading` will ensure w_zp and w_gidx are not
        #  None for marlin

        return apply_gptq_marlin_linear(
            input=x,
            weight=w_q,
            weight_scale=w_s,
            weight_zp=w_zp,  # type: ignore
            g_idx=w_gidx,  # type: ignore
            g_idx_sort_indices=layer.g_idx_sort_indices,
            workspace=self.workspace,
            wtype=c.weight_type,
            input_size_per_partition=c.partition_weight_shape[0],
            output_size_per_partition=c.partition_weight_shape[1],
            is_k_full=self.is_k_full,
            input_global_scale=getattr(layer, "input_global_scale", None),
            bias=bias,
            input_dtype=c.act_type,
        )

The GPTQ path is the same function with the same arguments; the only difference is that w_gidx and g_idx_sort_indices are non-empty when desc_act was set. Once the checkpoint is loaded, "GPTQ" and "AWQ" are the same kernel with different numbers in it. The algorithms differ entirely offline.

desc_act plus tensor parallelism

The most common surprise. A desc_act: true checkpoint at TP=1 is fine; at TP=8 the same file forces marlin_repeat_scales_on_all_ranks to True, so every rank stores the full scale tensor instead of its $1/8$ shard, and marlin_is_k_full goes False for every row-parallel layer. If a quantized model shows a smaller-than-expected memory win at high TP, check desc_act in quantize_config.json first.

Group size −1 silently disables act-order

vLLM normalises this on construction, with the reason in a comment:

        super().__init__()
        if desc_act and group_size == -1:
            # In this case, act_order == True is the same as act_order == False
            # (since we have only one group per output channel)
            desc_act = False

Correct — with one group there are no group boundaries to misalign — but it means a config claiming desc_act: true may produce a layer with no g_idx at all. Do not infer the runtime behaviour from the JSON.

Shapes that do not divide

SGLang raises a specific error when the output dimension is not a multiple of the pack factor after sharding:

        if output_size_per_partition % self.quant_config.pack_factor.numerator != 0:
            raise ValueError(
                "The output size is not aligned with the quantized "
                "weight shape. This can be caused by too large "
                "tensor parallel size."
            )

The message names the real cause. A 4-bit checkpoint packs 8 values per int32; if $C_{\text{out}}/\text{TP}$ is not a multiple of 8 the shard cannot be cut on an int32 boundary. Lower TP, or use a checkpoint whose fused QKV/MLP widths divide.

A W8A8 conversion that loses a few points

Symptom: GSM8K drops several points after an INT8 conversion that looked fine on perplexity. Debug order: (1) sweep smoothing_strength — 0.5, 0.65, 0.8, 0.9 — because $\alpha$ is the only knob that directly trades activation error for weight error; (2) confirm the calibration corpus resembles the eval; (3) check whether the eval needs add_bos_token=True, which vLLM's own docs flag as a source of spurious degradation in quantized models (docs/features/quantization/llm_compressor/int4.md:L126-L127). Only after those three is "INT8 is not viable for this model" a supportable conclusion.

Calibrating on the eval set

The quiet one. Disjoint representative calibration and evaluation samples from the same corpus are valid. Leakage arises from overlapping examples, near duplicates or tuning against the held-out test. Keep calibration, validation and final evaluation separate; add out-of-domain slices to measure robustness. Where the deployment domain is known, calibrate on it — vLLM's docs say this outright for fine-tuned models (docs/features/quantization/llm_compressor/int4.md:L135).

No GPU is needed to check every claim this chapter makes about stored artefacts — checkpoint headers expose shapes and dtypes without materializing full tensors. A sharded index maps tensor names to files; it does not contain all tensor shapes or prove runtime layouts.

pip install huggingface_hub
python - <<'PY'
from huggingface_hub import hf_hub_download
import json
repo = "TheBloke/Llama-2-7b-Chat-GPTQ"
print(json.load(open(hf_hub_download(repo, "quantize_config.json"))))
idx = hf_hub_download(repo, "model.safetensors.index.json")
keys = json.load(open(idx))["weight_map"]
for k in sorted(keys):
    if "layers.0." in k:
        print(k)
PY

Read off bits, group_size, desc_act, sym, then confirm the four tensors per linear: qweight, qzeros, scales, g_idx. Repeat for an AWQ repo (TheBloke/Llama-2-7b-Chat-AWQ, quant_config.json) and note the missing g_idx and the present zero_point. Then recompute the effective bits per weight from the shapes and check §7's 4.16 against a real file.

With a GPU, the flag to flip is the kernel selector. vLLM logs its choice ("Using %s for AutoAWQMarlinLinearMethod", auto_awq.py:L448-L450); launch with --quantization awq and again with --quantization awq_marlin and confirm from the log that both resolve to the same config and diverge only when the Marlin support checks fail. Latency consequences are §4.4.

For the calibration experiment, run the llm-compressor INT4 recipe twice with different ds.shuffle(seed=...) values and evaluate both with the documented harness:

lm_eval --model vllm \
  --model_args pretrained="./Meta-Llama-3-8B-Instruct-W4A16-G128",add_bos_token=true \
  --tasks gsm8k \
  --num_fewshot 5 \
  --limit 250 \
  --batch_size 'auto'

Two calibration seeds expose sensitivity but do not estimate a universal noise floor. Use multiple calibration draws and paired evaluation on identical held-out examples; report uncertainty for the algorithm difference and calibration variability separately.

• Dettmers, Lewis, Belkada, Zettlemoyer, LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (arXiv:2208.07339) — the emergent-outlier measurement that every later method cites. Section 4 is the one to read.
• Frantar, Ashkboos, Hoefler, Alistarh, GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv:2210.17323), and the reference implementation at IST-DASLab/gptq — the README's flag notes (--act-order, --true-sequential, --static-groups) are the best available documentation of what the serialised config fields mean.
• Lin, Tang, Tang, Yang, Dang, Han, AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (arXiv:2306.00978). Table 1 is the whole argument.
• Xiao, Lin, Seznec, Wu, Demouth, Han, SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (arXiv:2211.10438).
• Williams & Aletras, On the Impact of Calibration Data in Post-training Quantization and Pruning (ACL 2024, arXiv:2311.09755) — the systematic study of calibration-set variance across 9 models, 5 corpora, 4 methods.
• Ashkboos et al., QuaRot (arXiv:2404.00456) and Liu et al., SpinQuant (arXiv:2405.16406) — the rotation-based alternative to rescaling; see §2.5 for the transform itself.
• vllm-project/llm-compressor — where AutoAWQ and AutoGPTQ workflows have been consolidated; vLLM's own docs/features/quantization/auto_awq.md:L3-L5 marks AutoAWQ deprecated in favour of it.