GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Concepts: quantization | inference-efficiency | memory-efficiency | compression Leads to: awq-activation-aware-weight-quantization — AWQ replaces GPTQ’s Hessian-based reconstruction with activation-magnitude-guided scaling for similar accuracy at lower cost Leads to: qlora-efficient-finetuning-quantized-llms — QLoRA quantizes models to NF4 for fine-tuning; GPTQ established that aggressive quantization could be done without retraining

In late 2022, the only way to run a 175B-parameter GPT-class model was a multi-GPU server with hundreds of GB of FP16 weights. Most quantization work to that point had focused on smaller models — and existing 8-bit methods (LLM.int8(), SmoothQuant) saved memory but still needed multiple GPUs. GPTQ (Frantar et al., ICLR 2023) did something the field didn’t think was possible at this scale: quantize OPT-175B to 4 bits per weight, in 4 GPU hours, with negligible perplexity loss — and as a corollary, fit the entire model in a single 80GB A100 for the first time.

The core idea

The analogy: Quantizing weights is like converting a high-resolution photo to a coarse-grained version. Round-to-nearest does this independently for every pixel. GPTQ recognizes that pixels are not independent — when you round one pixel down, you can compensate by adjusting nearby pixels, so the overall image looks closer to the original. The “compensation” is computed from a small sample of “what does this image look like through a typical viewing operation” — which, for an LLM, is “what does this weight look like when multiplied by typical activations.”

Three layers of insight, building on prior work:

1. Per-layer reconstruction. Quantize each linear layer independently. The objective: minimize the squared error between the original layer’s output $WX$ and the quantized layer’s output $\hat{W}X$ on a calibration set $X$. This is a classic problem — Optimal Brain Surgeon (OBS) solved it in 1993 — but at $O(d^4)$ per layer, it was infeasible for billion-parameter models.

2. Column-by-column quantization with lazy compensation. Instead of solving the full optimization, GPTQ walks through weight columns left to right. After quantizing column $j$, the unquantized columns $j+1,\dots$ still have the original values. Update them to compensate for the quantization error in column $j$ — this is where the second-order (Hessian) information enters.

3. Cholesky-based stable solution. The Hessian inverse update is numerically unstable for large matrices. The paper precomputes a Cholesky decomposition of the Hessian inverse, then uses the upper-triangular structure to do the column updates without inverting the full matrix at every step.

Walkthrough

The per-column quantization step (simplified):

For a single linear layer with weight W (out_dim x in_dim):

PRECOMPUTE: 
  H = X X^T  (in_dim x in_dim, the input covariance)
  H_inv = inverse(H + lambda * I)   (regularized inverse)
  L = Cholesky(H_inv)                 (upper triangular)

FOR each column j = 1 to in_dim:
  
  STEP 1: Quantize column j of W:
    w_j_quant = round(w_j / scale[j]) * scale[j]
    err_j = w_j - w_j_quant
  
  STEP 2: Compensate the unquantized columns j+1, ..., in_dim:
    For k = j+1 to in_dim:
      W[:, k] -= err_j * (L[j, k] / L[j, j])
    
    (This update propagates the quantization error of column j
     into adjustments to the still-unquantized columns, in a way
     that minimizes the output reconstruction error.)
  
  STEP 3: Move on to column j+1. It now incorporates the
          compensation from columns 1..j.

After all columns are processed, every column has been quantized, and the cumulative error has been distributed across the weights such that the final output $\hat{W}X$ is close to $WX$ on the calibration distribution.

OPT-175B quantization budget:

Calibration set:    128 sequences (random C4 chunks).
Quantization time:  ~4 hours on a single A100 80GB.
                    (96 transformer layers, 6 linear layers each)
Memory overhead:    H matrix per layer = O(in_dim^2) = ~400 MB
                    for the largest layers; reused across columns.

Output:             OPT-175B at 4-bit weights, fits in 80GB.
                    Inference latency: ~3.25x faster than FP16
                    (memory-bound regime).

What’s clever — find the instinct

The first clever recognition: at the 175B scale, you cannot retrain or fine-tune to recover quantization loss. Each retraining run costs millions of dollars. So the entire approach has to be “one shot, no gradient.” This rules out most quantization-aware training methods. GPTQ commits to per-layer reconstruction with a fixed pretrained model, and accepts the constraint.

“We propose GPTQ, a new one-shot weight quantization method based on approximate second-order information.”

The second clever move: process columns sequentially, distributing error. This is the key insight from Optimal Brain Surgeon — when you delete (or quantize) a weight, the remaining weights can compensate. GPTQ generalizes this to the order-of-magnitude-larger setting by lazily updating only the columns that haven’t been quantized yet.

“We process the columns of W in arbitrary order, but it is more efficient to process them in arbitrary order while continuously updating only those weights that have not yet been quantized.”

The third clever move: Cholesky preconditioning. Direct manipulation of the Hessian inverse $H^{-1}$ for billion-parameter models is numerically catastrophic — the matrix is enormous and ill-conditioned. The paper precomputes the Cholesky factor $L$ such that $H^{-1}=L^{T}L$. The column-update formula then uses only the upper triangular part of $L$, which is stable and parallelizable.

“We propose a Cholesky reformulation that is significantly more numerically stable.”

The fourth (less talked about) clever move: 128 calibration sequences is enough. Most quantization research assumes you need orders of magnitude more calibration data. The paper shows that a few hundred sequences sampled from C4 is sufficient — quantization is fundamentally a low-data problem because you only need to characterize the input distribution to a coarse approximation.

Does it work? What breaks?

OPT-175B perplexity (lower is better):

Bits	Method	WikiText-2 PPL	C4 PPL
FP16	—	8.34	7.32
4	RTN (round-to-nearest)	110.2	109.6
4	GPTQ	8.34	7.40
3	RTN	1.4e4	8.0e3
3	GPTQ	8.68	7.54

At 4 bits, GPTQ is essentially lossless on perplexity. At 3 bits, RTN is catastrophic but GPTQ still recovers near-FP16 quality.

End-to-end inference speedup (A100, batch size 1):

Model	Format	Latency / token
OPT-175B	FP16	122 ms
OPT-175B	GPTQ INT3	38 ms (3.25x)
BLOOM-176B	FP16	119 ms
BLOOM-176B	GPTQ INT3	35 ms (3.4x)

The speedup comes from memory bandwidth — most LLM inference is memory-bound, so quantizing weights from 16 to 3 bits gives a near-linear speedup.

What breaks:

• Domain shift sensitivity. The Hessian is computed on calibration data. If your deployment domain (e.g., medical text) differs from the calibration domain (C4 web text), GPTQ can degrade by 2-5 PPL points. AWQ partially fixes this with activation-based scaling.
• INT2 is hard. At 2 bits, GPTQ degrades significantly (PPL doubles or worse). At this regime, methods like QuIP and SqueezeLLM that use rotation/codebook tricks do better.
• Activation quantization is separate. GPTQ only quantizes weights (W4A16). For W4A8 or W8A8 (integer-only inference for accelerators), you need additional methods (SmoothQuant).
• No support for fine-tuning the quantized model. Once quantized, the model is fixed. QLoRA later showed how to add LoRA adapters on top of a quantized base.

So what?

GPTQ was the breakthrough that made open-source LLM deployment practical. Before GPTQ, running LLaMA-65B locally required 130GB of FP16 weights — multiple GPUs or extreme CPU offload. After GPTQ INT4, the same model fit in 35GB, comfortably on two consumer GPUs or one server-grade A100. The lifelong-deployment story of llama.cpp, vLLM, TensorRT-LLM, and the entire local-LLM ecosystem starts here.

For Saikat’s career-gap target on large-scale model serving:

• Production rule of thumb: quantize before deploying. INT4-g128 with GPTQ or AWQ is the default. Test on a domain-representative validation set; if PPL gap > 5%, calibrate on in-domain data.
• Trade-off vs AWQ: AWQ has slightly better cross-domain generalization (uses activation magnitudes, less domain-tied than Hessian). GPTQ is sometimes more accurate for in-domain when calibrated correctly. Production today uses AWQ as the default; GPTQ remains a strong alternative when GPU compute is cheap and accuracy is critical.
• Pipeline-level recognition: quantization is one of three orthogonal axes of inference optimization (the others being attention engineering and serving-system efficiency). All three matter — read GPTQ alongside FlashAttention and PagedAttention.

The deeper principle GPTQ establishes: the cumulative error of greedy operations can be controlled by lazy compensation. This pattern shows up in numerical linear algebra (incomplete Cholesky), in compiler optimization (deferred passes), and in lossy compression (predictive coding). GPTQ brought it to LLM quantization with industrial impact.

“We are the first to show that an extremely accurate language model with hundreds of billions of parameters can be quantized to 3-4 bits/component.”

Connections

• awq-activation-aware-weight-quantization — AWQ replaces Hessian-based reconstruction with activation-magnitude scaling; faster and slightly more domain-robust
• qlora-efficient-finetuning-quantized-llms — QLoRA fine-tunes on top of a quantized base, addressing GPTQ’s “fixed once quantized” limitation
• quantization — GPTQ is the foundational PTQ method for LLMs
• inference-efficiency — 3-4x speedup on memory-bound inference
• memory-efficiency — 175B model in 80GB single-GPU memory
• compression — 4x compression with negligible quality loss

Citation

arXiv:2210.17323

Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023. https://arxiv.org/abs/2210.17323