GPTQ: How One-Shot Quantization Fits a 175B Model onto a Single GPU

Running a 175-billion-parameter language model in FP16 requires 326 GB of GPU memory, far more than any single card can hold. Before October 2022, the only practical solution was multi-GPU inference, spreading the weight tensors across eight or more high-end accelerators. Post-training quantization promised a way out: compress the weights to fewer bits after training, no costly retraining needed. The problem was that every post-training method accurate enough to be useful had only ever been demonstrated at 8-bit precision, and scaling those methods to models with hundreds of billions of parameters turned out to be prohibitively slow. The paper 'Accurate Post-Training Quantization for Generative Pre-trained Transformers,' published October 31, 2022 on arXiv (arXiv:2210.17323) and accepted at ICLR 2023, closed that gap. The authors, Elias Frantar, Saleh Ashkboos, and Torsten Hoefler at ETH Zurich, together with Dan Alistarh at IST Austria and NeuralMagic, introduced GPTQ: a quantization method that compresses a 175B model to 3 or 4 bits per weight in roughly four GPU hours, on a single machine, with a perplexity increase so small it barely registers on benchmarks.

The pre-GPTQ landscape: why 8 bits was the ceiling

By 2022, INT8 quantization for large language models was already in practical use. Tim Dettmers and colleagues at the University of Washington had published LLM.int8() (arXiv:2208.07339, August 2022), which used vector-wise quantization to keep most weights at INT8 while handling the small fraction of outlier weights at FP16. That approach worked well enough to run OPT-175B across two GPUs at 8-bit precision with no meaningful quality loss. But INT8 halves memory versus FP16. Reaching 4-bit would quarter it, letting a 175B model fit on two consumer GPUs or even one high-end data-center card. The existing methods that could achieve 4-bit accuracy were all training-based or used expensive iterative optimization loops that took impractically long on large models. Simple round-to-nearest quantization (RTN), the cheapest post-training option, broke down below 8 bits: perplexity on WikiText2 jumped dramatically for large models pushed to 4-bit with RTN, making it unusable for quality-sensitive deployments. The field needed a method that was both accurate enough to survive 4-bit compression and fast enough to run on models with hundreds of billions of parameters in hours, not days.

The second-order foundation: OBS and OBQ

GPTQ is built on a lineage of neural network compression methods that use second-order information, specifically the Hessian of the loss with respect to the weights, to make informed decisions about which weights to change and by how much. The foundational method in this family is Optimal Brain Surgeon (OBS), introduced by Babak Hassibi and David Stork in 1993, which used the inverse Hessian to find the exact weight changes that compensate for pruning any given weight, keeping the output of each layer as close as possible to the original. This framework was mostly dormant for two decades but was revived for quantization in the Optimal Brain Quantization (OBQ) paper by Frantar and Alistarh themselves in 2022. OBQ applied the OBS framework to quantization rather than pruning: instead of removing a weight, it rounds it to the nearest quantization level, then uses the Hessian to adjust all remaining unquantized weights in that row to compensate for the rounding error introduced. The result is more accurate than round-to-nearest at the same bit width, because each rounding step corrects its own error before it accumulates. The limitation of OBQ was that it was too slow for very large weight matrices: the Hessian updates required for each quantized weight had cubic complexity in the number of weights per row. For small models (up to a few billion parameters), OBQ was feasible, but for 175B-scale models it was not.

GPTQ replaces the per-weight sequential OBQ loop with three engineering advances that together reduce the cost by several orders of magnitude while preserving most of the accuracy benefit. First, it quantizes all weights in a row simultaneously in a fixed left-to-right order, rather than choosing the optimal quantization order dynamically. The dynamic ordering of OBQ dominated its runtime, and the paper shows empirically that fixed ordering with Hessian-based correction is nearly as accurate. Second, it uses a lazy batch update strategy: rather than recomputing the full Hessian inverse after quantizing each individual weight, GPTQ defers updates across a block of 128 weights and applies them together, amortizing the cost of the expensive matrix operations. Third, it processes Cholesky decompositions of the Hessian inverse preemptively, which avoids expensive matrix inversions per weight and improves numerical stability. Together, these three changes reduce the per-layer quantization time from quadratic or cubic dependence on weight count to something close to linear in practice, making it feasible to quantize a 175B model on a single GPU.

What the paper demonstrated: results on OPT and BLOOM

The paper's primary benchmarks used OPT-175B (Meta AI, 2022) and BLOOM-176B (BigScience, 2022), the two largest publicly available language models at the time of publication. For OPT-175B quantized to 4 bits, GPTQ achieved perplexity on WikiText2 that was very close to the FP16 baseline, while round-to-nearest at 4 bits produced a perplexity increase large enough to make the model noticeably worse at language modeling tasks. GPTQ also quantized BLOOM-176B to 3 bits with minimal perplexity degradation, a compression regime where no previous post-training method had produced usable results at that scale. The full quantization of either 175B-scale model ran in approximately four GPU hours on a single GPU. The authors also showed that the method works reasonably well at 2-bit precision and even ternary (3-level) quantization, though accuracy losses at those extremes are more pronounced. For the 4-bit case, the compressed OPT-175B model occupies roughly 87 GB of memory, down from 326 GB in FP16, which is small enough to run on a single NVIDIA A100 80GB GPU, or across two NVIDIA A6000 48GB GPUs.

• OPT-175B in FP16: 326 GB memory, requires multiple high-end GPUs.
• OPT-175B at 4-bit GPTQ: fits on a single NVIDIA A100 80GB for the first time.
• Quantization time: approximately 4 GPU hours for 175B parameters.
• Inference speedup vs FP16: approximately 3.25x on NVIDIA A100, 4.5x on NVIDIA A6000.
• BLOOM-176B quantized to 3 bits with minimal perplexity increase.
• Round-to-nearest at 4 bits failed for large models; GPTQ succeeded where RTN could not.

Why inference speeds up at all

A subtle point in the paper is that inference speedup from GPTQ does not come from faster arithmetic. In 2022, there was no mainstream GPU hardware support for INT4 multiply-accumulate operations, so the actual matrix multiplications still happened in FP16. The speedup comes entirely from memory bandwidth. At the scale of large language models, inference latency is almost entirely memory-bound during the autoregressive decoding phase: each new token requires loading the entire weight matrix from GPU memory into compute units, and the GPU's compute cores spend most of their time waiting for weights to arrive rather than doing arithmetic. Compressing weights to 4 bits reduces the amount of data that must be moved from HBM (high bandwidth memory) to compute by a factor of four, which is why the observed speedups are roughly in the 3x to 4.5x range rather than 4x. The paper implemented custom CUDA kernels that dequantize blocks of INT4 weights to FP16 on the fly and then multiply, capturing the memory bandwidth saving without requiring hardware support for INT4 arithmetic.

Downstream impact: AutoGPTQ and the open-weight ecosystem

GPTQ's practical impact on the open-weight LLM ecosystem was immediate and large. The method's one-shot property, no retraining, no dataset-specific tuning beyond a small calibration set, and a total runtime measured in hours rather than days, made it the first quantization technique that individuals and small teams could apply to frontier-scale models with consumer hardware. In early 2023, the community-built AutoGPTQ library wrapped the original IST-DASLab implementation into an accessible Python package compatible with Hugging Face Transformers and the PEFT fine-tuning library. The Hugging Face Optimum library later integrated GPTQ support directly. Quantized versions of LLaMA, Mistral, Falcon, and their fine-tuned variants became widely available on Hugging Face Hub in GPTQ format, letting users run models that would otherwise require server-grade multi-GPU setups on a single consumer RTX 4090 or two RTX 3090s. The QLoRA paper (arXiv:2305.14314, May 2023) by Dettmers et al. built directly on the memory-efficiency tradition GPTQ advanced, though QLoRA used a different quantization scheme (NF4, or NormalFloat4) designed specifically for fine-tuning rather than inference. The two methods became complementary: GPTQ for inference-optimized deployment, QLoRA for memory-efficient fine-tuning. ExLlama and ExLlamaV2, open-source inference engines written in C++ and CUDA, pushed GPTQ inference speed further by optimizing the dequantization kernels, and llama.cpp added its own GGUF quantization format as an alternative that runs on CPU as well as GPU.

Limitations and what came after

The GPTQ paper itself notes two significant limitations. First, the speedup is memory-bandwidth-driven and does not help with the compute-bound phases of inference (such as processing long prompt batches at high parallelism), where the bottleneck shifts to arithmetic throughput rather than memory loading. Second, at very high compression ratios (2-bit and below), accuracy losses become substantial and model-specific, with no reliable general solution. The calibration dataset matters too: GPTQ uses a small sample of text (the paper uses 128 sequences from C4) to estimate the Hessian, and the quality of quantization can degrade if that calibration data is very far from the model's intended use distribution. Follow-on work addressed several of these gaps. GPTQ's authors and others published AWQ (Activation-aware Weight Quantization, arXiv:2306.00978, 2023) as a complementary approach that identifies salient weights by examining activation magnitudes rather than Hessian information alone, achieving competitive 4-bit results with simpler computation. QuIP# (2023) and QuaRot (2024) explored Hadamard rotation of weights to reduce outlier magnitude before quantization, pushing 2-bit results closer to viability. NVIDIA's TensorRT-LLM and later Blackwell-generation GPU hardware added hardware-native INT4 support, eliminating the dequantization overhead that capped GPTQ's speedup at roughly 4x and making sub-8-bit quantization a first-class inference path rather than a software workaround. PromptingIndex covers GPTQ alongside QLoRA, Chinchilla scaling laws, BitNet B1.58, and FlashAttention as part of its series on the hardware and training decisions that determine how large language models are built and deployed.