Quantization: How 70-Billion-Parameter Models Fit on One GPU

A 70-billion-parameter language model sounds impossibly large to run at home. In raw 16-bit floating-point, it is: the weights alone occupy 140 GB, exceeding the memory of every consumer GPU on the market and most professional ones. Yet researchers and hobbyists routinely run models of this size on a single workstation GPU, and some run them on a laptop CPU with no GPU at all. The technique that makes this possible is quantization, one of the most practically important ideas in modern AI.

Why size is the binding constraint

Modern language models store each weight as a 16-bit floating-point number, taking 2 bytes of memory. Multiply that by 70 billion and you need 140 GB just to load the model, before any activations or caching. Running inference means the GPU must read those weights repeatedly from VRAM on every forward pass, so VRAM is what limits you. Four of Nvidia's A100 80 GB cards, a stack that rents for thousands of dollars a month, barely fit a LLaMA-70B in FP16. For most people, the model is simply inaccessible at full precision.

What quantization actually does

Quantization reduces the number of bits used to represent each weight. A 16-bit FP16 parameter can take roughly 65,000 distinct values. Reducing to INT8 (8-bit integer) cuts memory in half; INT4 (4-bit integer) cuts it to a quarter. A 70-billion-parameter model at INT4 needs only 35 GB, a size that fits on dual professional GPUs instead of four, and that becomes accessible on a single high-end GPU once activation memory is managed carefully.

The danger is accuracy. Mapping a continuous FP16 weight onto one of only 16 possible INT4 values introduces rounding error. Large transformers make this especially tricky because they contain outlier features: certain activation dimensions whose values are more than 100 times larger than the rest. Quantize those outliers naively and model quality can collapse.

LLM.int8(), GPTQ, and AWQ: the race to 4-bit

The first major breakthrough was LLM.int8(), published at NeurIPS 2022 by Tim Dettmers and colleagues. Their insight was that outliers concentrate in a small number of specific dimensions. By keeping those dimensions in FP16 and quantizing everything else to INT8, the method achieved zero measurable accuracy loss on OPT-175B. The implementation shipped as the bitsandbytes library, which became the standard quantization backend for the Hugging Face ecosystem.

Aggressive 4-bit compression came with GPTQ, published at ICLR 2023. It uses second-order information (an approximation of the Hessian matrix) to calculate the optimal compensation for each weight's rounding error, layer by layer, so errors don't accumulate. GPTQ compressed BLOOM-176B to 3 to 4 bits in a matter of hours, cutting its memory footprint from around 350 GB to under 90 GB with perplexity barely changed.

A year later, MIT's Song Han group published AWQ (Activation-aware Weight Quantization), which won Best Paper at MLSys 2024. AWQ's finding: not all weights matter equally. By identifying the 1% of weights most important to model behavior (those whose activation magnitudes are largest) and applying scaling protection before quantization, 4-bit results become nearly lossless. AWQ also generates formats that map cleanly onto GPU arithmetic units, achieving more than 3× inference speedup over FP16 on the same hardware.

GGUF and running on CPU

Not every user has a GPU. In March 2023, Georgi Gerganov released llama.cpp, a C/C++ inference engine that runs quantized LLaMA models entirely on a CPU. It initially used a custom format called GGML; in August 2023 that was replaced by GGUF, a single-file format that bundles weights, tokenizer data, and model metadata. GGUF supports quantization levels ranging from Q2 (smallest, fastest, lowest quality) to Q8 (near-lossless). The community settled on Q4_K_M as a practical sweet spot. A 7B model in Q4_K_M runs acceptably on a modern MacBook; a 13B model fits comfortably in 16 GB of system RAM. llama.cpp spread rapidly because it made local, private inference accessible without specialized hardware.

QLoRA: fine-tuning inside the compressed model

Quantization was initially a pure inference technique: compress after training, then deploy. QLoRA, published at NeurIPS 2023, changed that. It pairs 4-bit quantization with Low-Rank Adaptation (LoRA), which freezes the original weights and trains only small adapter matrices on top. QLoRA introduced NF4, a 4-bit format tuned for the bell-curve distribution typical of neural network weights, along with double quantization (quantizing the quantization constants themselves) and paged optimizers that spill GPU memory to RAM during spikes. The result: a single 48 GB GPU can fine-tune a 65-billion-parameter model. The Guanaco model, fine-tuned this way on the OASST1 dataset in 24 hours, scored 99.3% of ChatGPT quality on standard evaluations, a task that would otherwise require hundreds of thousands of dollars of compute.

The next frontier: training in 1.58 bits

Every method above compresses a model that was trained at full precision. Microsoft's BitNet research asked a different question: what if you train with quantization from day one? BitNet b1.58, published in 2024, uses ternary weights where each parameter can only be −1, 0, or +1, just 1.58 bits on average (log₂3). At the 3-billion-parameter scale, BitNet b1.58 matches full-precision LLaMA while running 2.71× faster, using 3.55× less memory, and consuming 71.4% less energy during matrix operations. The inference engine bitnet.cpp, released at ACL 2025, runs these ternary models efficiently on standard CPUs. If the advantages hold as scale increases, ternary networks may reshape AI deployment economics more thoroughly than any post-training compression technique.

Quantization is not a shortcut to worse AI. Done well, INT4 models are nearly indistinguishable from their full-precision originals on most tasks, and they run on hardware most people actually own. When you read a model spec, the parameter count tells you about knowledge capacity; the quantization level tells you what GPU you actually need. Find community-tested prompts that work across ChatGPT, Claude, Gemini, and the open quantized models running on local hardware in the PromptingIndex prompt library.