QLoRA: Finetuning 65B Models on a Single GPU with 4-Bit Quantization

Standard 16-bit finetuning of large language models is prohibitively expensive. The LLaMA 65B model, released by Meta in February 2023, required more than 780GB of GPU memory for a full 16-bit finetuning run, exceeding the capacity of even high-end multi-GPU server nodes. QLoRA, introduced in 'QLoRA: Efficient Finetuning of Quantized LLMs' (arXiv:2305.14314) by Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer at the University of Washington, submitted in May 2023 as an extended NeurIPS paper, reduces that requirement to under 48GB, the capacity of a single A40, A6000, or RTX 3090 Ti. The method freezes the base model at 4-bit precision and routes gradients only through a small set of learned Low-Rank Adapter (LoRA) weights that remain at 16-bit precision. Using QLoRA, the authors trained the Guanaco family of models, demonstrating that Guanaco 65B reaches 99.3% of ChatGPT performance on the Vicuna benchmark after 24 hours of finetuning on a single professional GPU. Their smallest Guanaco model, at 7B parameters, requires just 5GB of memory and outperforms the 26GB Alpaca model by more than 20 percentage points on the same benchmark.

Why standard quantization breaks finetuning

Prior to QLoRA, quantization methods such as GPTQ and bitsandbytes reduced the memory footprint of LLMs for inference. The problem is that quantization breaks finetuning: forward passes through a quantized model lose enough precision that backpropagated gradients become numerically unstable or accumulate enough quantization error to degrade the finetuned model's performance. The QLoRA paper describes this directly: 'such techniques only work for inference and break down during training.' The solution is to use quantization only for storage and frozen-parameter forward passes, while performing gradient accumulation in a higher-precision adapter space. The adapter weights, the LoRA matrices, receive and accumulate gradients normally at BFloat16. The quantized weights are never updated. The quantization error in the frozen base model is tolerated because the adapters can learn residual corrections that compensate for it.

4-bit NormalFloat: an information-theoretically optimal quantization type

NF4 (4-bit NormalFloat) is the first of QLoRA's three technical innovations. Standard 4-bit integer quantization maps values uniformly across the 16 representable levels in a 4-bit space, which works well when values are uniformly distributed. Neural network weights, however, follow approximately normal (Gaussian) distributions centered on zero. Uniform spacing wastes most of the representational budget on low-probability tails and under-samples the high-density region near zero. NF4 addresses this by setting quantization levels at quantiles of the standard normal distribution, so that each representable level covers an equal probability mass, concentrating levels where weight values cluster. The paper demonstrates that NF4 is information-theoretically optimal for normally distributed weights and that it outperforms both 4-bit integers and 4-bit floats empirically across multiple model scales. The baseline memory impact: storing weights at 4 bits rather than 16-bit BFloat16 cuts the base model footprint to roughly one quarter, reducing a 780GB LLaMA 65B finetuning load to approximately 195GB before the other two techniques are applied.

• NF4 levels are placed at quantiles of the standard normal, not at uniform intervals, concentrating representational budget near zero.
• Outperforms 4-bit integers (INT4) and 4-bit floats (FP4) empirically across LLaMA 7B to 65B.
• Information-theoretically optimal for normally distributed data: no 4-bit scheme can do better on that distribution.
• Implemented in the bitsandbytes CUDA kernels released alongside the paper.

Double Quantization and Paged Optimizers

Quantization at 4-bit precision requires storing quantization constants, the per-block scale factors used to convert between 4-bit levels and the original float values. For a typical block size of 64 weights, one 32-bit floating-point constant is stored per block. Across 65B weights, those constants add roughly 8GB of additional memory. Double Quantization solves this by quantizing the quantization constants themselves: it applies 8-bit quantization with a block size of 256 to the set of 32-bit constants. The net saving is approximately 0.37 bits per parameter, which translates to about 3GB for a 65B model. The third innovation, Paged Optimizers, addresses a different failure mode. During finetuning, processing a mini-batch containing a long sequence can trigger a momentary memory spike large enough to crash the run with an out-of-memory error, even when average memory use is within budget. Paged Optimizers use NVIDIA unified memory, the same hardware mechanism that lets CUDA programs address CPU RAM as an extension of GPU memory, to page optimizer states to CPU RAM during these spikes and page them back when needed. The result is stable training across variable-length batches without padding every sequence to the maximum length.

Guanaco: 1,000 models, 8 datasets, and a data quality finding

The authors used QLoRA to finetune more than 1,000 models across 8 instruction datasets, multiple model types (LLaMA and T5), and sizes ranging from 80M to 65B parameters. The Guanaco models are the top performers from this sweep. In Elo-rated tournament evaluation judged by GPT-4 (averaged across 10,000 random initial orderings), Guanaco 33B and 65B rank above all previously released open models, with Guanaco 13B scoring higher than Bard. Guanaco 65B achieves 99.3% Elo parity with ChatGPT; Guanaco 33B achieves 97.8% in under 12 hours on a single consumer GPU. One of the most practically significant findings from the large-scale comparison concerns dataset size versus quality. A 9k-sample dataset (OASST1, a human-curated multi-turn conversation set) outperformed a 450k-sample dataset (FLAN v2, subsampled) on chatbot performance, even though FLAN v2 is fifty times larger. The authors conclude that dataset suitability for the target task matters more than raw sample count. A second finding: high MMLU benchmark scores do not predict strong Vicuna chatbot scores, and vice versa. Optimizing for one does not transfer to the other.

LoRA placement: adapters at every layer

Standard LoRA deployments, following the original Hu et al. (2021) paper, typically add low-rank adapters to the query and value projection matrices in each attention block and leave feed-forward network layers untouched. QLoRA's ablations find that adding adapters to all linear layers, including the gate, up, and down projections in the MLP blocks, significantly reduces the accuracy gap relative to 16-bit full finetuning. The paper explicitly attributes prior LoRA accuracy gaps to this incomplete adapter placement rather than to LoRA's low-rank structure itself. With adapters placed at every linear layer and rank between 16 and 64, QLoRA recovers essentially all 16-bit finetuning performance while the base model remains frozen at 4-bit throughout. Code for QLoRA is available at github.com/artidoro/qlora; the 4-bit NF4 CUDA kernels are part of the bitsandbytes library at github.com/TimDettmers/bitsandbytes. PromptingIndex covers QLoRA alongside LoRA, DPO, and RLHF as part of its series on parameter-efficient finetuning and LLM alignment.