GaLore: How Projecting Gradients Instead of Weights Unlocks 7B Pretraining on a Single RTX 4090

Pretraining a LLaMA 7B model from scratch requires at least 58 gigabytes of GPU memory: 14 gigabytes for the parameters in BF16, roughly 42 gigabytes for Adam optimizer states and weight gradients, and another 2 gigabytes for activations at a sequence length of 2048. That total puts 7B pretraining well above the 24 gigabytes available on a consumer NVIDIA RTX 4090 and firmly in the territory of multi-GPU server runs. Gradient Low-Rank Projection (GaLore), from Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian at the University of Texas at Austin, Carnegie Mellon University, Caltech, and Meta AI, changes that equation. Published in 'Memory-Efficient LLM Training by Gradient Low-Rank Projection' (arXiv:2403.03507, submitted March 6, 2024, and accepted at ICML 2024 as an Oral presentation), GaLore reduces optimizer state memory by up to 65.5% and, in its 8-bit variant, cuts total training memory by 63.3% compared to a BF16 full-rank baseline. It is the first published demonstration of pretraining a 7B parameter model on a single consumer GPU without model parallelism, gradient checkpointing, or CPU offloading.

Why Adam eats so much memory and why the weights are not the main culprit

A naive reading of LLM memory requirements focuses on parameter count. At 2 bytes per parameter in BF16, a 7B model occupies 14 gigabytes, which sounds like it should fit in an RTX 4090. The problem is Adam. Adam maintains two running statistics per parameter: a first-moment estimate (mean of gradients) and a second-moment estimate (mean of squared gradients). Both are stored at full precision, typically float32, consuming 4 bytes per parameter each. For a 7B model, those two moment tensors alone occupy 56 gigabytes, roughly four times the parameter storage. Add the weight gradients themselves and the activations for a modest batch, and the total balloons to 58 gigabytes or more. Techniques like 8-bit Adam (Tim Dettmers et al., 2022) reduce the moment precision to 8-bit integers using block-wise dynamic quantization, cutting moment storage by about 4x. Gradient checkpointing trades compute for activation memory. Memory offloading moves tensors to CPU RAM when not needed but introduces PCIe bandwidth bottlenecks. Each of these attacks a different term in the memory budget, but none removes the fundamental cost of storing per-parameter optimizer state.

LoRA's limitation: the weights may not be low-rank, even if the gradients are

The most widely deployed approach to memory reduction in LLM adaptation is Low-Rank Adaptation (LoRA, Hu et al., 2022). LoRA freezes the pretrained weight matrix W and learns a low-rank correction B times A, where B is m by r and A is r by n with r much smaller than min(m, n). Because only A and B are trainable, Adam only needs moment tensors for those small matrices, dramatically reducing optimizer memory. For fine-tuning, LoRA has become ubiquitous. For pretraining from scratch, it has a fundamental limitation: the optimal weight matrix W may not be low-rank. If the true solution is high-rank, constraining W to a low-rank subspace forever prevents the model from reaching it, regardless of the optimizer used. ReLoRA (Lialin et al., 2024) partially addresses this by periodically merging low-rank updates back into the frozen weight and starting fresh low-rank adaptors, but it still requires a full-rank warmup phase before the low-rank training begins. GaLore avoids both problems by observing that the answer is not to constrain the weights to be low-rank. The key insight is that the gradients are empirically low-rank even when the weights are not.

The GaLore algorithm: project the gradient, store optimizer state in the compact space

GaLore leaves the weight matrix W at full rank. It does not add low-rank adaptors and does not freeze any parameters. Instead, at each training step, it computes two projection matrices: P of shape m by r (left projector) and Q of shape n by r (right projector). These are the leading singular vectors of the gradient matrix G, computed via truncated singular value decomposition. The projected gradient is then P-transpose times G times Q, which has shape r by r, far smaller than the original m by n gradient. Adam's moment accumulators are maintained in this compact space, not in the original parameter space. After the optimizer update produces an update in the compact space, GaLore projects it back to the original space by computing P times update times Q-transpose and applies it to W directly. Because the moments have shape r by r rather than m by n, their memory cost scales with r squared rather than m times n. For typical LLM layers where m and n are 4096 or 8192 and r is 128 to 256, this represents a reduction in optimizer state memory by a factor of roughly m times n divided by r squared, which in practice yields the 65.5% reduction reported in the paper. One important design choice: P and Q are not recomputed every step. The paper uses a subspace change frequency of approximately every 200 iterations. Recomputing the SVD of an m by n gradient matrix every 200 steps adds amortized overhead that the paper reports as less than 10% of total training time for the 7B scale experiments.

Results: LLaMA 1B and 7B pretraining on the C4 dataset

The paper's primary benchmarks are LLaMA 1B and LLaMA 7B pretrained on the C4 dataset with up to 19.7 billion tokens. For the 1B scale, GaLore with AdamW achieves perplexity within 0.2 to 0.4 points of the full-rank baseline across different token budgets, while consuming significantly less optimizer memory. For the 7B scale, 8-bit GaLore combined with 8-bit Adam and per-layer weight updates achieves perplexity comparable to the full-rank BF16 baseline, with less than 10% of the optimizer state memory. The 8-bit GaLore variant stacks two memory reductions: the low-rank projection reduces the size of the state tensors, and then 8-bit quantization reduces the precision of those already smaller tensors. Together, optimizer memory drops by 82.5% compared to full-rank BF16 Adam, and total training memory (including weights, activations, and gradients) drops by 63.3%. This reduction is what makes the RTX 4090 experiment possible. Using GaLore with 8-bit Adam and per-layer updates, the authors pretrain LLaMA 7B on a single GPU with 24 gigabytes of VRAM without any offloading or parallelism strategy. On fine-tuning, GaLore applied to RoBERTa-Base on GLUE tasks with rank 4 achieves an average score of 85.89, outperforming LoRA at 85.61 in the same configuration. GaLore also demonstrates compatibility with three different optimizers: AdamW, 8-bit Adam, and Adafactor, with similar memory savings and minimal perplexity degradation across all three.

• Pretraining LLaMA 7B from scratch in BF16 requires at least 58 GB: 14 GB weights, 42 GB Adam states and gradients, 2 GB activations.
• GaLore projects gradients into a low-rank subspace via SVD; Adam moment tensors are stored in that compact space, not in full parameter space.
• Optimizer state memory reduced by up to 65.5%; 8-bit GaLore cuts optimizer memory 82.5% and total memory 63.3% versus a BF16 baseline.
• Subspace projection matrices P and Q are updated every approximately 200 steps, adding less than 10% amortized computational overhead.
• First published result for pretraining a 7B model on a single 24 GB consumer GPU (NVIDIA RTX 4090) without model parallelism, checkpointing, or offloading.
• arXiv:2403.03507, submitted March 6, 2024; ICML 2024 Oral; authors at UT Austin, CMU, Caltech, and Meta AI.

What GaLore changes in practice

GaLore requires roughly two lines of code added to a standard training loop: one to project the gradient before passing it to the optimizer, and one to project the update back before applying it to the weights. The paper's reference implementation on GitHub (github.com/jiaweizzhao/GaLore) integrates with standard PyTorch and wraps existing optimizers. Because GaLore operates on the gradient rather than the weight, it is fully compatible with any parameter group, any layer type, and any optimizer that maintains first- and second-moment statistics. It is particularly effective for the large linear projections inside transformer attention and feed-forward blocks, which account for most of the parameter count and most of the optimizer state in LLMs. The projection is not applied to layer normalization weights or bias terms, which are too small to benefit from the technique. One practical constraint worth noting: because the projection matrices P and Q are shared across the optimizer step, GaLore requires that gradients accumulate in the original parameter space before projection, which means it does not natively compose with ZeRO stage 3's parameter sharding without modification. The paper focuses on single-device and data-parallel training rather than tensor-parallel or pipeline-parallel regimes. The broader implication is that GaLore shifts the optimization subspace question from the weights to the gradients. Empirical evidence across LLM training suggests that gradients live in a low-dimensional manifold even when the weight matrices they update do not, and GaLore turns that empirical observation into a concrete memory saving. PromptingIndex covers GaLore alongside AdamW, ZeRO, QLoRA, and Chinchilla scaling laws in its series on the training infrastructure decisions that define how modern language models are built.