AdamW: Why L2 Regularization and Weight Decay Are Not the Same in Adam

Almost every modern large language model is trained with a variant of Adam that carries one small but consequential modification: weight decay is applied directly to the parameters rather than added to the loss as an L2 penalty. The distinction sounds pedantic until you work through the math. For stochastic gradient descent, L2 regularization and weight decay are mathematically identical, and the two terms have been used interchangeably in deep learning for years. But for Adam, they are not the same, and using L2 in place of true weight decay produces poorly calibrated regularization that is harder to tune and leads to worse generalization. This is the central finding of 'Decoupled Weight Decay Regularization,' arXiv:1711.05101, submitted November 14, 2017, by Ilya Loshchilov and Frank Hutter at the University of Freiburg and accepted at ICLR 2019. The fix they proposed, now called AdamW, became the default optimizer for every major LLM trained since GPT-2.

Why L2 and weight decay are equivalent for SGD

For plain stochastic gradient descent, L2 regularization adds a gradient term of lambda times the current parameters, so the update becomes: theta_new = theta_old minus lr times (task_gradient plus lambda times theta_old), which expands to (1 minus lr times lambda) times theta_old minus lr times task_gradient. Direct weight decay does the same thing by a different route: theta_new = (1 minus wd) times theta_old minus lr times task_gradient. Setting wd equal to lr times lambda makes the two formulas identical at every step. This equivalence is exact for constant learning rates and is why engineers have called L2 regularization 'weight decay' interchangeably for decades. When a learning rate schedule is used, the effective weight decay from L2 changes proportionally to lr, which can be a feature or a problem depending on the use case, but for SGD the two formulations remain functionally equivalent and the coupling is at least transparent.

How Adam's adaptive scaling breaks the equivalence

Adam stores a running average of squared gradients (the second moment) per parameter. The update rule divides each gradient by the square root of that second moment estimate, giving each parameter its own effective learning rate scaled by its gradient history. When L2 regularization is added to Adam, the term lambda times theta is added to the gradient before it enters Adam's moment accumulators. The L2 signal then gets divided by the same adaptive denominator as the task gradient. For parameters with high-variance gradients, the second moment is large and the adaptive denominator suppresses the L2 term, reducing effective regularization on those parameters. For parameters with stable, near-zero gradients, the adaptive denominator is small and the L2 term is amplified relative to its intended strength. The result is that Adam with L2 applies parameter-specific, gradient-magnitude-dependent regularization: the effective weight decay per parameter is roughly lambda divided by the square root of its second moment estimate, a quantity that varies across layers and changes throughout training. Changing the learning rate also shifts the effective regularization strength, coupling two hyperparameters that should be independent.

The AdamW modification

AdamW makes one structural change to Adam. After the adaptive gradient step, weight decay is applied as a separate multiplicative term directly to the parameters: theta_new = theta_old minus lr times (first_moment divided by (sqrt(second_moment) plus eps)) minus wd times theta_old. The wd term does not pass through the moment accumulators. It is applied uniformly to all parameters regardless of their gradient history, and it does not scale with the learning rate. Changing lr does not change the regularization strength. The paper also introduced SGDW, applying the same decoupling to SGD with momentum, though the practical benefit is smaller because SGD's coupling between L2 and the update is weaker. The key consequence is that wd and lr become independently tunable: finding a good weight decay value does not require a new hyperparameter sweep for each learning rate setting.

Results on image classification

The paper's experiments used wide residual networks with shake-shake regularization on CIFAR-10 and CIFAR-100. The authors swept learning rate and regularization coefficient jointly for both Adam+L2 and AdamW, combined with cosine annealing with warm restarts (SGDR, from the same authors' arXiv:1608.03983, ICLR 2017). Two findings stood out. First, the optimal weight decay for AdamW was nearly constant across different learning rates, confirming that the decoupling makes the two hyperparameters independent. For Adam+L2, the optimal L2 coefficient shifted substantially with the learning rate, requiring a fresh sweep for each lr setting. Second, AdamW with SGDR achieved generalization performance competitive with SGD and momentum, which had consistently outperformed standard Adam on image benchmarks. The paper attributed a meaningful portion of that generalization gap to the misspecified L2 regularization in conventional Adam, not to any fundamental limitation of adaptive gradient methods.

• L2 and weight decay are mathematically equivalent for SGD but not for adaptive methods.
• In Adam+L2, the regularization signal enters the moment accumulators and gets parameter-specific scaling.
• AdamW applies weight decay separately, after the gradient step, at uniform strength across all parameters.
• Optimal wd in AdamW is stable across learning rates; in Adam+L2 the two are coupled.
• arXiv:1711.05101, submitted November 2017, accepted ICLR 2019, authors at University of Freiburg.
• PyTorch and Hugging Face Transformers both include AdamW as a first-class optimizer.

AdamW as the standard for large-scale LLM training

GPT-2 (OpenAI, February 2019) used AdamW. GPT-3 (Brown et al., 2020) used AdamW with beta_1 of 0.9, beta_2 of 0.95, and weight decay of 0.1. LLaMA 2 (Meta AI, 2023) used AdamW with weight decay of 0.1. LLaMA 3 (Meta AI, 2024) used AdamW with beta_1 of 0.9, beta_2 of 0.95, eps of 10 to the power of negative 5, and weight decay of 0.1. The cosine learning rate schedule with linear warmup that is now standard for pretraining pairs naturally with AdamW because the decoupled weight decay stays constant across the warmup and decay phases, providing stable regularization throughout training. The standard convention applies weight decay to most parameters but excludes normalization layer weights and bias terms, which are small enough that shrinking them toward zero provides negligible regularization benefit at a real risk of degraded performance. PyTorch and Hugging Face Transformers support this through parameter group APIs that let callers specify which parameter subsets receive wd. The paper's legacy is unusual: it corrected a bug hidden inside a dominant algorithm rather than introducing a new one, and the correction was adopted completely. There is no published large-scale LLM pretraining run in the open literature that uses standard Adam with L2 regularization. PromptingIndex covers AdamW alongside ZeRO, QLoRA, GPTQ, and Chinchilla scaling laws in its series on the training infrastructure decisions that underlie modern language models.