SwiGLU: The Activation Function Inside LLaMA, PaLM, Mistral, and Gemma

Every Transformer-based language model contains a feed-forward network (FFN) layer that runs after each attention block. In the original 2017 Transformer paper, that FFN was a two-step operation: multiply the input by a weight matrix, apply ReLU to clip negatives to zero, then multiply by a second weight matrix and return. The hidden dimension of that first matrix was set to four times the model dimension, a convention that held across GPT-2, T5, BERT, and dozens of successors. It is a simple design, well-understood, and hardware-efficient. It is also no longer what any leading open model uses. LLaMA 1, 2, and 3, Mistral 7B, Mixtral, PaLM, PaLM 2, Gemma, DeepSeek, and Apple's reported foundation model architecture all use a replacement called SwiGLU, introduced in a five-page paper by Noam Shazeer submitted to arXiv on February 12, 2020.

The paper and the author

'GLU Variants Improve Transformer' (arXiv:2002.05202) has a single author: Noam Shazeer, then at Google. Shazeer is one of the eight co-authors of 'Attention Is All You Need' (2017), the paper that introduced the Transformer architecture. He also co-authored the 2017 Mixture of Experts paper with Jeff Dean, which seeded the sparse MoE architectures now used in Mixtral and GPT-4. After publishing the GLU variants paper, Shazeer left Google in 2021 and co-founded Character.ai. The GLU variants paper itself is deliberately low-key: Shazeer describes it as reporting 'empirical observations' and closes with the line 'We have no explanation as to why these variants perform better than the original.' The paper tests six replacement FFN layers, all derived from Gated Linear Units, and measures them against ReLU and GELU baselines on a T5-style language model trained on C4 text.

From GLU to SwiGLU: the gating family tree

Gated Linear Units were introduced by Dauphin et al. in 2016 (arXiv:1612.08083). The original GLU multiplies two parallel linear projections of the input, where one projection passes through a sigmoid function first: GLU(x) = sigmoid(xW) times (xV). The sigmoid output acts as a gate, a value between 0 and 1 that controls how much of the second projection passes through. Shazeer's 2020 paper notes that the sigmoid is one of several nonlinear functions that could serve as the gate and tests four alternatives: ReLU (producing ReGLU), GELU (producing GEGLU), Swish (producing SwiGLU), and no activation at all (producing a bilinear layer). Swish, defined as Swish(x) = x times sigmoid(x), was itself introduced by Ramachandran et al. in 2017 and is also called SiLU in PyTorch. SwiGLU substitutes Swish for the sigmoid gate: SwiGLU(x, W, V) = Swish(xW) times (xV). The gate is now data-dependent in a richer way: large positive values are passed nearly unchanged, small values are suppressed, and negative values are partially retained rather than zeroed.

• GLU (Dauphin 2016): sigmoid(xW) times (xV), gate output is between 0 and 1.
• ReGLU: max(0, xW) times (xV), uses ReLU as the gate.
• GEGLU: GELU(xW) times (xV), uses GELU as the gate.
• SwiGLU: Swish(xW) times (xV), uses Swish (x times sigmoid(x)) as the gate.
• All variants introduce a third weight matrix, requiring a compensating reduction in hidden dimension.

The 2/3 trick: keeping parameter count equal

A standard FFN has two weight matrices: one projecting from d_model to 4 times d_model, and one projecting back. SwiGLU has three matrices: W for the gate path, V for the value path, and W2 for the output projection. To keep the total parameter count and multiply-accumulate operations equal to the standard two-matrix FFN, Shazeer reduces the hidden dimension d_ff by a factor of 2/3. In the paper's base configuration with d_model = 768 and standard d_ff = 3072 (four times 768), the SwiGLU d_ff is 2048 (two-thirds of 3072). In LLaMA 7B, d_model is 4096. A standard FFN would set d_ff to 16,384. LLaMA instead sets the SwiGLU hidden dimension to 11,008, which is close to 8/3 times 4096 (the exact value is 10,922.67, rounded up to a multiple of 256 for hardware efficiency). The result is three smaller matrices instead of two larger ones, with comparable total parameter counts but a different computational structure.

What the paper measured

Shazeer tests each FFN variant on a T5-style encoder-decoder model with d_model = 768, d_k = d_v = 64, and 12 attention heads, trained on a sampled subset of C4 text. The primary metric is language model perplexity, lower is better. At 65,536 training steps, FFNReLU scores 1.997, FFNGELU scores 1.983, FFNBilinear scores 1.960, FFNReGLU scores 1.953, and FFNGEGLU scores the best at 1.942. SwiGLU and GEGLU are closely matched in the paper's experiments, with both outperforming GELU and ReLU by meaningful margins. The paper also reports SQuAD reading comprehension results: FFNSwiGLU achieves an exact match score of 83.42 and F1 of 91.03, versus 83.18 and 90.87 for FFNReLU. Shazeer notes that results are noisy at smaller scale and does not declare any single variant the definitive winner. The paper's contribution is establishing that the entire GLU family of replacements improves over ReLU and GELU, and that the variants are worth testing in larger-scale training.

Why SwiGLU won when PaLM and LLaMA chose it

Shazeer's paper benchmarks multiple variants at roughly equal quality. The reason SwiGLU became the industry standard rather than GEGLU or ReGLU is a matter of adoption, not a clear-cut experimental win in the original paper. Google's PaLM model, published in April 2022 (arXiv:2204.02311), selected SwiGLU for its FFN layers across all scales from 8 billion to 540 billion parameters, with the hidden dimension set to 8/3 of d_model. Meta's LLaMA paper (arXiv:2302.13971, February 2023) also uses SwiGLU and cites the Shazeer paper directly. Once LLaMA's weights became publicly available and were widely adopted as base models for fine-tuning, every model built on LLaMA checkpoints inherited the SwiGLU FFN. Mistral 7B (September 2023), Mixtral 8x7B (December 2023), Gemma (February 2024), LLaMA 2 (July 2023), and LLaMA 3 (April 2024) all use SwiGLU. DeepSeek's base models use SwiGLU as well. The activation function that Shazeer described in five pages in 2020 with no theoretical justification became the default FFN for the majority of open-weight LLMs released over the following four years.

What this means for practitioners and model builders

For practitioners working with existing models, SwiGLU is a fact of the architecture rather than a choice: any fine-tuning or quantization of a LLaMA, Mistral, or Gemma checkpoint inherits the SwiGLU FFN. The three-matrix structure means that quantization tools must handle W, V, and W2 as a unit rather than treating the FFN as a simple two-layer MLP. For practitioners building new architectures from scratch, the practical guidance from the research literature is to use SwiGLU with the 2/3 hidden dimension reduction and expect a modest perplexity improvement over GELU at equal parameter counts. The improvement is not dramatic on small models, but it compounds at scale and has been validated across models from 7B to 540B parameters. One common implementation pitfall: setting d_ff to 8/3 times d_model yields a non-integer for most d_model values. Standard practice is to round up to the nearest multiple of 64 or 256 to preserve efficient tensor shapes on GPU hardware, which is why LLaMA 7B uses 11,008 rather than the exact value of 10,922. PromptingIndex covers SwiGLU alongside related architecture choices including RoPE position encodings, grouped-query attention, and BitNet b1.58 ternary weights, all of which feed into the design of efficient modern LLMs.