Mixture of Experts: How Modern LLMs Activate Only a Fraction of Their Parameters

A standard transformer processes every token through every parameter in every feed-forward layer. A sparse Mixture of Experts model processes each token through only a selected subset of the network, leaving most parameters idle for that particular token. Mixtral 8x7B, released by Mistral AI in December 2023, has 47 billion total parameters but activates only 13 billion for each token at inference time. DeepSeek-V3, released in December 2024, has 671 billion total parameters but activates only 37 billion per token. The architecture that makes this possible is called sparse MoE, and it changes the tradeoff between model capacity and compute cost in a way that has reshaped how the most capable open-weight models are built.

Where the idea came from: the 2017 sparsely-gated layer

The modern form of MoE in language models traces to a January 2017 paper by Noam Shazeer and colleagues at Google Brain, 'Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer' (arXiv:1701.06538). The paper introduced a sparse gating mechanism applied to the feed-forward sublayers of a recurrent language model. Instead of one feed-forward network, each layer contained thousands of expert networks, and only the top-k were activated per input. Shazeer et al. demonstrated that this approach could scale a model's parameter count by orders of magnitude while keeping computation approximately constant, because the vast majority of experts were never activated for any given input. The paper identified the core promise that still drives MoE research today: more parameters without proportionally more compute.

How the routing mechanism works

In a transformer, MoE modifies only the feed-forward sublayer of each block. The attention mechanism remains unchanged. For each token, a learned router (a small linear layer followed by softmax) produces a probability distribution over all N available experts. The model then selects the top-K experts from this distribution, passes the token through each one, and combines their outputs as a weighted average, with the weights given by the router's scores for those experts. The experts themselves are structurally identical: each is an independent copy of the standard feed-forward network from that layer, with its own separate weights. Because only K out of N experts process any given token, the active parameter count is K/N of the total parameter count in the MoE layers. The remaining experts hold their weights in memory but contribute no computation for that token.

Switch Transformers: one expert per token, trillion parameters

The next major step came from William Fedus, Barret Zoph, and Noam Shazeer at Google, who submitted Switch Transformers (arXiv:2101.03961) in January 2021. The paper simplified the routing algorithm: instead of routing each token to two or more experts, each token routes to exactly one (K=1), which is why the technique is called a switch. The simplification reduced inter-device communication costs and made training more stable. Applied to a T5-style encoder-decoder architecture, Switch Transformers achieved up to 7x pre-training speedups compared to T5-Base using the same total compute budget, and a 4x speedup over T5-XXL. The paper also demonstrated scaling to trillion-parameter models trained on the Colossal Clean Crawled Corpus, and showed that large sparse models could be trained with bfloat16 precision rather than full float32.

The load balancing problem and how it is solved

MoE introduces a training failure mode called routing collapse. The router, which is trained by gradient descent alongside the rest of the model, tends to develop a preference for specific experts early in training. Once a few experts receive more gradient signal, they improve faster, which makes the router prefer them even more. The result is that most of the network's capacity goes unused: a handful of experts handle nearly every token while the rest contribute almost nothing.

Switch Transformers addressed this with an auxiliary loss that penalizes imbalanced expert utilization, tracking both how many tokens are assigned to each expert and what routing probability the gating network assigns. A later paper, ST-MoE (arXiv:2202.08906, 2022), introduced a second auxiliary term called the router z-loss, which constrains the magnitude of the logits entering the router's softmax. Large logits cause numerical precision problems at scale, and the router z-loss reduces training instability without conflicting with the load balancing objective. Both auxiliary terms are added on top of the standard language modeling loss during training, with small coefficients that are tuned per model.

Mixtral 8x7B and DeepSeek-V3: MoE at practical scale

Mixtral 8x7B, published by Mistral AI in December 2023, became the first widely adopted open-weight MoE language model. Each transformer layer contains 8 experts, each with the same architecture as the corresponding layer in Mistral 7B. Each token is routed to 2 of the 8 experts (K=2, N=8). Total parameters across all experts are 47 billion; active parameters per token are 13 billion. The model was trained with a 32k-token context window. In benchmark evaluations, Mixtral matched or exceeded Llama 2 70B on all reported benchmarks while using approximately 5 times fewer active parameters per token, which translates directly to lower memory bandwidth requirements and higher throughput on the same hardware.

DeepSeek-V3, published in December 2024 by DeepSeek-AI (arXiv:2412.19437), pushed MoE to a substantially larger scale: 671 billion total parameters, with 37 billion activated per token. The model adopted DeepSeekMoE, an architecture variant that includes a set of shared experts that always process every token alongside the routed experts, giving the model a stable baseline of general-purpose capacity. DeepSeek-V3 replaced the auxiliary load balancing loss used in earlier work with a bias-based routing adjustment: a per-expert bias term is added to the router's scores during selection, then subtracted before the weighted output is computed, so the bias influences routing without affecting the output values. It was trained on approximately 14.8 trillion tokens at a reported cost of about 2.788 million H800 GPU hours, equivalent to roughly $5.6 million at $2 per GPU-hour.

• Mixtral 8x7B: 47B total parameters, 13B active per token, 8 experts per layer, top-2 routing, 32k context.
• DeepSeek-V3: 671B total parameters, 37B active per token, trained on 14.8 trillion tokens, auxiliary-loss-free load balancing.
• Switch Transformer: 1.6 trillion parameters at largest scale, K=1 routing, 7x speedup over T5-Base at same compute.

What the active parameter count means for inference

The MoE architecture separates a model's knowledge capacity, measured by total parameters, from its computational cost per token, measured by active parameters. For anyone deploying these models, the two numbers have different practical consequences. A model's VRAM footprint at serving time is determined by total parameters: all expert weights must be loaded into memory so that any expert can be selected at any time. But throughput and latency are determined by active parameters, since only the selected experts run per token. A 47-billion-parameter MoE model and a 13-billion-parameter dense model have similar latency profiles per token, but the MoE model carries knowledge encoded across all 47 billion parameters during training. For inference at scale, this tradeoff favors MoE: larger capacity at comparable serving cost. The constraint is that all expert weights must fit in available memory, so the VRAM requirement tracks total parameters, not active ones. PromptingIndex covers both the architecture and the serving layer because decisions made during model design, such as how many experts to use and what K to set, propagate directly into the hardware requirements and throughput numbers that govern how these models are actually deployed.