Mixture of Experts: How Huge AI Models Stay Fast

One of the quiet tricks behind the largest AI models is that they don't actually use all of themselves at once. A model can advertise hundreds of billions of parameters yet run about as cheaply as one a fraction of its size. The technique that makes this possible is called mixture of experts, or MoE, and once you understand it a lot of confusing model specs suddenly make sense.

The problem MoE solves

In a standard, dense transformer, every parameter is used to process every token. Doubling the parameter count roughly doubles the compute needed for each word the model reads or writes. That relationship is brutal: it means the only way to make a dense model smarter by scaling it up is to pay proportionally more for every single token, forever. Researchers wanted a way to grow a model's total knowledge without growing the per-token bill at the same rate.

What a mixture of experts actually is

The idea is to replace the dense feed-forward layer of a transformer with many smaller copies of it, called experts, plus a small routing network sometimes called the gate. For each token, the router picks only a handful of experts, often just one or two, and sends the token to them. The other experts sit idle for that token.

This is the key move. The model has a large total number of parameters spread across all its experts, but only a small subset, the ones the router selected, does any work on a given token. Engineers describe this with two different numbers: total parameters, which reflects how much the model can store and learn, and active parameters, which reflect how much compute each token actually costs. In a dense model those two numbers are the same. In an MoE model the total can be many times larger than the active count.

A concrete example

Mistral's Mixtral 8x7B, released in December 2023, is the clearest illustration. As the name hints, each layer holds eight experts, and the router sends every token to two of them. The full model contains roughly 47 billion parameters, but because only two of the eight experts fire per token, only about 13 billion parameters are active for any given step. You get the knowledge capacity closer to a 47-billion-parameter model at something much nearer the running cost of a 13-billion-parameter one.

The approach scales further than that. Google's Switch Transformer, published in 2021, simplified routing down to a single expert per token and used the savings to push the total parameter count past a trillion, years before dense models reached that range. Several of the most capable proprietary models are also widely reported to use MoE internally, though the labs behind them rarely confirm architecture details.

The trade-offs

MoE is not free. Sparse activation buys cheaper compute, but it introduces its own headaches:

• Memory stays expensive. Even though only a few experts run per token, all of them must be loaded into memory in case the router calls on them, so an MoE model's memory footprint tracks its total parameter count, not its active one.
• Routing can go lopsided. Left unchecked, the gate tends to funnel most tokens to a few favorite experts while others go unused, so training adds a load-balancing objective to keep the experts evenly fed.
• Fine-tuning is trickier. The router and the experts have to stay in sync, and naive fine-tuning can unbalance a model that was carefully balanced during training.
• Behavior is harder to interpret. Which expert handles which token is learned, not designed, so the specialization rarely maps to anything as tidy as grammar or math.

Why this matters when you prompt

You never address experts directly, and there's no prompt trick that steers the router. What MoE changes is the economics of the models you use. It is a big reason providers can offer very capable models at low per-token prices, and why a model with an eye-popping parameter count can still respond quickly. When you compare models, it's worth looking past the headline parameter number to the active parameter count, because that is what really drives speed and cost.

The craft of prompting stays the same whether the model underneath is dense or sparse: clear instructions, useful context, and testing across models still win. Browse the community-rated prompts on PromptingIndex to see which ones hold up across ChatGPT, Claude, Gemini, and the open models, and upvote the ones that earn a place in your workflow.