Llama 4 Scout, Maverick, and Behemoth: How Meta Built Its First MoE Model Family

When Meta released the Llama 4 family on April 5, 2025, it marked a structural break from every previous Llama model. Llama 1, 2, and 3 all used dense decoder-only transformers: every parameter participates in every forward pass. Llama 4 replaced that design with a sparse Mixture-of-Experts (MoE) architecture, where only a small fraction of total parameters activates for any given token. The result is a family of three models, Scout, Maverick, and Behemoth, each built around the same 17-billion-active-parameter core but scaled very differently in total parameter count and expert pool size. This post covers what those numbers mean, how the MoE routing works, and where each model sits on the performance curve.

From dense to sparse: what MoE changes

A standard transformer applies the same feed-forward network weights to every token in every layer. A MoE transformer replaces those feed-forward layers with a set of expert sub-networks and a router. For each token at each MoE layer, the router computes a weighted score over all experts and selects the top two to activate. Only those two experts process the token; all others are idle for that step. The key advantage is that total parameter count (which governs model capacity and knowledge storage) can grow much larger than the active parameter count (which governs compute cost per token). Llama 4 is the first generation of Llama models to use this approach. Its two released models, Scout and Maverick, both activate 17 billion parameters per forward pass, but draw from pools of 109 billion and 400 billion total parameters respectively.

Scout: 16 experts, one H100, 10 million token context

Llama 4 Scout has 17 billion active parameters drawn from a pool of 109 billion total parameters, organized across 16 experts. The router selects 2 of those 16 experts for each token at each MoE layer, keeping the active compute close to that of a 17B dense model while quadrupling the stored knowledge capacity. Scout's most notable specification is its context window: 10 million tokens, the longest of any open-weight model at launch, per Meta's announcement. The hardware requirement is correspondingly modest, with inference fitting on a single NVIDIA H100 (80 GB) in 4-bit quantization. On benchmarks, Scout outperformed Gemma 3, Gemini 2.0 Flash-Lite, and Mistral 3.1 across coding, reasoning, long-context, and image tasks at release.

Maverick: 128 experts, 400 billion total parameters, LMArena 1417

Llama 4 Maverick shares the same 17 billion active parameter budget as Scout but spans a pool of 400 billion total parameters across 128 experts. Fine-grained expert pools of this size allow the router to select genuinely specialized experts for different domains, a design pattern similar to DeepSeek-v3. Maverick requires approximately four H100 GPUs for inference and offers a 1 million token context window. At launch, Meta reported that Maverick beat GPT-4o and Gemini 2.0 Flash on image and text benchmarks, and achieved comparable results to DeepSeek-v3 on reasoning and coding tasks at less than half the active parameters. An experimental chat version scored an ELO of 1417 on LMArena. One concrete weakness: Maverick scored only 16% on the Aider Polyglot coding benchmark (state-of-the-art at the time was around 80%), indicating the models have meaningful gaps in complex multi-file code generation tasks.

Behemoth: the 2-trillion-parameter teacher

Llama 4 Behemoth was still training when Scout and Maverick were released. What Meta disclosed: Behemoth is a MoE model with 288 billion active parameters, 16 experts per layer, and nearly 2 trillion total parameters, making it among the largest models ever disclosed by any organization. Scout and Maverick were produced via distillation from Behemoth, meaning the smaller models were trained to reproduce the larger model's output distributions rather than being trained entirely from scratch on raw data. Meta stated that Behemoth outperformed GPT-4.5, Claude Sonnet 3.7, and Gemini 2.0 Pro on several STEM benchmarks at the time of disclosure, though the model was not released publicly.

Native multimodality and the iRoPE context architecture

Both Scout and Maverick are natively multimodal: they were trained from the start to process images and text together, rather than adding vision capability as a post-hoc adapter on a text-only backbone. This contrasts with Llama 3's approach, where vision was added as a separate component. The extreme context lengths in Llama 4 (10M for Scout, 1M for Maverick) are enabled by an architecture called iRoPE, which interleaves standard attention layers with layers using a modified rotary position embedding designed to extrapolate far beyond training context lengths. Meta's announcement noted that Scout and Maverick represent the beginning of a new era of natively multimodal AI, with Behemoth serving as the foundation from which the released models were distilled.

• Released April 5, 2025. First Llama generation to use sparse Mixture-of-Experts architecture.
• Scout: 17B active parameters, 16 experts, 109B total parameters, 10M token context window, single H100 inference.
• Maverick: 17B active parameters, 128 experts, 400B total parameters, 1M token context window, 4x H100 inference.
• Maverick ELO 1417 on LMArena; beat GPT-4o and Gemini 2.0 Flash on image and text benchmarks at launch.
• Maverick 16% on Aider Polyglot benchmark, versus approximately 80% state-of-the-art at the time.
• Behemoth: 288B active parameters, 16 experts, nearly 2T total parameters. Still training at launch; used as distillation teacher for Scout and Maverick.
• Both Scout and Maverick are natively multimodal (text + image) from initial training, not a post-hoc adapter.
• PromptingIndex has covered the general Switch Transformer and MoE architecture separately; this post focuses on Llama 4 specifics.

What the numbers tell practitioners

The Llama 4 release changed the practical calculus for open-weight model deployment in two ways. First, Scout's single-H100 footprint with 10M context makes it the most accessible long-context open model for organizations with limited GPU budgets. A 10M token context window means a model can hold approximately 7.5 million words in a single pass, enough to process entire codebases, legal document sets, or book collections without chunking. Second, Maverick's performance-to-cost ratio (400B total parameters but only 17B active) demonstrated that sparse MoE is now competitive with the best closed-source models at a fraction of the inference cost. The limitation in complex code generation, visible in the Aider Polyglot score, is a reminder that parameter counts and aggregate benchmark scores do not uniformly predict performance on specific tasks. PromptingIndex covers the underlying MoE architecture (Switch Transformer, mixture-of-experts explainer), distillation (Hinton et al. 2015 and knowledge distillation), and related inference topics (speculative decoding, KV cache) in separate deep-dives.