MMLU: The 57-Subject Benchmark That Became the De Facto Standard for LLM Evaluation

When GPT-3 was released in 2020, the field lacked a single benchmark that could probe a language model across the full breadth of academic and professional knowledge. Existing evaluations tended to test one domain at a time, or measured narrow linguistic competencies that bore little resemblance to the kind of knowledge a doctor, lawyer, or engineer would need. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt, working at the Center for AI Safety, UC Berkeley, and Columbia University, proposed a different approach. Their paper 'Measuring Massive Multitask Language Understanding,' submitted to arXiv on September 7, 2020 (arXiv:2009.03300) and accepted at ICLR 2021, introduced a benchmark called MMLU that covered 57 distinct subjects with multiple-choice questions sourced from practice exams, professional certification tests, and academic study materials. MMLU became the de facto standard for comparing large language models, cited in nearly every frontier model release from 2021 onward and widely reported in model technical reports as a primary measure of general knowledge and reasoning.

How the benchmark was constructed

The 57 subjects in MMLU span four broad areas: STEM, humanities, social sciences, and a category the authors call 'other,' which includes professional and applied domains. The STEM subjects include abstract algebra, anatomy, astronomy, college biology, college chemistry, college computer science, college mathematics, college physics, conceptual physics, electrical engineering, elementary mathematics, formal logic, high school biology, high school chemistry, high school computer science, high school mathematics, high school physics, high school statistics, and machine learning. The humanities subjects include formal logic, high school European history, high school US history, high school world history, international law, jurisprudence, logical fallacies, moral disputes, moral scenarios, philosophy, prehistory, professional law, and world religions. The social sciences include econometrics, high school geography, high school government and politics, high school macroeconomics, high school microeconomics, high school psychology, human sexuality, political science, public relations, security studies, sociology, and US foreign policy. The 'other' category adds subjects including clinical knowledge, college medicine, global facts, human aging, management, marketing, medical genetics, miscellaneous, nutrition, professional accounting, professional medicine, professional psychology, and virology. Each subject has a dedicated split in the dataset with questions drawn from materials that test domain-specific knowledge rather than general reading comprehension. The questions use a four-option multiple-choice format throughout, making the random-chance baseline 25% for every subject.

GPT-3 performance and the 2020 baseline

The paper evaluated a range of models available in 2020, including GPT-3 in its various sizes (published by OpenAI in May 2020), UnifiedQA, and several other systems. The central finding was stark: most models of that era scored at or near random chance across the 57 subjects. The very largest GPT-3 model, with 175 billion parameters, improved over random chance by almost 20 percentage points on average, landing around 43 to 44 percent overall accuracy on the test set. That figure is roughly the score a test-taker who knew the answer to fewer than half the questions would achieve. The paper reported that even the best models had near-random accuracy on some of the most socially consequential subjects, including those covering morality and law. Expert human performance on the same subjects, estimated by the authors, was approximately 89.8%. The gap between GPT-3 and that human baseline was not a matter of fine-tuning or prompting strategy; it reflected a fundamental deficit in world knowledge and reasoning across specialized domains.

The benchmark as a progress tracker: GPT-4 and saturation

Between 2021 and 2023, MMLU performance became one of the primary numbers reported in model releases. Each generation of models pushed the score upward, with GPT-4 achieving 86.4% on the benchmark when it was released in March 2023. That figure was significant because it came within a few points of the estimated expert human baseline, marking the first time a commercially available model crossed what had seemed like a ceiling only three years earlier. After GPT-4, progress stalled in a way that became a subject of active discussion among researchers. Models released in the first half of 2024, including GPT-4-Turbo, Gemini 1.5 Pro, Claude 3 Opus, and LLaMA-3-400B, all settled into a range of roughly 86 to 87 percent. GPT-4o, released in May 2024, reached 87.4%, a gain of approximately 1 percentage point over GPT-4. At that point MMLU was effectively saturated: the remaining gap to estimated expert human performance was small enough that further improvements were hard to measure, and score differences between top models had compressed to within statistical noise. This saturation problem was not unique to MMLU but MMLU became the most visible case study of what researchers call benchmark saturation: the point at which a test no longer discriminates between the capabilities of frontier models.

• Hendrycks, Burns, Basart, Zou, Mazeika, Song, Steinhardt (arXiv:2009.03300, submitted September 7, 2020; ICLR 2021): 57 subjects across STEM, humanities, social sciences, and applied domains.
• Four-option multiple-choice format throughout; random-chance baseline is 25%.
• GPT-3 (175B parameters, 2020): improved by almost 20 percentage points over random chance on average, reaching roughly 43 to 44 percent accuracy.
• Expert human performance estimated at approximately 89.8% in the original paper.
• GPT-4 (March 2023): 86.4%, closing most of the gap to estimated expert human performance.
• GPT-4-Turbo, Gemini 1.5 Pro, Claude 3, LLaMA-3-400B (early to mid 2024): clustered between 86 and 87 percent, indicating benchmark saturation.
• GPT-4o (May 2024): 87.4%, approximately 1 percentage point above GPT-4.
• MMLU-Pro (Wang et al., arXiv:2406.01574, 2024): successor benchmark with 10 options, 12,032 questions across 14 domains; GPT-4o scored 72.6% on MMLU-Pro, restoring a meaningful margin for discrimination.

Why the four-option format became a limitation

The same design choice that made MMLU easy to administer, a consistent four-option multiple-choice format with a well-defined random-chance baseline, turned out to contribute to its saturation problem. With only three distractors per question, frontier models can exploit statistical patterns and eliminate wrong answers without fully understanding the domain. Researchers noted that model performance on MMLU was sensitive to prompt formatting in ways that were larger than they appeared: variations of 4 to 5 percentage points were observed across different prompt styles for the same model on the same questions, meaning that leaderboard rankings could shift based on how a model was prompted rather than what it actually knew. A follow-up paper from the University of Waterloo and Carnegie Mellon University, 'MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark' (arXiv:2406.01574, submitted June 2024), addressed these issues directly. MMLU-Pro expanded the answer set from 4 to 10 options, reducing the chance-level accuracy to 10%, merged the 57 original subjects into 14 broader categories, removed questions that models from the era of Llama-2-7B could answer correctly, and added harder reasoning-focused questions from STEM platforms and academic competitions. Prompt sensitivity on MMLU-Pro dropped from 4 to 5 percent to roughly 2 percent, and the performance gap between GPT-4o and GPT-4-Turbo widened from 1 percentage point on MMLU to 9 percentage points on MMLU-Pro, restoring the discriminative power that had been lost.

MMLU in the context of AI evaluation history

MMLU arrived at a specific moment in evaluation history: after GLUE and SuperGLUE had established that language models could be measured on linguistic tasks in a standardized way, but before the research community had confronted the full difficulty of knowledge-intensive evaluation. GLUE, released in 2018, and SuperGLUE, released in 2019, both saturated quickly as models improved; the highest-performing SuperGLUE systems exceeded the human baseline within about two years of the benchmark's release. MMLU's 57-subject structure was designed to be more durable by requiring genuine world knowledge rather than surface linguistic competence. It remained competitive as a primary evaluation tool for roughly three years, from 2021 through 2023, which is a relatively long lifespan for an AI benchmark given the pace of model improvement. Its successor benchmarks, including MMLU-Pro, BIG-Bench Hard, and subject-specific evaluations like MATH and HumanEval for code, each target specific failure modes that MMLU could not measure: multi-step mathematical reasoning, code generation, and calibrated uncertainty estimation across domains where the model might confidently give wrong answers. The original MMLU remains in use as a lower-bound sanity check for model releases, its score providing a point of historical comparison even when it no longer differentiates between the most capable current systems. PromptingIndex covers MMLU alongside related benchmark and evaluation topics, including scaling laws, benchmark saturation, and the methodological challenges of measuring general intelligence in large language models.