Thinking Longer: How Inference-Time Compute Reshaped AI Reasoning

From 2020 to 2023, the dominant story in AI was simple: bigger models trained on more data got smarter. Researchers published scaling laws showing that performance climbed predictably as you poured more compute into training. Then, in September 2024, OpenAI released a model that quietly broke that frame. o1 was not obviously larger than GPT-4o, but it was dramatically better at hard reasoning tasks. The difference was where the compute went: not into training, but into thinking.

The original scaling paradigm and its limits

The dominant scaling recipe came from OpenAI's 2020 scaling laws paper and DeepMind's 2022 Chinchilla paper. Both showed that model quality improved predictably when you scaled parameters, data, and compute together in the right ratios. Chinchilla argued that many models were overtrained on too-small datasets relative to their size, and that a smaller model trained on more tokens could outperform a larger model trained on fewer. These laws shaped every major model from 2020 onward.

The limitation is inherent: training compute is spent once, and the resulting model is fixed. Whatever reasoning capability the model did not learn during training, it cannot acquire later. Hard problems in mathematics, logic, and science require generating and checking intermediate steps, and a model trained only to predict the next token has no built-in mechanism to try again, backtrack, or verify its own work.

What test-time compute actually means

Test-time compute (also called inference-time compute) is compute spent during generation rather than training. There are two main approaches.

The first is sequential scaling: the model generates a long internal chain of thought before producing its final answer, reasoning through intermediate steps the way a human might work through a problem on scratch paper. This is the approach o1 popularized. More thinking steps mean more tokens generated, which means more compute per query.

The second is parallel scaling: generate many candidate solutions simultaneously and use a verifier or voting procedure to pick the best one. A method called majority voting (also called self-consistency) runs the model dozens or hundreds of times on the same question and returns the most common answer. This can be combined with learned reward models that score candidate outputs more precisely than raw vote counting.

Both approaches share a core insight: you can trade generation tokens for capability. A model that generates 10,000 tokens on a hard problem can outperform a model 10 times its size that generates 200 tokens. That is a fundamental reversal of how AI economics had worked.

OpenAI o1: the benchmark shock

OpenAI released o1 on September 12, 2024. The numbers it posted on mathematical reasoning benchmarks were striking. On the 2024 AIME exam, a competition designed to challenge the top high school mathematicians in the United States, GPT-4o solved an average of 12 percent of problems (about 1.8 out of 15). o1 solved 74 percent (11.1/15) with a single attempt per problem, 83 percent (12.5/15) using consensus across 64 samples, and 93 percent (13.9/15) when re-ranking 1,000 samples with a learned scoring function. A score of 13.9 placed it among the top 500 students nationally, above the cutoff for the USA Mathematical Olympiad.

The results extended beyond math. On GPQA Diamond, a benchmark testing PhD-level knowledge in chemistry, physics, and biology, o1 became the first model to exceed the accuracy of human experts with relevant doctorates. On Codeforces competitive programming problems, it ranked in the 89th percentile among human contestants. These were not marginal improvements; they were categorical jumps over GPT-4o.

OpenAI explained the mechanism: o1 was trained with reinforcement learning to generate productive chains of thought. The model learned to recognize its own mistakes, break hard steps into simpler ones, and abandon approaches that were not working. Crucially, its performance scaled smoothly with both more RL training time and more thinking time at inference, suggesting two parallel levers for improvement.

DeepSeek-R1: the open-source answer

In January 2025, the Chinese AI lab DeepSeek released R1, an open-weights reasoning model that matched o1 on most benchmarks while costing roughly 70 percent less to run via API. The technical report explained that DeepSeek-R1-Zero was trained using pure reinforcement learning with no supervised fine-tuning phase, using an algorithm called Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization that drops the separate value function. The model was simply rewarded for producing correct answers, with no human-labeled reasoning traces as starting material.

The results were remarkable. On the same AIME benchmark, DeepSeek-R1 improved accuracy from a baseline of 15.6 percent to 71 percent through extended chain-of-thought generation, reaching 86.7 percent with majority voting across multiple samples. The report also disclosed which approaches did not work: process reward models and Monte Carlo tree search both failed to produce the expected gains, suggesting that longer response chains during RL training may act as implicit inference-time scaling by themselves.

DeepSeek-R1's release changed the field's assumptions about what was required to build a reasoning model. The open weights meant any researcher could inspect, fine-tune, and deploy the approach, and a wave of derivative and competing models followed within weeks.

The cost of thinking longer

Extended chain-of-thought reasoning generates orders of magnitude more tokens than a standard model reply. That compute has to be paid for somewhere. OpenAI's reported inference spend in 2024 reached roughly $2.3 billion, approximately 15 times the reported training cost for GPT-4. Jensen Huang, CEO of Nvidia, stated in 2025 that next-generation reasoning models demand up to 100 times more computational resources than prior generation models, a claim that captures the infrastructure pressure reasoning models impose.

Industry analysts project that inference compute will exceed training compute demand by a factor of 118 by 2026. The AI inference market, estimated at around $106 billion in 2025, is projected to grow to roughly $255 billion by 2030 at a compound annual growth rate near 19 percent. These projections carry real uncertainty, but the directional shift is clear: the balance of AI compute is moving from training clusters to inference clusters.

• A 7B parameter model running extended chain-of-thought can match a 70B dense model on specific hard tasks, compressing the hardware requirement while shifting cost to per-query token generation.
• Majority voting over 64 or more samples multiplies per-query cost by the same factor, so the price of a hard reasoning query can be 50 to 100 times that of a simple one.
• Latency grows with chain length: a model thinking through thousands of intermediate steps adds seconds or minutes to response time, which matters for interactive applications.
• Purpose-built inference chips, including Nvidia's Blackwell architecture (targeting 1.4 exaFLOPS per rack for inference workloads) and Groq's LPU, emerged specifically to address this bottleneck.

What this shift means for the field

Inference-time compute does not replace training-time scaling. It adds a second axis. A model that is smarter from training also reasons better when given more time to think. But the implication is that two models with similar training costs can perform very differently depending on how much generation budget they are allocated, a variable that was previously not part of the comparison.

This also has policy consequences. Existing AI regulation, including the EU AI Act's compute thresholds, is framed around training FLOP counts. A model trained below the regulatory threshold can in principle exceed that threshold's intended capability level by applying heavy inference-time compute. Policymakers are beginning to grapple with this gap.

For practitioners, reasoning models work best on tasks with verifiable answers: math, coding, logic, structured analysis. On tasks without a clear right answer, more thinking does not reliably produce better results. Knowing which type of task you have is the most important factor in deciding whether to use a reasoning model or a faster standard one. Find prompts tested across both categories, rated by the PromptingIndex community, in the prompt library.