GRPO: How DeepSeek Replaced PPO's Critic With a Group of Answers

Training a language model to reason is, at its core, a reinforcement learning problem. The model generates a chain of thought and an answer, earns a reward if the answer is correct, and updates its weights to make correct answers more likely. The dominant algorithm for that weight update was Proximal Policy Optimization, or PPO, borrowed from robotics and games. PPO works well, but it carries a heavyweight piece of infrastructure: a critic, a second copy of the model trained to predict expected future reward. For a 7-billion-parameter model, the critic roughly doubles GPU memory at training time. In February 2024, researchers at DeepSeek introduced Group Relative Policy Optimization (GRPO), which removes the critic entirely by replacing it with a simple statistical baseline computed across a batch of sampled answers to the same question.

The paper: origin and authors

GRPO was introduced in 'DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models' (arXiv:2402.03300), submitted February 5, 2024, by Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. The paper's primary goal was to improve mathematical reasoning in open-source models using a combination of web-scale math data and a new RL algorithm. DeepSeekMath 7B, the model the paper introduced, achieved 51.7% on the competition-level MATH benchmark without external calculators or voting techniques, approaching Gemini-Ultra and GPT-4 at the time. With self-consistency over 64 samples, it reached 60.9% on MATH. The algorithm that powered the RL stage of that training was GRPO.

Why PPO needs a critic and why that is expensive

To understand what GRPO removes, it helps to be precise about what the PPO critic does. In standard policy gradient RL, the update signal for a given action (a generated token or response) is the reward minus a baseline. Without a baseline, high-variance rewards produce noisy gradients that destabilize training. The PPO critic, a neural network called the value function, learns to predict the expected cumulative reward from any given state. Subtracting this prediction from the actual reward produces the advantage estimate, a lower-variance measure of how much better than average a particular action was.

For language model RL, the critic is typically a copy of the language model itself with a regression head attached. That means the critic has roughly the same parameter count as the policy being trained. During PPO, four objects must fit in GPU memory simultaneously: the policy being trained, a frozen reference copy of the policy used for KL penalty computation, the critic, and a frozen reference copy of the critic. For a 7B model, this is roughly equivalent to holding 28 billion parameters in memory across the four copies, plus activations and optimizer states. At larger model scales, this becomes a severe constraint.

The GRPO mechanism: groups instead of a critic

GRPO's key insight is that for domains with verifiable rewards, a simple within-group baseline can replace the critic. For each training prompt, GRPO samples a group of G responses from the current policy rather than a single response. All G responses are scored by a reward function. The baseline for each response is the mean reward of the group it belongs to. The advantage for each response is its reward minus the group mean, divided by the group standard deviation. A response that is the only correct answer in a group of mostly wrong answers gets a high positive advantage. A response that is correct but no better than the average of the group gets a small advantage. A response that is wrong gets a negative advantage proportional to how far below the group mean it falls.

The policy update then maximizes the expected advantage-weighted log-probability of the generated tokens, subject to the same PPO-style clipping that prevents the policy from taking too large a step in any single update. GRPO still maintains a frozen reference policy for the KL divergence penalty that keeps the model from straying too far from its starting point. But the critic is gone. Two of the four objects that PPO requires in memory are collapsed into a single statistical computation over a batch of samples.

• Group size G is a hyperparameter; DeepSeekMath and DeepSeek-R1 used groups of 8 to 64 responses per prompt during different training stages.
• Rewards are binary for verifiable tasks: 1 for a correct final answer, 0 for incorrect. This is verified by string matching for math and by sandbox execution for code.
• The KL penalty is computed between the updated policy and the frozen reference policy, weighted by a coefficient beta, serving the same anchoring role as in DPO and PPO.
• GRPO is online: responses are sampled from the current policy at each training step, not from a fixed static dataset. This distinguishes it from offline algorithms like DPO.
• Removing the critic eliminates the value function loss that PPO computes in parallel with the policy loss, simplifying the training loop and reducing the number of distinct gradient computations per step.

DeepSeek-R1 and the emergence of reasoning

GRPO gained widespread attention when DeepSeek released the R1 model and its training report in January 2025 (arXiv:2501.12948). DeepSeek-R1 is fine-tuned from DeepSeek-V3, a 671-billion-parameter mixture-of-experts model. The training pipeline uses GRPO as the RL optimizer throughout.

The report describes an intermediate model called DeepSeek-R1-Zero, trained by applying GRPO directly to DeepSeek-V3 without any supervised fine-tuning as a warm-up. R1-Zero was trained purely with RL. The result was that the model naturally learned to generate progressively longer chains of thought as training progressed, reflecting on its own intermediate answers and exploring alternative solution paths, without those behaviors being explicitly programmed. The model's average response length on reasoning tasks increased substantially over the course of training as the model discovered that longer deliberation improved accuracy. R1-Zero reached performance comparable to o1-preview on the AIME 2024 benchmark, which the report tracked as a metric during training, demonstrating that pure RL from verifiable rewards can produce strong reasoning capabilities.

The full DeepSeek-R1 model uses a multi-stage pipeline: a cold-start supervised fine-tuning phase on long chain-of-thought examples, a large-scale GRPO RL phase with verifiable rewards on math and code, a second SFT phase on 800,000 examples combining 600,000 reasoning examples and 200,000 general-purpose examples, and a final combined RLVR and RLHF stage that adds preference-based rewards for helpfulness and harmlessness alongside the verifiable rewards. DeepSeek-R1 matches or exceeds o1 on most reasoning benchmarks in the published results.

What GRPO requires from the reward signal

GRPO's within-group baseline works cleanly when rewards are binary or at least clearly ordered. If most responses in a group are wrong and one is correct, the correct response gets a strongly positive advantage signal and the wrong ones get negative signals. Training converges because the signal is unambiguous. If rewards are dense and continuous, the within-group variance is lower, the advantage estimates are less decisive, and GRPO can still converge but the benefits over a learned critic are less pronounced.

This is why GRPO has been adopted almost exclusively for reinforcement learning with verifiable rewards (RLVR): math problems where the final numerical answer can be checked by string matching, and coding problems where correctness can be verified by running test cases. For open-ended generation tasks where correctness is subjective and requires a learned reward model rather than a rule-based verifier, PPO or DPO typically remain the tools of choice because a learned reward model can provide the dense, nuanced signal that GRPO's simple group statistics cannot.

Memory savings and practical adoption

The practical advantage of GRPO over PPO for reasoning research is the memory reduction. Removing the critic frees roughly 25 to 30 percent of the total memory that PPO's four-model setup requires. At 7B parameters, this means GRPO training can run on hardware that PPO could not. At 70B parameters, the savings make the difference between fitting on two nodes and requiring four. Several post-DeepSeek reasoning model releases from academic groups cited GRPO's memory efficiency as a key factor in their ability to run RL training without industry-scale GPU clusters.

The group sampling approach also provides a side benefit for data efficiency. By sampling G responses per prompt and scoring all of them, GRPO extracts a learning signal from every response in the group, not just the single response that PPO or DPO would generate. A group of eight responses per prompt means that each training prompt contributes eight advantage estimates per update step. This multi-sample structure tends to produce more stable gradient estimates and can reduce the number of unique prompts needed to achieve a given level of training convergence. PromptingIndex tracks GRPO-trained model releases and RLVR training recipes across open-source reasoning model projects, including implementations that apply the algorithm to smaller models and non-math domains.