Reflexion: Teaching Language Agents to Learn from Failure Without Fine-Tuning

When a language model agent fails a task, the standard options are to retry with a different prompt, collect the failure as training data, or fine-tune the model. All three are expensive in different ways: retrying without memory repeats the same mistakes, collecting data at scale is slow, and fine-tuning requires significant compute and infrastructure. Reflexion, introduced in 'Reflexion: Language Agents with Verbal Reinforcement Learning' (arXiv:2303.11366, Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao, first submitted March 2023), takes a different path. Instead of updating model weights, Reflexion agents write a natural language self-critique after each failed trial and store it in a memory buffer that conditions every subsequent attempt. The result is a lightweight feedback loop that achieved 91% pass@1 on HumanEval coding benchmarks, surpassing the GPT-4 baseline of 80% at publication time, a 22% absolute gain on AlfWorld sequential decision-making tasks in 12 learning steps, and a 20% improvement on HotPotQA multi-hop reasoning questions.

The core problem: agents that cannot remember their mistakes

LLM agents built on ReAct or chain-of-thought prompting generate actions and observations in sequence, but each trial starts from the same static prompt with no knowledge of past failures. If the agent chose the wrong object in a text game, picked a broken code path, or retrieved an irrelevant document, the next trial has no access to that information unless it is explicitly injected. Traditional reinforcement learning solves this by using gradient-based updates to push the policy away from failure states, but gradient RL requires thousands of environment interactions and model fine-tuning, which is impractical for large frozen models. The Reflexion paper frames this as a credit assignment problem in natural language: how do you give an agent a useful signal that points at what specifically went wrong and what to do instead, using only inference-time context?

Three components: Actor, Evaluator, and Self-Reflection

Reflexion defines three interacting modules. The Actor is any LLM prompted to take actions in an environment. The paper experiments with both chain-of-thought and ReAct variants. The Evaluator scores the Actor's output at the end of each trial. Depending on the task, this can be an exact-match comparator for reasoning, a heuristic function for game states, or a second LLM used as a classifier. The Self-Reflection model takes the trial trajectory, the Evaluator's score, and the existing memory buffer, then generates a short verbal summary of what went wrong and what the agent should do differently. That summary is appended to a persistent long-term memory store, which is prepended to the Actor's prompt on the next trial.

• Actor: generates actions and text given the current state. Can be any prompted LLM, including ReAct and chain-of-thought variants.
• Evaluator: produces a reward signal after each trial. Options include exact-match grading, task-specific heuristics, and LLM-as-judge scoring.
• Self-Reflection model: converts the reward and trajectory into a natural language critique stored in long-term memory for future trials.
• Short-term memory: the trajectory from the current episode (actions and observations), visible only within the active trial.
• Long-term memory: accumulated self-reflections from all past trials, prepended to the Actor's context at the start of each new attempt.

How the feedback loop works in practice

Consider a coding task on HumanEval. On trial one, the Actor generates a function implementation that fails two hidden test cases. The Evaluator runs the code and returns failure. The Self-Reflection model is then prompted with the failed code, the error messages, and any previous reflections in memory. It writes something like: 'The function assumes the input list is always sorted, but the problem states otherwise. On the next attempt, sort the list before processing.' That text is saved to long-term memory. On trial two, the Actor reads its prior reflection alongside the original problem statement and produces a revised implementation. The Self-Reflection model does not need special training or separate weights; it is the same base model prompted differently, which means the full setup runs with no additional model downloads or fine-tuning infrastructure.

For sequential decision-making tasks on AlfWorld (a text-based household environment with tasks like 'put the soapbar in the cabinet'), the reflection step identifies which object was in the wrong location, which action sequence failed, or which room the agent searched unnecessarily. After each failed episode the memory grows with a short note about the error. Over 12 episodes the Reflexion agent accumulated enough self-corrective guidance to improve by 22 absolute percentage points over the ReAct baseline.

Benchmark results across coding, reasoning, and decision-making

The paper tests Reflexion on three categories of tasks. On HumanEval, the Python function completion benchmark, Reflexion reaches 91% pass@1. The GPT-4 baseline without Reflexion sat at 80% at the time of publication, making this an 11 percentage point gain for coding alone. The paper also introduces LeetcodeHardGym, a custom RL environment of 40 hard-difficulty LeetCode problems across 19 programming languages, to test whether Reflexion generalizes beyond HumanEval-style completions. On HotPotQA, a multi-hop question answering benchmark that requires chaining facts across multiple retrieved documents, Reflexion improves accuracy by 20% over the baseline agent. On AlfWorld sequential decision-making, the 22% improvement over 12 trials is achieved without any gradient update to the underlying model. Ablation experiments in the paper show that removing the long-term memory store and only using single-trial reflections reduces gains substantially, confirming that accumulating reflections across trials is the key mechanism rather than any single self-critique.

• HumanEval pass@1: 91% with Reflexion versus 80% for GPT-4 baseline (11 point improvement).
• AlfWorld sequential decision-making: 22% absolute improvement over ReAct baseline across 12 iterative trials.
• HotPotQA multi-hop reasoning: 20% improvement over the baseline agent.
• No weight updates required: the entire improvement comes from in-context memory, not fine-tuning.
• LeetcodeHardGym: a new benchmark of 40 hard LeetCode problems in 19 languages released with the paper.

Limitations and when Reflexion falls short

Reflexion depends on two capabilities the base model must already possess: the ability to accurately evaluate its own outputs, and the ability to generate reflections that correctly identify causes of failure rather than post-hoc rationalizations. If the model cannot distinguish a passing solution from a failing one, the Evaluator step produces noise and the memory buffer fills with unhelpful or misleading notes. The paper notes this explicitly: performance scales with the quality of the LLM's self-evaluation. For tasks where success is unambiguous, such as code that either passes or fails a unit test, Reflexion has a reliable external signal. For tasks where quality is subjective, the feedback loop is weaker. A second practical limit is context length. Long-term memory grows with each trial, and at some point the accumulated reflections push older task context out of the context window. The paper uses a simple sliding window, keeping only the most recent reflections when the buffer overflows. Finally, Reflexion assumes a finite number of retries. It is not a guarantee of eventual success; it is a method for making each retry more informed than the last. The code and datasets are available at github.com/noahshinn024/reflexion. PromptingIndex covers Reflexion alongside Self-Refine, ReAct, and Auto-CoT as part of its series on iterative and self-corrective prompting techniques.