OPRO: How Google DeepMind Taught LLMs to Write Their Own Prompts

Writing a good prompt is hard. Practitioners spend hours drafting, testing, and tweaking instruction text, only to discover that a small wording change shifts accuracy by several percentage points. Optimization by PROmpting, known as OPRO, automates that process by asking a large language model to act as its own prompt engineer. The paper, 'Large Language Models as Optimizers' (arXiv:2309.03409), was submitted to arXiv on September 7, 2023 by seven researchers at Google DeepMind, accepted at ICLR 2024, and released with code at github.com/google-deepmind/opro. The core result: prompts found by OPRO outperform prompts written by humans by up to 8 percentage points on GSM8K and by up to 50 percentage points on select Big-Bench Hard tasks, without changing any model weights.

The paper and its seven authors

The seven authors are Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen, all at Google DeepMind. Quoc V. Le is known for co-developing Neural Architecture Search and the original seq2seq paper. Denny Zhou co-authored the Chain of Thought prompting paper with Wei et al. and has authored several other foundational prompting papers. The combination of optimization theory, NLP, and inference-time prompting expertise in this group shaped the framing of OPRO as a general optimization framework rather than a narrow prompt-search tool. The paper runs to 42 pages with 26 figures and 15 tables, covering both toy optimization problems and real-world task accuracy experiments.

What OPRO does: the meta-prompt and the optimizer loop

OPRO reframes prompt search as an optimization problem where the objective is task accuracy on a small training set. The optimizer is itself an LLM. At each step, the optimizer receives a meta-prompt containing two things: a description of the optimization task (find an instruction that maximizes accuracy on these examples) and an ordered list of previously generated candidate instructions together with their measured accuracy scores. The LLM reads this history and proposes new candidate instructions. The new candidates are evaluated by running them on the training set, and the results are appended to the meta-prompt for the next round. The loop continues for a fixed number of steps, typically 20 to 40 steps in the paper's experiments. The best-scoring instruction found at any point in the trajectory is taken as the final prompt.

The meta-prompt structure is what distinguishes OPRO from prior automatic prompt engineering methods. Instead of sampling randomly or applying gradient-based updates to continuous embeddings, the optimizer LLM reads the history of attempts and scores as natural language and generates new candidates that are informed by what has and has not worked before. This is analogous to how a human engineer might look at a log of previous prompt attempts and their outcomes before drafting the next version. The paper describes this as optimization via natural language, where the optimizer's 'gradient' is the textual description of past performance patterns.

• Meta-prompt input: task description plus an ordered list of prior (instruction, accuracy) pairs.
• Optimizer LLM: generates several new candidate instructions per step by reading the meta-prompt.
• Evaluation: each candidate is scored on a small held-out training subset (the paper uses 3.5% of GSM8K training data).
• New (instruction, score) pairs are appended to the meta-prompt for the next step.
• Loop runs for 20 to 40 steps; the highest-scoring instruction across all steps is the output.
• No gradient computation, no weight updates, and no embedding-space search are involved.

Benchmark results: GSM8K and Big-Bench Hard

The paper evaluates OPRO on two primary benchmarks. GSM8K is a dataset of 8,500 grade-school math word problems (7,473 training, 1,319 test). Big-Bench Hard is a subset of 23 hard tasks from the BIG-Bench benchmark where GPT-4 and PaLM 2 were not yet near ceiling performance at the time of writing. On GSM8K, using PaLM 2-L as both the optimizer and the scorer, OPRO finds an instruction that achieves 80.2% accuracy on the test set. The paper's human-designed baseline using the same model and the standard Chain of Thought instruction reached 72.0%. The gap is 8.2 percentage points. On Big-Bench Hard, gains of up to 50 percentage points were observed on individual tasks compared to human-designed instructions, though results varied substantially by task.

The paper also tested GPT-3.5 and GPT-4 as both the optimizer LLM and the scorer LLM in various combinations. Using GPT-4 as the optimizer and GPT-3.5 as the scorer, OPRO found prompts that improved over the GPT-3.5 human-designed baseline on most Big-Bench Hard tasks. A follow-up paper from USC researchers (arXiv:2405.10276, May 2024) found that OPRO is substantially less effective when smaller models (LLaMA-2-7B, LLaMA-2-13B, Mistral 7B) serve as the optimizer, because those models lack the inference capability needed to reason over the history of past attempts and generate meaningfully improved candidates. For large models such as Gemini Pro, OPRO continued to show gains, producing instructions like 'To attain the utmost precision in solving diverse grade school mathematical problems, meticulously adhere to this comprehensive and rigorously developed methodology' that outperformed zero-shot CoT.

OPRO beyond prompt optimization: linear regression and TSP

The paper opens with two proof-of-concept experiments on classical optimization problems before moving to prompt search. In the linear regression experiment, the optimizer LLM is given a meta-prompt listing pairs of (w, b) parameter values and their associated mean squared error, and is asked to propose new (w, b) values. The LLM converges to the true parameters through iterative refinement. In the traveling salesman experiment, the LLM is given a list of cities with distances and a set of previously tried routes with their total distances, and is asked to propose shorter routes. These experiments demonstrate that the OPRO loop is a general optimization framework: the scorer is a black-box function that maps a candidate solution to a scalar, and the optimizer LLM proposes new candidates in natural language at each step. Prompt optimization is the application with the most practical relevance, but the underlying mechanism is not specific to prompts.

Practical use: when to use OPRO and when not to

OPRO is most effective when three conditions hold. First, the optimizer LLM must be large enough to reason over a history of attempts and generate meaningfully different candidates. The USC follow-up paper confirmed that models below roughly 70 billion parameters show limited gains, and the original paper used PaLM 2-L and GPT-4 as optimizers. Second, there must be a reliable automated scoring function. OPRO needs to evaluate dozens of candidate instructions on a labeled subset; tasks without ground-truth labels or with subjective quality criteria cannot be scored automatically and are therefore not good fits. Third, the training subset used for scoring must be representative of the test distribution. The paper uses a randomly sampled 3.5% subset of GSM8K for training evaluation, which is small enough to make each step cheap but large enough to produce a stable accuracy signal.

OPRO is not appropriate as a replacement for all prompt engineering. For tasks where a human expert can write a near-optimal prompt in a single attempt, the overhead of running 20 to 40 optimizer steps is unlikely to be worth the compute cost. For tasks with no evaluation set, OPRO cannot function at all. The technique also inherits a quirk that the paper itself notes: the best instructions found by OPRO are sometimes counterintuitive or verbose in ways that seem unlikely to a human engineer. The instruction 'Let us work this out in a step by step way to be sure we have the right answer' on some tasks outperformed 'Let's think step by step' by a measurable margin despite being superficially similar. This suggests that the instruction text acts as a soft key into the model's latent knowledge, and that small phrasings can make a real difference in ways that are hard to predict without empirical search. OPRO is the mechanism that makes that search systematic.