Automatic Prompt Engineer: How LLMs Write Better Prompts Than Humans

Prompt engineering is a craft most practitioners learn through trial and error: write a phrasing, test the output, revise the wording, repeat. The time cost is real, and the results depend heavily on the engineer's intuition about what the model responds to. The Automatic Prompt Engineer (APE), introduced at ICLR 2023 in 'Large Language Models Are Human-Level Prompt Engineers' (arXiv:2211.01910) by Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba from the University of Toronto and Vector Institute, turns this problem around. Instead of asking a human to write the instruction, APE asks a language model to propose hundreds of candidate instructions from a few input-output examples, evaluates each candidate using execution accuracy on a validation set, and selects the best one. On 24 NLP tasks from the Instruction Induction benchmark, APE-generated instructions reached human-level or better performance on 21 of 24 tasks, with an interquartile mean (IQM) of 0.810 compared to 0.749 for human-written prompts.

The core idea: instruction as program

APE frames automatic prompting as natural language program synthesis. A conventional program synthesizer searches a space of code strings to find one that satisfies given input-output specifications. APE does the same thing in natural language: given a handful of input-output demonstrations, it searches a space of instruction strings to find one that causes an LLM to reproduce those outputs when given new inputs. The instruction is the program, the LLM is the compiler, and execution accuracy on a held-out validation set is the fitness function. This framing connects automatic prompting to a well-studied research tradition while keeping the entire search space in plain natural language, with no gradient computation or weight updates required.

Generating and scoring instruction candidates

APE generates candidates in two modes. In forward mode, the proposer LLM receives a set of input-output pairs and is asked to write an instruction that could have produced those pairs. In reverse mode, available for models with infilling capability, the instruction slot is left blank inside a structured template and the model fills it in. The paper finds forward mode produces better candidates in most settings. After generation, APE uses a multi-stage evaluation pipeline to avoid scoring all candidates on the full validation set: it filters on a small random subset first, then runs the most promising instructions on progressively larger subsets. Only the top candidates reach a full scoring pass, cutting total API calls significantly.

• Forward mode: the proposer LLM generates instructions that explain given input-output pairs. Up to 250 candidates can be sampled in one batched call.
• Reverse mode: a blank-filling template is used with infilling-capable models; the instruction slot is filled in from context.
• Multi-stage filtering: candidates are scored on growing subsets, so costly full-set evaluation is reserved for top contenders.
• Iterative refinement: the highest-scoring instruction is used to seed a local search, where the LLM generates semantically similar variants for a second scoring pass.
• Scoring function: execution accuracy outperforms log probability as a selection criterion, with better correlation to held-out test performance.

The zero-shot CoT trigger APE discovered

One of the most widely cited results from the APE paper is what happens when APE is applied to the zero-shot chain-of-thought (CoT) setting. The standard zero-shot CoT trigger, introduced by Kojima et al. in 2022, is 'Let's think step by step.' APE was tasked with finding a better instruction using arithmetic reasoning examples. The trigger it selected was: 'Let's work this out in a step by step way to be sure we have the right answer.' This instruction improved performance on MultiArith from 78.7% to 82.0% and on GSM8K from 40.7% to 43.0% when evaluated with InstructGPT (text-davinci-002). The difference is subtle: the original version instructs the model to reason step by step, while the APE-found version adds a self-verification framing ('to be sure we have the right answer') that appears to push the model toward reviewing its arithmetic before committing to a final answer. Both triggers are zero-shot; no demonstrations are needed.

Benchmark results across 24 tasks and BIG-Bench

The Instruction Induction benchmark, introduced by Honovich et al. (2022), contains 24 NLP tasks covering translation, reformatting, classification, and reasoning, each paired with a human-written reference instruction. APE achieves an IQM score of 0.810 with InstructGPT as the executor, compared to 0.749 for human-written instructions, and matches or outperforms humans on 21 of 24 tasks. On BIG-Bench, a separate suite of challenging tasks requiring reasoning and world knowledge, APE achieves comparable or better performance than human prompts on 17 of 21 tasks. The paper also tests prepending APE-selected instructions to standard in-context learning examples, and finds that this consistently improves few-shot accuracy relative to using no instruction, showing that APE-generated prompts generalize beyond the pure zero-shot setting. One important caveat: APE-generated prompts tend to be model-specific. An instruction optimized for InstructGPT often transfers poorly to vanilla GPT-3, and vice versa, so APE should be run separately for each model family.

How to apply APE in practice

The reference implementation is available at github.com/keirp/automatic_prompt_engineer. The algorithm needs three components: a proposer LLM for generating candidates, an executor LLM for scoring them, and a validation set of input-output pairs (typically 20 to 50 examples per task is sufficient). The paper finds that larger, instruction-tuned models produce better candidates in the proposal step, and that diminishing returns appear around 64 candidate samples. Generating more than 64 distinct candidates adds cost without proportional accuracy gains. For users who want to apply the idea without running the full pipeline, the key takeaway is the iterative refinement step: take any high-scoring instruction, ask the LLM to generate semantically similar variants, score the variants on your validation set, and replace the original with the best variant. This local search often improves on the initial result, especially for narrow or domain-specific tasks. PromptingIndex covers APE alongside OPRO (Optimization by Prompting), Plan-and-Solve prompting, and Auto-CoT, all of which address different aspects of reducing the human labor needed to elicit reliable outputs from large language models.