Medprompt: How Microsoft Combined kNN Few-Shot, Self-Generated CoT, and Choice Shuffle Ensembling to Hit 90.2% on MedQA

For most of the deep-learning era, the assumption was that specialist performance requires specialist training. If you needed a model that could answer medical licensing exam questions, you fine-tuned a language model on medical text, clinical notes, and curated question-answer pairs from biomedical corpora. The resulting models, including Med-PaLM and Med-PaLM 2, were competitive because they combined a powerful base model with domain-specific supervision. The paper 'Can Generalist Foundation Models Outcompete Special-Purpose Fine-Tuning? A Case Study in Medicine,' submitted to arXiv on November 28, 2023 (arXiv:2311.16452) by Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Hackney, and Eric Horvitz from Microsoft Research, challenged that assumption. The authors introduced a prompting framework called Medprompt that combines three composable techniques: dynamic kNN-based few-shot example selection, self-generated chain-of-thought reasoning, and a choice shuffle ensembling step. Applied to GPT-4, Medprompt reached 90.2% accuracy on MedQA, surpassing Med-PaLM 2 at 86.5% (achieved with intensive fine-tuning) and setting a new state of the art on nine of nine medical benchmarks tested in the paper.

The baseline problem: why random few-shot is not enough

The starting point for the Medprompt ablation is GPT-4 with no prompting strategy beyond a zero-shot instruction: 81.7% on MedQA. Adding random few-shot examples from a training split, without any selection logic, raises the score to 83.9%. Adding chain-of-thought reasoning to those randomly selected few-shot examples brings it to 87.3%. These three numbers tell a clear story: both few-shot selection and chain-of-thought contribute independently, and their contributions are additive rather than overlapping. But neither intervention alone closes the gap to 90%. The paper's core claim is that the remaining gain comes from two specific improvements: replacing random example selection with kNN retrieval, and addressing order-sensitivity bias through a choice shuffle ensemble. Each component is ablated separately so the contribution of each can be measured.

Dynamic few-shot selection via kNN retrieval

Standard few-shot prompting selects examples either by hand or at random from a training pool. The limitation is that a fixed or random set of examples may not be similar to the specific test question being answered. Medprompt replaces random selection with kNN retrieval over a pre-computed embedding index of the training examples. For each test question, the system retrieves the k nearest neighbors by semantic similarity in embedding space and uses those as the few-shot demonstrations. The intuition is that a multiple-choice question about cardiac physiology will be better served by examples that are also about cardiac physiology than by examples drawn from clinical pharmacology or medical genetics. Moving from random few-shot to kNN few-shot while holding chain-of-thought constant raises the MedQA score from 87.3% to 88.4%, a gain of 1.1 percentage points that comes entirely from better example selection. The authors use k = 5 few-shot examples in the standard configuration reported as the 90.2% result.

Self-generated chain of thought

Standard chain-of-thought prompting requires a human to write out reasoning steps for each demonstration example. This is time-consuming and introduces a bottleneck when the training pool is large. Medprompt automates it: GPT-4 is used to generate the chain-of-thought reasoning for each training example before that example is stored in the retrieval index. The model writes out its reasoning for each question and the answer it arrives at, and that self-generated explanation becomes part of the few-shot demonstration. This has two practical advantages. First, it removes the manual annotation step, making Medprompt deployable to any domain where labeled multiple-choice data exists, without requiring human experts to write reasoning traces. Second, the model's own reasoning patterns tend to be a closer match to what the model will actually generate at inference time, compared to reasoning steps written by a human annotator who may frame explanations differently than the model would.

Choice shuffle ensembling and order sensitivity

Multiple-choice language model benchmarks have a known failure mode: model accuracy is sensitive to which answer option appears in position A, B, C, or D. A model may prefer option A at a rate higher than chance, or may shift its answer if the same correct content is moved from position B to position D. Medprompt addresses this with a choice shuffle ensemble. For each test question, the system runs the model multiple times, each time with the answer choices presented in a different order. The final prediction is a majority vote over those runs. In the standard Medprompt configuration, five ensemble shuffles are used. Moving from kNN few-shot plus chain-of-thought (88.4%) to the full Medprompt system with choice shuffle ensembling (90.2%) adds 1.8 percentage points, the largest single-step gain in the ablation. The paper notes that ensembling with shuffled choices reduces sensitivity to positional bias and improves calibration across all nine benchmarks.

• Nori, Lee, Zhang, Hackney, Horvitz (arXiv:2311.16452, submitted November 28, 2023): Microsoft Research.
• MedQA ablation: zero-shot 81.7%, random few-shot 83.9%, random few-shot plus CoT 87.3%, kNN few-shot plus CoT 88.4%, full Medprompt with choice shuffle 90.2%.
• Benchmark comparison: Med-PaLM 2 with intensive fine-tuning reached 86.5% on MedQA; Medprompt exceeded it by 3.7 percentage points with no fine-tuning.
• Standard configuration: k = 5 kNN-retrieved few-shot examples, 5 ensemble shuffles.
• Self-generated CoT: GPT-4 writes its own reasoning traces for training examples, removing the need for human annotation.
• Results replicated across nine medical benchmarks: MedQA, MedMCQA, PubMedQA, and six MMLU medical subsets (Clinical Knowledge, Medical Genetics, Anatomy, Professional Medicine, College Biology, College Medicine).
• Medprompt achieved state-of-the-art on all nine benchmarks evaluated in the paper.

What Medprompt generalizes to

The medical domain was the test case, not the constraint. The components of Medprompt, kNN few-shot retrieval, self-generated chain-of-thought annotation, and choice shuffle ensembling, are each general-purpose techniques with no dependency on medical content. The paper demonstrates generalization by applying Medprompt to a set of non-medical benchmarks from the MMLU suite, including tasks covering law, economics, and other professional domains, and observing consistent improvements over both zero-shot and random few-shot baselines. The authors argue that the medical case is instructive precisely because it is one of the domains where fine-tuning was most expected to be necessary: the knowledge is specialized, the stakes are high, and several well-resourced fine-tuned models existed as comparison points. The fact that a composable prompting strategy closes the gap to fine-tuned performance in that domain suggests that the technique has meaningful applicability wherever labeled multiple-choice data exists and retrieval infrastructure can be set up. PromptingIndex covers Medprompt alongside related techniques including kNN-based dynamic few-shot selection, chain-of-thought prompting, self-consistency via majority vote, and ensembling strategies for reducing order-sensitivity bias in multiple-choice evaluation.