Buffer of Thoughts: How a Shared Thought-Template Library Cuts Reasoning Costs by 88 Percent

Every multi-step reasoning framework before Buffer of Thoughts treated each problem as a fresh start. Chain-of-Thought prompts a model to produce a sequence of intermediate steps in a single pass. Tree of Thoughts branches into multiple candidate paths and evaluates them. Graph of Thoughts allows arbitrary merges between reasoning nodes. All three generate their reasoning structure from scratch for every new query, which means they repeatedly rediscover the same high-level problem-solving patterns, burning tokens and cost each time. Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E. Gonzalez, and Bin Cui identified this redundancy as the core inefficiency to address. Their paper 'Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models,' submitted to arXiv on June 6, 2024 (arXiv:2406.04271) and accepted as a Spotlight at NeurIPS 2024, introduces a shared library of reusable high-level reasoning templates that any problem can retrieve and adapt, rather than constructing reasoning structures from zero each time.

The meta-buffer and the thought-template abstraction

The central data structure in BoT is the meta-buffer: a persistent store of thought-templates distilled from solved problems across diverse tasks. A thought-template is not a worked example or a few-shot demonstration. It is a high-level description of a reasoning process, stripped of problem-specific content but preserving the structural pattern that made the solution work. A template for a logic puzzle might encode the pattern of constraint propagation and backtracking without encoding the specific constraints. A template for a counting task might encode the pattern of partitioning, counting independently, and summing. Because the template captures structure rather than content, it can be retrieved for a new problem in the same task family and instantiated with the specific details of that problem without modification. This distinction between the reusable template and the problem-specific instantiation is what allows BoT to separate the expensive work of discovering a good reasoning structure, done once when a problem is solved and its pattern is distilled, from the much cheaper work of applying that structure to a new problem.

Retrieval and adaptive instantiation at inference time

When a new problem arrives, BoT performs retrieval against the meta-buffer to find the thought-template most relevant to the current problem type. The retrieval step does not generate a new reasoning structure; it identifies which pre-distilled template to use. The model then performs adaptive instantiation: it receives the retrieved template along with the problem and produces a reasoning chain that follows the template's structural pattern while filling in the specific details of the current problem. The paper describes this as a two-phase process at inference time, retrieval followed by instantiation, that is substantially lighter than the process of generating and evaluating multiple reasoning paths from scratch. The key engineering challenge is that the quality of BoT depends on the quality and breadth of templates in the meta-buffer, so the buffer must grow in a principled way as the system encounters new problem types.

The buffer-manager and how the library grows

The buffer-manager is the component responsible for keeping the meta-buffer current. After a problem is solved, the buffer-manager determines whether the reasoning process used to solve it represents a structurally novel pattern not already captured by existing templates. If it does, the buffer-manager distills the pattern into a new thought-template and adds it to the meta-buffer. If the pattern is already well-represented, the existing template may be updated or the new instance discarded. This update mechanism prevents the meta-buffer from growing unboundedly with near-duplicate templates while ensuring it gains coverage as the system encounters genuinely new problem structures. The authors describe the buffer-manager as guaranteeing scalability and stability: scalability because the meta-buffer can grow to cover more task families, and stability because redundant templates do not accumulate and degrade retrieval quality. The paper's GitHub project at github.com/YangLing0818/buffer-of-thought-llm includes an initial meta-buffer pre-populated with templates from the paper's ten benchmark tasks.

• Yang et al. (arXiv:2406.04271, submitted June 6, 2024; NeurIPS 2024 Spotlight): authors from Peking University and UC Berkeley; code at github.com/YangLing0818/buffer-of-thought-llm.
• Meta-buffer: persistent store of high-level thought-templates distilled from solved problems; templates encode structural reasoning patterns, not problem-specific content.
• Buffer-manager: dynamically updates the meta-buffer after each solved problem, adding structurally novel templates and discarding near-duplicates to maintain retrieval quality.
• Benchmark results on 10 reasoning-intensive tasks: 11% improvement on Game of 24 over previous SOTA, 20% on Geometric Shapes, and 51% on Checkmate-in-One.
• Cost: BoT requires only 12% of the token cost of multi-query prompting methods such as Tree of Thoughts and Graph of Thoughts on average, an approximately 88% reduction.
• Scale transfer: Llama3-8B with BoT shows potential to match or exceed the performance of Llama3-70B without BoT on several benchmarks.

Benchmark results: 51 percent on Checkmate-in-One at 12 percent of the cost

The paper's most striking result is on Checkmate-in-One, a chess endgame task where the model must identify the single move that delivers checkmate. This task requires precise constraint satisfaction over a large structured state space, and it has historically been resistant to improvement by standard prompting methods. BoT achieves a 51% improvement over the previous state-of-the-art result on this task. The Game of 24 benchmark, a classic test of multi-step arithmetic reasoning where four numbers must be combined with arithmetic operations to reach 24, shows an 11% improvement. Geometric Shapes, which tests spatial and compositional reasoning, shows 20% improvement. These gains are achieved while requiring only 12% of the token cost of methods like Tree of Thoughts and Graph of Thoughts, which generate and evaluate multiple independent reasoning paths. The cost reduction follows directly from the BoT architecture: rather than generating many candidate paths and scoring each one, BoT retrieves one relevant template and instantiates it once. Most of the reasoning machinery is moved offline into template distillation, where it can be amortized across all future problems that share the same structural pattern.

Where BoT fits among other structured reasoning methods

BoT sits in a different design space from Chain-of-Thought, Tree of Thoughts, and Graph of Thoughts. Those methods answer the question of how reasoning should be structured within a single problem. BoT answers the question of how reasoning patterns discovered across many problems can be stored and reused. In practice, BoT and those methods are complementary rather than competing: the thought-templates in the meta-buffer can themselves encode tree-shaped or graph-shaped reasoning patterns, so BoT can serve as a retrieval layer on top of any structured reasoning scheme. The scale-transfer result reported in the paper is the most consequential practical finding: a Llama3-8B model equipped with a well-populated meta-buffer shows potential to match outputs from a Llama3-70B model running standard prompting. If that finding holds broadly across task families and template libraries, it suggests that the meta-buffer encodes reasoning capability that would otherwise require a much larger model to derive from scratch. PromptingIndex covers BoT alongside Graph of Thoughts, Tree of Thoughts, and Skeleton of Thought in its series on structured reasoning frameworks that treat inference-time computation as a resource to be allocated deliberately rather than consumed uniformly.