New AI Research Reveals Complexity-Dependent Crossover Between Retrieval Paradigms in Tool-Augmented LLM Agents

As artificial intelligence systems evolve from simple conversational chatbots into autonomous agents capable of complex multi-step reasoning, researchers face a fundamental architectural challenge: how to equip large language models with the right external tools. Modern tool-augmented language models rely on vast registries containing tens of thousands of application programming interfaces to execute specialized tasks. However, finite context windows strictly prohibit injecting every available tool definition into a single prompt. Consequently, finding the correct API efficiently has become one of the most critical bottlenecks in modern AI engineering.

A team of researchers consisting of Mingfei Zhang, Amit Surana, Ziauddin Majid, Chenxi Wang, and Michael Xu has released a comprehensive study examining this exact problem. Their work provides a deep dive into how LLMs discover relevant APIs from massive registries before planning their execution steps. The study contrasts the dominant industry standard—known as Retrieve-then-Plan—with an alternative paradigm called Plan-then-Retrieve, uncovering a distinct complexity-dependent crossover that could fundamentally alter how developers design agentic AI architectures moving forward.

Understanding the Battle Between Retrieve-then-Plan and Plan-then-Retrieve

To appreciate the significance of the new research, one must understand how tool-augmented LLM agents traditionally operate. The prevailing approach in the field is the Retrieve-then-Plan paradigm. In this standard pipeline, when a user submits a query, the system embeds the text and queries a database using approximate nearest-neighbor search to retrieve a handful of similar tools directly. Once those tools are pulled into the context window, the language model attempts to plan the execution sequence using only those pre-selected functions.

While straightforward, Retrieve-then-Plan operates under the assumption that the raw user query contains all the semantic signals necessary to locate the correct tools. However, complex user requests often require multiple distinct capabilities that may not be explicitly named or described in a single sentence.

To address this limitation, the researchers studied an alternative workflow: the Plan-then-Retrieve paradigm. In this setup, instead of searching the registry immediately using the raw user prompt, the language model first analyzes the request and generates what the authors call Abstract Tool Stubs. These stubs are hypothesized function signatures that describe the specific capabilities needed to solve the problem. These generated stubs then serve as the retrieval queries against the API registry, rather than the original user prompt itself.

To evaluate these competing strategies rigorously, the research team tested both paradigms on ToolBench, a massive benchmark comprising 46,984 unique API endpoints categorized across 50 distinct domains. The evaluation was split into three specific query subsets designed to test varying degrees of complexity: single-tool queries, intra-category multi-tool queries where all required APIs belong to the same functional domain, and cross-category multi-tool queries requiring disparate functional capabilities.

The Complexity-Dependent Crossover and Empirical Findings

The empirical results of the study challenge the notion that one retrieval paradigm is universally superior to the other. Instead, the researchers discovered a clear complexity-dependent crossover between Retrieve-then-Plan and Plan-then-Retrieve.

According to the evaluation data, Retrieve-then-Plan consistently outperforms Plan-then-Retrieve when handling simple, single-tool queries. When a user asks an agent to perform a straightforward task that maps cleanly to a single API, skipping the intermediate generation step of creating abstract stubs is faster, more accurate, and more efficient.

However, the dynamics shift dramatically when query complexity increases. For multi-tool queries—particularly those spanning multiple functional categories—Plan-then-Retrieve outperforms Retrieve-then-Plan. When an agent is tasked with a multifaceted objective, allowing the LLM to first hypothesize the functional sub-needs via abstract stubs yields much better retrieval targets from the large registry than searching with the raw, overarching user prompt.

Crucially, the research team verified that this crossover behavior is not an artifact of specific experimental settings. The crossover holds steady across every tested retrieval budget, ranging from retrieving just one top candidate up to twenty candidates. Furthermore, the findings are robust across varying registry scales, holding true whether the system searches through a modest collection of 1,000 APIs or the full scale of nearly 47,000 endpoints. The consistency of these results across different model families, including Gemini 2.5 Flash and gpt-oss-120b, underscores the fundamental nature of the trade-off.

Stub Hallucination and the Token Overhead Dilemma

A central contribution of the paper is the characterization of a specific failure mode intrinsic to the Plan-then-Retrieve paradigm: stub hallucination. During the initial planning phase, the language model occasionally invents plausible-sounding but entirely non-existent API names and function signatures.

The researchers characterize this phenomenon as a double-edged sword. On simple queries, stub hallucination acts as a liability because the LLM’s generated stubs abstract away proper-noun signals and exact keywords that Retrieve-then-Plan can match directly against the registry. On complex queries, however, this exact same mechanism serves as a powerful asset. By breaking down a complex, multifaceted intent into functional sub-needs—even if some generated names are slightly imprecise—the model successfully decomposes the problem, enabling the retrieval system to pull more relevant and diverse tool definitions than a direct keyword or vector search could manage.

This trade-off extends beyond accuracy into the realm of computational efficiency and inference cost. The study analyzes the token overhead associated with each approach, revealing that Plan-then-Retrieve requires an extra discovery call. This additional generation step roughly doubles the total inference cost for the agentic pipeline. For simple, single-tool queries, this extra cost represents an unnecessary penalty that hurts performance. For complex, multi-tool queries, however, the investment pays off handsomely through superior task completion and tool discovery.

These insights point toward a practical path forward for system architects: complexity-adaptive routing. Rather than forcing an application to rely exclusively on either Retrieve-then-Plan or Plan-then-Retrieve for all incoming traffic, systems could dynamically route queries based on estimated complexity. Intriguingly, the authors note that such routing can be driven by a zero-cost signal already present within the standard pipeline—specifically, the top-1 retrieval score generated during the initial Retrieve-then-Plan phase.