New AI Research Reveals Complexity-Dependent Crossover Between Tool Retrieval Paradigms for LLM Agents

A team of researchers has released a comprehensive new study examining how tool-augmented Large Language Model (LLM) agents discover and select relevant Application Programming Interfaces (APIs) from massive registries. Authored by Mingfei Zhang, Amit Surana, Ziauddin Majid, Chenxi Wang, and Michael Xu, the research investigates the fundamental trade-offs between two competing paradigms for tool retrieval: Retrieve-then-Plan and Plan-then-Retrieve.

As language models are increasingly deployed to execute complex, multi-step workflows, their utility heavily relies on interacting with external digital environments via APIs. However, because modern tool registries contain tens of thousands of endpoints, finite context windows strictly prohibit injecting every available tool definition directly into a single prompt. Consequently, efficient and accurate tool retrieval has emerged as a central bottleneck in agent architecture.

The Clash of Paradigms: Retrieve-then-Plan Versus Plan-then-Retrieve

To navigate massive tool repositories, developers and researchers have traditionally relied on what the authors term the Retrieve-then-Plan (RtP) approach. In this dominant framework, the user’s raw natural language query is embedded into a vector space and matched against tool embeddings via approximate nearest-neighbor search. While straightforward and efficient, RtP assumes that a single pass of semantic matching can capture all the nuanced, multi-faceted requirements of a complex prompt without knowing the execution steps beforehand.

The new study investigates an intriguing alternative known as the Plan-then-Retrieve (PtR) paradigm. Under this inverted workflow, the LLM first generates what the researchers call Abstract Tool Stubs—hypothesized function signatures that describe the specific capabilities needed to solve parts of the problem. These generated stubs are then utilized as retrieval queries to search the registry, bridging the gap between abstract intent and concrete API endpoints.

To rigorously evaluate these competing strategies, the research team conducted large-scale experiments using the ToolBench benchmark. This dataset comprises 46,984 unique API endpoints spanning 50 distinct categories. The evaluation was carefully segmented across three complexity tiers: single-tool queries, labeled as G1; intra-category multi-tool queries, labeled as G2; and cross-category multi-tool queries, labeled as G3.

A Complexity-Dependent Crossover

The empirical findings reveal a clear and consistent complexity-dependent crossover between the two paradigms. According to the study, Retrieve-then-Plan outperforms Plan-then-Retrieve on single-tool queries where the intent is concentrated and straightforward. Conversely, Plan-then-Retrieve takes the lead, outperforming RtP on multi-tool queries that demand a broader scope of functional coordination.

Crucially, the research demonstrates that this performance crossover is exceptionally robust. The behavioral shift holds steady across every tested retrieval budget—ranging from a modest budget of $k=1$ up to $k=20$ retrieved items. Furthermore, the crossover remains consistent across registry sizes scaling dramatically from 1,000 APIs to nearly 47,000 APIs, and holds true across distinct model families, including Gemini 2.5 Flash and gpt-oss-120b.

The Double-Edged Sword of Stub Hallucination

A key contribution of the paper is its deep characterization of a specific failure mode inherent to the Plan-then-Retrieve approach: stub hallucination. When utilizing PtR, the LLM occasionally invents plausible-sounding but entirely non-existent API names and function structures during the initial planning phase.

The authors discovered that this phenomenon acts as a double-edged sword within the agent pipeline. On simple queries, the LLM’s generated stubs inadvertently abstract away crucial proper-noun signals and specific keywords that the traditional Retrieve-then-Plan method can match directly against the registry. This abstraction introduces unnecessary friction and lowers accuracy on less complex tasks.

However, on complex multi-faceted queries, this exact same hallucination and abstraction mechanism becomes a powerful asset. By breaking down a convoluted user request into structured functional sub-needs, the LLM effectively decomposes the problem, allowing the retrieval phase to source more relevant and diverse tool sets than a single direct query ever could.

Token Overhead and Adaptive Routing

Beyond accuracy and failure modes, the study provides a detailed analysis of the computational and token overhead associated with each paradigm. The researchers show that Plan-then-Retrieve requires an extra discovery call that roughly doubles the overall inference cost. This additional expenditure pays off handsomely on multi-tool queries where comprehensive discovery is critical, but acts as a heavy penalty on simpler single-tool tasks where efficiency is paramount.

This clear economic and performance trade-off naturally motivates a practical architectural direction: complexity-adaptive routing. By dynamically routing queries based on their inherent complexity, systems could leverage RtP for simple tasks and PtR for complex ones.

Significantly, the authors point out that such an adaptive routing mechanism can be driven by a zero-cost signal already natively present in the standard execution pipeline—specifically, the top-1 retrieval score generated during the initial RtP phase. This insight offers a pathway toward building more efficient, context-aware agent systems that optimize both performance and inference cost without introducing auxiliary classification models.

Related Readings and Context

The broader academic and engineering community continues to explore advanced retrieval strategies to enhance LLM agency. Large language models are increasingly relied upon for intricate multi-step planning tasks, cementing tool retrieval as a foundational pillar for successful task completion. While single-step retrieval methods utilize the complete query in one go, alternative sequential strategies segment full queries into discrete atomic subtasks through task decomposition.

Research in this domain continues to evolve, as highlighted by prior work presented at venues such as the UncertaiNLP workshop at EACL. While Retrieval Augmented Generation successfully bridges external knowledge gaps, traditional retrieval steps heavily depend on the prerequisite that all necessary contextual information is explicitly present within the user’s initial prompt, driving ongoing research into dynamic context tuning and progressive tool retrieval frameworks.