New Study Reveals Complexity-Dependent Crossover in Tool-Augmented LLM Agent Retrieval Paradigms

As Large Language Models (LLMs) increasingly evolve into autonomous agents capable of executing complex workflows, a major bottleneck continues to challenge developers: how to connect these models with the vast ecosystems of external application programming interfaces (APIs) required to interact with the digital world. Tool-augmented LLM agents must discover relevant APIs from massive registries before they can plan and execute tasks. However, finite context windows strictly prohibit injecting thousands of tool definitions into a single prompt.

To address this challenge, a team of researchers—comprising Mingfei Zhang, Amit Surana, Ziauddin Majid, Chenxi Wang, and Michael Xu—has released a comprehensive study examining two competing paradigms for tool retrieval. Their findings uncover a striking, complexity-dependent performance crossover that could fundamentally shape how next-generation AI systems are architected.

Understanding the Retrieval Divide: Retrieve-then-Plan versus Plan-then-Retrieve

The dominant paradigm in current tool-augmented LLM systems is known as Retrieve-then-Plan, or RtP. Under this conventional approach, the system embeds the raw user query and retrieves similar tools directly from a registry via approximate nearest-neighbor search. While efficient, this method relies entirely on the initial text of the prompt matching the descriptive metadata of the correct API endpoints.

Seeking an alternative, the researchers studied a paradigm designated as Plan-then-Retrieve, or PtR. In this workflow, the LLM first analyzes the user request and generates what the authors call Abstract Tool Stubs. These are essentially hypothesized function signatures that describe the capabilities needed to solve the problem. These generated stubs are then utilized as retrieval queries against the API registry, rather than relying on the raw user text.

To rigorously evaluate these two competing strategies, the research team tested them on ToolBench, a massive benchmark comprising 46,984 unique API endpoints spanning 50 distinct categories. The evaluation was broken down into three specific query subsets to test varying degrees of task complexity: single-tool queries, classified as G1; intra-category multi-tool queries, classified as G2; and cross-category multi-tool queries, classified as G3.

A Clear Complexity-Dependent Crossover

The results of the ToolBench evaluation revealed a sharp and consistent crossover in performance between the two methods, determined entirely by the complexity of the user’s request.

According to the study, Retrieve-then-Plan consistently outperforms Plan-then-Retrieve when handling single-tool queries. When a user asks an AI agent to perform a straightforward, isolated task that requires only one API, going straight to the vector database with the user’s query yields superior results.

Conversely, the study found that Plan-then-Retrieve outperforms Retrieve-then-Plan when dealing with multi-tool queries. For complex tasks requiring the orchestration of multiple APIs—whether within the same functional category or spanning across entirely different domains—generating abstract tool stubs first proves to be a vastly superior discovery mechanism.

Crucially, the research demonstrated that this complexity-dependent crossover is remarkably robust. The pattern holds true across every tested retrieval budget, ranging from a top-k value of 1 up to 20. Furthermore, the crossover remains consistent across varying registry scales, holding firm from compact registries containing just 1,000 APIs up to the full-scale benchmark of nearly 47,000 endpoints. The phenomenon was also verified across different foundational model families, specifically testing with Gemini 2.5 Flash and gpt-oss-120b, indicating that the behavior is a fundamental characteristic of the retrieval paradigms rather than an artifact of a specific LLM architecture.

The Double-Edged Sword of Stub Hallucination

To understand why this performance split occurs, the researchers investigated the internal mechanics of the PtR paradigm, identifying a specific failure mode they term "stub hallucination."

In this scenario, the LLM invents plausible-sounding but non-existent API names when generating its abstract function signatures. Interestingly, the study characterizes this phenomenon as a double-edged sword. On simple queries, stub hallucination actively harms performance. The LLM’s generated stubs abstract away specific proper-noun signals and exact keywords that the simpler RtP approach can match directly against the registry using standard nearest-neighbor search.

However, on complex queries, that exact same hallucination and abstraction mechanism acts as a powerful asset. For multi-faceted intents, the LLM breaks down a complicated, convoluted user request into distinct functional sub-needs. Even if the exact names of the functions are initially fabricated or imprecise, the structural abstraction decomposes the problem in a way that successfully captures the fragmented requirements of the task, guiding the subsequent retrieval step to uncover the correct tools.

Token Overhead and the Case for Complexity-Adaptive Routing

Beyond accuracy and retrieval performance, the researchers also analyzed the operational costs associated with token overhead. Their analysis revealed that the Plan-then-Retrieve approach requires an extra discovery call, which roughly doubles the overall inference cost of the system.

This added computational expense is a worthwhile trade-off on multi-tool queries, where the superior discovery performance justifies the cost. However, the extra call becomes an unnecessary penalty when applied to simple, single-tool queries.

This economic and performance divide highlights a promising practical direction for system developers: complexity-adaptive routing. Rather than forcing all incoming user requests through a single, static retrieval pipeline, an efficient agent architecture could dynamically route queries to either RtP or PtR based on the estimated complexity of the task.

Remarkably, the authors note that this routing decision does not require an expensive auxiliary classifier. Instead, it can be driven by a zero-cost signal already natively present in the standard pipeline—specifically, the top-1 retrieval score generated during the initial stages of an RtP approach. By evaluating this confidence signal, systems can intelligently decide whether to execute a fast, direct retrieval or invest in a multi-step planning and stub-generation phase, optimizing both performance and operational cost for AI-driven workflows.