Google’s Gemini API now loads tool definitions on demand, cutting token waste and latency
THE SHIFT
Imagine an AI assistant that can call on hundreds of functions without padding every request with massive parameter data. Today many developers build multi‑tool agents, but each tool’s full schema travels with every prompt, inflating token usage, adding latency, and raising costs. When a conversation spans many turns, the cumulative overhead can become a bottleneck for real‑time applications.
Google’s Gemini Interactions API now offers a preview feature called Tool Retrieval. Instead of sending every tool definition up front, developers can flag less‑used tools for deferred loading. The model still sees each tool’s name and description, but the heavy parameter schema is fetched only when the model explicitly requests it, keeping the prompt lean.
Source: Google AI Blog: Introducing Tool Retrieval in the Gemini API ↗
THE PROOF
Google explains that loading parameter schemas only when needed cuts input token counts and reduces latency, while also sharpening tool‑selection accuracy as the candidate list grows. In internal tests, the company reports measurable token savings for large tool collections, confirming the efficiency claim. These reductions translate into lower API costs and faster response times for developers deploying complex agents.
The feature works by adding a tool_search tool to the request and setting defer_loading:true on selected functions. In a sample interaction, the model asks for a specific tool by name, the API streams the full schema, and the model immediately emits the function call—all within a single interaction step, making debugging straightforward. Developers can inspect the steps array to see which tools were considered, simplifying troubleshooting of tool selection logic.
Source: Google AI Blog: Introducing Tool Retrieval in the Gemini API ↗
THE CATCH
Tool Retrieval is currently supported only on Gemini 3.6 and later models, including all Gemini 4 variants. Developers relying on earlier Gemini releases must continue to embed full tool schemas in each request, so the benefit is unavailable for legacy applications. This restriction means teams planning migrations need to assess whether their target model version aligns with the new feature before refactoring code.
Enabling tool_search adds a modest fixed token overhead, and each loaded schema is billed as regular input tokens, so overall pricing remains unchanged. If developers opt for client‑side retrieval, an extra round‑trip is introduced for the lookup, which can add a slight delay in highly time‑sensitive flows. However, for most applications the added latency is outweighed by the token savings when the tool library contains dozens of rarely used functions.
YOUR MOVE
To experiment with Tool Retrieval, add a tool_search entry to your Interactions API request and mark long‑tail functions with defer_loading:true. Start with a Gemini 4‑Hydrogen or newer model, run a few test conversations, and compare token usage against a baseline that loads all schemas up front. Track the total input tokens reported by the API to see the concrete savings in your own workload.
Keep your most frequently used tools loaded continuously and defer only the long‑tail ones to maximize savings. Monitor the token counts in your logs to verify the reduction, and adjust the defer list as usage patterns evolve. For teams that need on‑premise control, the client‑side execution mode is available, remembering that it adds one extra lookup round‑trip per deferred tool.
Tool Retrieval gives developers a practical way to trim token waste and latency when scaling agent toolsets, though it requires Gemini 3.6+ and careful monitoring of added lookup overhead.
Sources & further reading
Published by The Breakdown. How we report · Our source desk.