Tool search and deferred loading

[kzu]

What are the thoughts and potential recommendations on how to implement this feature? It's quite conceivable that it will be ubiquitous, now that both Claude and GPT models implement it: https://developers.openai.com/api/docs/guides/tools-tool-search

[stephentoub]

What do you think about #7377 ?

[Code-by-AB]

@stephentoub
Won't it be better to use a wrapping function instead of deferredTools/NonDeferredTools?

tools = [new HostedToolSearchTool(), new DeferableAIFunction(get_weather)]

or an extension AIFunctionFactory.Create(..).AsDeferedTool()?

Guess it depends on how non open-ai providers plan to implement this tool search feature.

Would this HostedToolSearchTool also support client execution + args?

something like HostedToolSearchTool.Create(MyToolSearch.Search)

given i have

 public class MyToolSearch
 {

     public record SearchRequest(
     [Description("The category of the tool to search for provided is system prompt")] string ToolCategory,
        [Description("The query to search for")] string Query
     );

     [Description("Finds the tools needed based on category and semantic query")]
     public HostedToolSearchToolResponse Search(SearchRequest request)
     {
         //do whatever to search for tools based on the request.ToolCategory and request.Query
     }

 }

[stephentoub]

Won't it be better to use a wrapping function instead of deferredTools/NonDeferredTools?

That's where I started and then I switched to this. My theory is that either no tools will be deferred (in which case just don't use HostedToolSearchTool), all will be deferred (in a which case just adding the raw tool achieves that), only a small number should be deferred (in which case add those To Deferred) or most will be deferred (in which case add the others to NonDeferred). It's possible to wrap each tool, but for most of these cases it ends up being more work / less ergonomic.

So to your question... Why? Is there a scenario that you his doesn't handle? Am I missing an aspect of the ergonomics that makes that better for a developer?

[Code-by-AB]

My theory is that either no tools will be deferred (in which case just don't use HostedToolSearchTool), all will be deferred (in a which case just adding the raw tool achieves that),

Makes sense. Majority will always want some tools allowed with everything else deferred.

Am I missing an aspect of the ergonomics that makes that better for a developer?

Not really. I was just thinking of our case where we have a bunch of ToolRegistry .cs classes and all logic of if user can see this tool or if the tool can be executed + AIFunction creation of the tool lives in there.

Something like:

 public sealed class MyCategoryToolRegistry : IToolProvider
{
    public MyCategoryToolRegistry(tools dependency injection)
    {
        _factory = new Dictionary<string, Func<AITool>>(StringComparer.Ordinal)
        {
            [ToolNames.GetItems] = () => AIFunctionFactory.Create(_toolServ.GetItemsAsync, ToolNames.GetItems),
            [ToolNames.SearchItems] = () => AIFunctionFactory.Create(_svc.SearchItems, ToolNames.SearchItems),
            [ToolNames.AddItems] = () => AIFunctionFactory.Create(_svc.AddItems, ToolNames.AddItems),
            [ToolNames.GetUserPageDetails] = () => AIFunctionFactory.CreateDeclaration(ToolNames.GetUserPageDetails,
                Tools.GetUserPageDetails.Description,
                default),

           ....bunch of other tools + hosted tools
        };
    }

  public List<AITool> GetAllTools()
  {
      return [.. _factory.Values.Select(x => x())];
  }
      public List<AITool> GetByNames(params string[] toolNames)
  {
       //more logic/access check
          return [.. _factory.Values.Select(..)]; 

  }

and we pull tools as need depending of the agent + some similarity check, across all categories.
May be deferred logic would best suit if it stays with the agent than within a tool Registry

[Code-by-AB]

@stephentoub
Thinking this thru, wrt both OpenAI and Anthropic docs, where defer_loading is conceptually tied to the individual tool declaration and same for Anthropic's code_execution: true

The current DeferredTools / NonDeferredTools API makes sense as a convenience layer, but long term I wonder whether this configuration should stay closer to the tool itself rather than HostedToolSearchTool becoming the primary place where that behavior is expressed.

Provider authors mapping this abstraction into their own SDKs, may reasonably expect these settings to stay closer to the tool declaration and would also make it easier to support custom search / execution flows (for example via FunctionInvokingChatClient)

Not opposed to current approach for this specific use case, but if providers continue adding tool-level capabilities, it may be worth preserving a path where provider-specific properties stay attached to the tool itself, while Hosted Tools remain the orchestration mechanism and optional convenience layer.