Microsoft.Extensions.AI.OpenAI and rate limiting

[phatcher]

Is it possible to change the design so that we can use the Microsoft.Extensions.Http.Resilience extensions to wrap the underlying calls to the API.

This would make it easier to deal with things like rate limiting and provide a more general caching behaviour

[stephentoub]

Microsoft.Extensions.AI.OpenAI just wraps OpenAIClient from the OpenAI library. It's what's actually interacting with HTTP. And it already supports the ability to plug in your own HttpClient (which you could get from DI configured however you like with resilience handlers), e.g.

HttpClient http = services.GetRequiredService<HttpClient>();

OpenAIClient oaiClient = new(
    new ApiKeyCredential(Environment.GetEnvironmentVariable("AI:OpenAI:ApiKey")!),
    new OpenAIClientOptions
    {
        // disable OpenAI's built-in resiliency mechanism
        RetryPolicy = new ClientRetryPolicy(0),
        NetworkTimeout = Timeout.InfiniteTimeSpan,

        // use your own HttpClient with its resiliency mechanisms
        Transport = new HttpClientPipelineTransport(http),
    });

IChatClient chatClient = oaiClient.GetResponsesClient("gpt-4o-mini").AsIChatClient();

[stephentoub]

If you want to do your own rate limiting separate from HttpClient, you can also do it as part of an IChatClient decorator. See https://learn.microsoft.com/en-us/dotnet/ai/ichatclient#custom-ichatclient-middleware for an example using System.Threading.RateLimiting.

[phatcher]

Thanks, idea was to wrap it all in Polly policies since all our other HttpClients are being handled that way