[Feature]: prompt caching - Explicit and Implicit

[qdrddr]

Use Case

An agent that uses a memory. Both explicit and Implicit caching requires prompt to be at the beginning of the request.

Problem Statement

1. LLM use is expensive. I want to benefit from Explicit and Implicit caching available with various providers supporting this feature.
2. Explicit caching is a two-edged sword: it can provide the benefit of significant savings, but it is not free. Explicit caching can save money, but it may end up consuming more money than it saves.

How This Feature Would Help

Reduces LLM costs

Proposed Solution

Move repetitive instruction text to the top of the prompt over the API requests.
Keep track of the statistics for repetitive text to evaluate whether the text was eligible for caching and allow the user to evaluate whether it was actually cached.

Explicit Caching

Providers such as Google and Anthropic support explicit caching.
When used with Anthropic SDK, Gemini SDK, and LiteLLM SDK (with Anthropic/Gemini provider or OpenRouter Provider with the Anthropic/Gemini models), I'd like to be able to enable explicit caching.

TTL

Via CLI/UI/Envs, I'd like to be able to:

1. Manipulate the cache TTL
2. Monitor cache spending vs. savings with a given TTL in production, so I could try adjusting TTL and later see how it affects my savings with the previous TTL.

Alternatives Considered

No response

Priority

Nice to have

Additional Context

The goal is to

1. Ensure Prompt is cachable and can benefit the most from cache
2. Provide reasonable defaults for the Explicit cache
3. Allow the user to find the best TTL settings to optimize caching benefit

Checklist

• I would be willing to contribute this feature

[qdrddr]

This is in addition to Batch API Processing.
#365

Batch API processing can benefit from cache, further decreasing the LLM costs.

[nicoloboschi]

our prompts are already built towards implicit caching and you should be able to see them through the metrics usage.

the explicit caching is quite hard and tricky to manage and the benefits can be achieved with proper implicit prompt caching strategies

since we're adding Batch API, I'd say to not add more levels of complexity for now unless we have a clear design problem with the current implicit caching implementation