[Bug]: fail_closed_budget_enforcement does not reject failed atomic budget reservations

[emerzon]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate

What happened?

With general_settings.fail_closed_budget_enforcement set to true, a configured budget still fails open when its atomic pre-call reservation cannot be written

The strict setting is consulted by get_current_spend and returns 503 when the current spend cannot be verified against Redis or the database

litellm/litellm/proxy/proxy_server.py

Lines 2094 to 2114 in 3f9b71c

	def _fail_closed_budget_enforcement() -> bool:
	return general_settings.get("fail_closed_budget_enforcement") is True
	def _raise_budget_unverifiable(counter_key: str) -> None:
	verbose_proxy_logger.warning(
	"fail_closed_budget_enforcement: rejecting request — spend for %s could "
	"not be verified against Redis or the database",
	counter_key,
	)
	raise HTTPException(
	status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
	detail={
	"error": (
	"Budget enforcement unavailable: current spend could not be "
	"verified against Redis or the database, and "
	"fail_closed_budget_enforcement is enabled, so the request was "
	"rejected to avoid exceeding the configured budget. Retry shortly."
	)
	},
	)

litellm/litellm/proxy/proxy_server.py

Lines 2184 to 2189 in 3f9b71c

	# Opt-in hard guarantee: when the spend backing this admit decision came
	# only from a per-pod cache (Redis and DB both unreadable), reject rather
	# than admit on an unverifiable budget. No-op unless the flag is set, so
	# default behavior is unchanged.
	if not verified and _fail_closed_budget_enforcement():
	_raise_budget_unverifiable(counter_key)

The atomic reservation runs later, after the read-time common checks. reserve_budget_for_request catches _CounterReservationUnavailable for each required counter, removes that counter from the reservation, and continues. If every counter fails, it returns None and the request proceeds without an atomic admission hold

litellm/litellm/proxy/auth/user_api_key_auth.py

Lines 2372 to 2386 in 3f9b71c

	await _reserve_budget_after_common_checks(
	user_api_key_auth_obj=user_api_key_auth_obj,
	request_data=request_data,
	route=route,
	llm_router=llm_router,
	team_object=team_object,
	user_object=user_object,
	end_user_id=end_user_id,
	end_user_object=end_user_object,
	prisma_client=prisma_client,
	user_api_key_cache=user_api_key_cache,
	proxy_logging_obj=proxy_logging_obj,
	skip_budget_checks=skip_budget_checks,
	general_settings=general_settings,
	)

litellm/litellm/proxy/spend_tracking/budget_reservation.py

Lines 177 to 223 in 3f9b71c

	try:
	for counter in counters:
	entry = _counter_to_reservation_entry(
	counter=counter,
	reserved_cost=reservation_cost,
	)
	applied_entries.append(entry)
	try:
	reserved_value = await _reserve_counter(
	counter=counter,
	reservation_cost=reservation_cost,
	)
	except _CounterReservationUnavailable as exc:
	if exc.touched_counter and not exc.counter_invalidated:
	await _release_applied_entries_best_effort(
	entries=[entry],
	default_reserved_cost=reservation_cost,
	)
	applied_entries.remove(entry)
	continue
	if reserved_value is not None:
	current_spend = reserved_value
	else:
	cached_spend = current_spend_by_counter_key.get(counter.counter_key)
	if cached_spend is None:
	cached_spend = await _get_current_counter_value(counter=counter)
	current_spend = cached_spend + reservation_cost
	if current_spend > counter.max_budget:
	reservation_cost = await _apply_over_budget_reservation_policy(
	counter=counter,
	valid_token=valid_token,
	entry=entry,
	applied_entries=applied_entries,
	reservation_cost=reservation_cost,
	current_spend=current_spend,
	)
	continue
	except Exception:
	await _release_applied_entries_best_effort(
	entries=applied_entries,
	default_reserved_cost=reservation_cost,
	)
	raise
	if not applied_entries:
	return None

A verified read does not replace the reservation. Concurrent requests can all verify the same under-budget value, then all lose their reservation writes during a Redis timeout or failover, and all reach the provider before any completed cost is recorded

The warning used when reservation is explicitly disabled describes this read-time-only mode as allowing concurrent overspend and calls reservation hard per-request budget enforcement

litellm/litellm/proxy/auth/user_api_key_auth.py

Lines 2413 to 2421 in 3f9b71c

	if general_settings.get("disable_budget_reservation") is True:
	verbose_proxy_logger.warning(
	"disable_budget_reservation is enabled: skipping optimistic budget "
	"reservation. Budget enforcement is read-time only — concurrent "
	"requests can each pass the spend check before their cost is recorded, "
	"so a configured budget may be briefly exceeded under high concurrency. "
	"Set disable_budget_reservation to False or remove it to restore "
	"hard per-request budget enforcement."
	)

The default availability-first behavior can remain unchanged if intentional. When fail_closed_budget_enforcement is true, a reservation infrastructure failure for any configured budget counter should reject the request with 503 instead of silently degrading to read-time-only enforcement

This is separate from #33323, which covers exception swallowing in the legacy max_budget_limiter hook. This issue is in the central atomic reservation path after common_checks

Steps to Reproduce

1. Configure a virtual key with max_budget and set general_settings.fail_closed_budget_enforcement to true
2. Let the read-time budget check return a verified value below max_budget
3. Make the reservation counter increment or initialization raise, for example by simulating a Redis timeout
4. Call reserve_budget_for_request with a positive estimated request cost
5. Observe that the function logs a warning and returns None instead of raising 503
6. Run several requests concurrently and observe that each can proceed without a shared reservation

The existing regression test already locks in the fail-open result by forcing the counter increment to raise and asserting reservation is None

litellm/tests/test_litellm/proxy/test_budget_reservation.py

Lines 1517 to 1558 in 3f9b71c

	async def test_should_skip_reservation_when_counter_increment_fails(
	spend_counter_state,
	monkeypatch,
	):
	counter_cache, key_cache = spend_counter_state
	proxy_logging_obj = ProxyLogging(user_api_key_cache=key_cache)
	valid_token = UserAPIKeyAuth(
	token="key-budget-reserve-unavailable",
	spend=0.0,
	max_budget=1.0,
	)
	async def fail_increment_cache(*args, **kwargs):
	raise RuntimeError("counter unavailable")
	monkeypatch.setattr(counter_cache, "async_increment_cache", fail_increment_cache)
	with (
	patch(
	"litellm.proxy.spend_tracking.budget_reservation.estimate_request_max_cost",
	return_value=0.5,
	),
	patch(
	"litellm.proxy.spend_tracking.budget_reservation.verbose_proxy_logger.warning"
	) as mock_warning,
	):
	reservation = await reserve_budget_for_request(
	request_body=_request_body(),
	route="/chat/completions",
	llm_router=None,
	valid_token=valid_token,
	team_object=None,
	user_object=None,
	prisma_client=None,
	user_api_key_cache=key_cache,
	proxy_logging_obj=proxy_logging_obj,
	)
	assert reservation is None
	assert mock_warning.call_count >= 1
	assert (
	counter_cache.in_memory_cache.get_cache(

A strict-mode regression should run the same failure with general_settings.fail_closed_budget_enforcement true and assert HTTP 503. A companion test can preserve the current default behavior when the setting is absent or false

Relevant log output

Skipping budget reservation for spend:key:<key> because spend counter reservation failed

The request continues with budget_reservation set to None

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.94.0, current litellm_internal_staging at 3f9b71c. The affected files are unchanged in fetched upstream main and litellm_internal_staging as of 2026-07-19

Twitter / LinkedIn details

No response

[zeweihan]

Thanks for the precise write-up — the line references and the regression test that locks in fail-open make this easy to reason about. Adding a regulated-domain angle on why this specific silent degradation is the most expensive class of budget bug, plus a small companion to your proposed strict-mode test.

1. fail_closed_budget_enforcement=true is a compliance-cert artifact, not just a runtime switch. The reason the flag exists separately from the default is that some operators must be able to answer "does budget enforcement survive an infrastructure failure?" with a deterministic yes. An auditor reads fail_closed_budget_enforcement=true in config and certifies enforcement-on-failure; the silent degradation to read-time-only means the cert is structurally false, and the only emitted signal is a warning log line most SIEM pipelines don't alert on. This is the same shape as the Flagged-vs-Blocked conflation in guardrails and the send()->True-on-transport-failure in tracer cores: a surface that claims one posture while the runtime holds another. For a flag whose entire purpose is to be the strict-mode guarantee, silently not being strict is worse than not having the flag — because the flag's presence suppresses the manual controls an operator would otherwise build.

2. The TOCTOU window between read-time check and reservation write is where all the enforcement actually lives — and a Redis blip disables it for the whole concurrent burst, not per-request. Your repro correctly identifies that N concurrent requests can all verify under-budget then all lose their reservation writes. The amplification worth naming explicitly: a 30-second Redis timeout does not degrade enforcement by 30s worth of traffic. It disables atomic admission for every request that clears the read-time check during that window, which at peak is the entire in-flight load. So the blast radius of one reservation-store degradation event is "all concurrent spend for the degradation duration," and the事后 reconstruction depends on noticing the simultaneity — most incident tooling surfaces per-provider cost spikes, not "enforcement was off for this window." Same class as cross-tenant reads via a shared poisoned pool: the failure is in a shared substrate, but it gets filed as N independent provider incidents.

3. The warning log is not an audit record. Skipping budget reservation for spend:key:... because spend counter reservation failed is searchable but unstructured — it doesn't carry request_id, the specific counter that failed, the estimated_cost that was admitted unreserved, or the reason (timeout vs init-failure vs counter-missing). For the question "show me every request where budget enforcement was bypassed" to have a deterministic answer, the bypass needs to emit a typed decision, not a log line. Concretely: {event: budget_enforcement_bypassed, request_id, key, counter, reason, estimated_cost, mode: "fail_closed_degraded_to_read_only", ts}. This is the same durable shape as the per-check-decision-line fix for #33871 and the audit-line-per-data-plane-op for cross-tenant reads — the bug class is "silent success / silent degradation," and the durable answer across all of them is "make the silent path emit a record it cannot hide." Worth doing in both modes: even when fail-open is the intended default, the bypass record should fire, otherwise operators can't tell "enforcement held" from "enforcement was silently off for 8 minutes."

4. Property test complementing your strict-mode regression. Your proposed test asserts 503 when fail_closed_budget_enforcement=true and the counter increment raises. The companion I'd lock in alongside it, to prevent a future "we wired the 503 but silently dropped the audit emission" regression:

For any request R: if fail_closed_budget_enforcement=true AND any counter C in required_counters raises during reservation → R receives 503 AND a budget_enforcement_bypassed decision record is emitted with {request_id=R.id, counter=C, reason, estimated_cost}.
For any request R: if fail_closed_budget_enforcement is false/absent AND any counter C raises → R proceeds AND a budget_enforcement_bypassed decision record is emitted (same shape, different mode).

This way the rejection and the observability are both contract-locked. The default fail-open behavior is defensible for availability — you don't want a Redis blip to take down all LLM traffic — but it must be a conscious, observable choice, not a silent code path, and especially not a silent code path that activates under the very flag that promises it won't.

Separate from #33323 (legacy hook exception swallowing) as you noted — this is the central atomic-reservation path, and the fix here is the one that makes fail_closed_budget_enforcement mean what the flag name says.