Bug: CircuitBreaker in HalfOpen state allows unlimited concurrent requests, risking backend overload #1612

Describe the bug

While reviewing the error recovery and resilience logic, I noticed a concurrency bug in the CircuitBreaker implementation. When the circuit transitions from the Open state to the HalfOpen state after the recovery timeout, it currently allows all incoming requests to pass through simultaneously, without capping the number of concurrent probes.

Because CircuitBreaker::call() simply checks whether the state is HalfOpen and immediately allows the operation, a sudden burst of traffic will pass right through the breaker while it is still waiting for the first few operations to succeed or fail. This defeats the protective purpose of the HalfOpen state, as we end up hammering the recovering backend with full traffic before the circuit can decide whether to close or re-open.

To Reproduce

1. Configure a CircuitBreaker with a success_threshold of 2.
2. Force the circuit into an Open state by triggering consecutive failures.
3. Wait for the recovery_timeout to elapse so the next request transitions the state to HalfOpen.
4. Fire 100 concurrent asynchronous requests through the CircuitBreaker.
5. Notice that all 100 requests are allowed to hit the backend simultaneously because none of them have completed yet to update the state.

Expected behavior

During the HalfOpen state, the circuit breaker should act as a valve restricting traffic. It should only allow a limited number of concurrent probe requests (typically up to the success_threshold) to test if the underlying service is healthy. Any excess requests arriving while the probes are still "in-flight" should be fast-failed/rejected with an error (similar to how they are rejected in the Open state) until the probes either succeed and close the circuit, or fail and reopen it.

Code snippet causing the issue

In crates/mofa-foundation/src/recovery.rs:

 CircuitState::Closed |
 CircuitState::HalfOpen => {
     // Allow the call
 }

Suggested Fix

We should introduce an active_probes counter in CircuitBreakerState to track in-flight requests while in HalfOpen.

• When a request attempts to pass through during HalfOpen, we check if active_probes < success_threshold.
• If true, increment active_probes and allow the call.
• If false, reject the call immediately to prevent overwhelming the recovering service.
• Decrement the active_probes counter once the operation completes.