Skip to content

Commit 619da3b

Browse files
improve documentation for adaptive circuit breaker
1 parent 9688443 commit 619da3b

1 file changed

Lines changed: 42 additions & 14 deletions

File tree

README.md

Lines changed: 42 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -616,21 +616,41 @@ monitors error rates and adjusts its behavior accordingly.
616616

617617
##### How It Works
618618

619-
The adaptive circuit breaker uses the error function:
619+
The adaptive circuit breaker has two components:
620+
621+
1. An ideal error rate estimator that determines when the service is starting to become unhealthy
622+
2. A PID controller that opens the circuit fully or partially based on how bad the situation is.
623+
624+
The ideal error rate estimator uses a "simple exponential smoother", which means it simply takes the average error rate
625+
that it observes as the ideal. With the following caveat:
626+
627+
1. It ignores any data that is too high from its calculations. For example, we know that 20% error rate is an anamolous
628+
observation so we ignore it.
629+
1. It starts with an educated guess about the ideal error rate,
630+
and then converges down quickly if it observes a lower error rate, and slowly if it observes a higher error rate.
631+
1. After 30 minutes, it becomes more confident of its guess, and thus converges even slower in either directions.
632+
633+
The PID controller uses the following equation to determine whether to open or close the circuit:
634+
620635
```
621636
P = (error_rate - ideal_error_rate) - (1 - (error_rate - ideal_error_rate)) * rejection_rate
622637
```
623638

624-
This formula ensures that:
625-
- Rejection rate increases when the service is unhealthy
626-
- Rejection rate decreases when the service recovers
627-
- The system finds an equilibrium that protects against cascading failures while allowing recovery
639+
Or, more simply, if you define `delta_error = error_rate - ideal_error_rate` then:
628640

629-
##### Adaptive Circuit Breaker Configuration
641+
```
642+
P = delta_error - (1 - delta_error) * rejection_rate
643+
```
644+
645+
In simple terms: This equation says: open more when the error rate is higher than the rejection rate,
646+
and less when the opposite. The multiplier of `(1 - delta_error)` is called the aggressiveness multiplier.
647+
It allows the circuit to open more aggressively depending on how bad the situation is.
630648

631-
To enable the adaptive circuit breaker, simply set:
649+
This P is fed into a typical PID equation, and is used to control the rejection rate of the circuit breaker.
650+
651+
##### Adaptive Circuit Breaker Configuration
632652

633-
- **adaptive_circuit_breaker**. Enable adaptive circuit breaker instead of traditional. Defaults to `false`.
653+
To enable the adaptive circuit breaker, simply set **adaptive_circuit_breaker** to true.
634654

635655
Example configuration:
636656
```ruby
@@ -641,15 +661,23 @@ Semian.register(
641661
)
642662
```
643663

644-
The adaptive circuit breaker uses carefully tuned internal parameters based on extensive testing:
645-
- PID controller gains optimized for stability and responsiveness
646-
- 10-second window for rate calculations
647-
- 1-hour history for ideal error rate calculation (p90)
648-
- 1-second interval for background health checks
649-
650664
**Note**: When `adaptive_circuit_breaker: true` is set, traditional circuit breaker
651665
parameters (`error_threshold`, `error_timeout`, etc.) are ignored.
652666

667+
668+
We **_highly_** recommend just setting that configuration and not any other.
669+
One of the main goals of the adaptive circuit breaker is that it "just works".
670+
Configuring it might be difficult and not provide much value. That said, here are the configurations you can set:
671+
* **kp:** The contribution of P in the PID equation. Increasing it means you react more quickly to the latest data. Defaults to 1.0
672+
* **ki**: The contribution of the integral in the PID equation. Increasing it means adding more "memory", which is useful to ignoring noise. Defaults to 0.2
673+
* **kd**: The contribution of the derivative in the PID equation. Its behaviour can be complex because of our complex P equation. Defaults to 0.0
674+
* **integral_upper_cap**: Maximum value of the integral, prevents integral windup. Default to 10.0
675+
* **integral_lower_cap**: Minimum value of the integral, prevents integral windup. Default to -10.0
676+
* **window_size**: How many seconds of observations to take into account. Note that this window is a sliding window of 1 second sliding interval. To control the sliding interval you should set the environment variable SEMIAN_ADAPTIVE_CIRCUIT_BREAKER_SLIDING_INTERVAL (shared among all adaptive circuit breakers). window_size default to 10 seconds
677+
* **dead_zone_ratio**: An error percentage above the ideal_error_rate to ignore. This helps remove noise. Defaults to 0.25
678+
* **initial_error_rate**: The guess to start with for the ideal error rate. Defaults to 0.05 (5%)
679+
* **ideal_error_rate_estimator_cap_value**: The value above which we ignore observations for the ideal error rate. Defaults to 0.1 (10%)
680+
653681
### Bulkheading
654682

655683
For some applications, circuit breakers are not enough. This is best illustrated

0 commit comments

Comments
 (0)