feat(plugins): enforce per-tool timeouts and enhanced circuit breaker (IBM#2569)

Implement strict per-tool timeout enforcement for all transports
(REST, SSE, StreamableHTTP, A2A) and enhance the CircuitBreakerPlugin
with half-open states, retry headers, and granular configuration.

Changes:
- Wrap all tool invocations in asyncio.wait_for with effective_timeout
- Add per-tool timeout_ms support (ms to seconds conversion)
- Add half-open state for circuit breaker recovery testing
- Add half_open_in_flight flag to prevent concurrent probe requests
- Add retry_after_seconds in violation response for rate limiting
- Add tool_timeout_total and circuit_breaker_open_total Prometheus metrics
- Add cb_timeout_failure context flag for timeout detection in plugins
- Add tool_overrides for per-tool circuit breaker configuration
- Handle both asyncio.TimeoutError and httpx.TimeoutException
- Log actual elapsed time instead of configured timeout

Fixes applied during review:
- Fix _is_error() to detect camelCase isError from model_dump(by_alias=True)
- Fix half-open probe guard: only check when st.half_open is True
- Add stale-probe timeout to prevent permanent wedge if plugin blocks
- Add timeout enforcement to A2A tool invocations
- Call tool_post_invoke on exceptions so circuit breaker tracks failures
- Add ToolTimeoutError subclass to distinguish timeouts from other errors
- Only skip post_invoke for ToolTimeoutError (not all ToolInvocationError)
- Set error_message and span attributes for ToolTimeoutError observability
- Update README to document isError camelCase support

Timeout precedence:
1. Per-tool timeout_ms (if set and non-zero)
2. Global TOOL_TIMEOUT setting (default: 60s)

Closes IBM#2078

Signed-off-by: Mihai Criveti <[email protected]>
Co-authored-by: Mihai Criveti <[email protected]>

Author: kevalmahajan

mcpgateway/services/metrics.py

@@ -44,12 +44,26 @@
 
 # Third-Party
 from fastapi import Response, status
-from prometheus_client import Gauge, REGISTRY
+from prometheus_client import Counter, Gauge, REGISTRY
 from prometheus_fastapi_instrumentator import Instrumentator
 
 # First-Party
 from mcpgateway.config import settings
 
+# Global Metrics
+# Exposed for import by services/plugins to increment counters
+tool_timeout_counter = Counter(
+    "tool_timeout_total",
+    "Total number of tool invocation timeouts",
+    ["tool_name"],
+)
+
+circuit_breaker_open_counter = Counter(
+    "circuit_breaker_open_total",
+    "Total number of times circuit breaker opened",
+    ["tool_name"],
+)
+
 
 def setup_metrics(app):
     """