[HYBIM-490] Enhance error reporting for evaluation framework #129
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adityamehra
reviewed
Jan 22, 2026
util/opentelemetry-util-genai-evals/src/opentelemetry/util/genai/evals/manager.py
Outdated
Show resolved
Hide resolved
| def has_evaluators(self) -> bool: | ||
| return any(self._evaluators.values()) | ||
|
|
||
| def get_error_summary(self) -> dict[str, Any]: |
Contributor
There was a problem hiding this comment.
Can you please capture more details on the diagnostic purpose this API is being exposed for?
| "Evaluator processing failed", | ||
| extra={ | ||
| "error_type": "processing_error", | ||
| "component": "worker", |
Contributor
There was a problem hiding this comment.
It would be good to have the thread name in the component.
util/opentelemetry-util-genai-evals/src/opentelemetry/util/genai/evals/manager.py
Show resolved
Hide resolved
Contributor
|
@keith-decker Once the recent changes for queue and concurrency from main are merged the error handling needs to be updated accordingly. |
- Added support for tracking errors by worker name and distinguishing between async and sync errors in ErrorTracker. - Improved ErrorEvent to include worker name and async context. - Updated Manager to log detailed error information during queue full and processing failures. - Added unit tests for concurrent error scenarios and validation of error tracking functionality.
Contributor
Author
|
@adityamehra This PR has been updated to support concurrency and the queue. |
adityamehra
approved these changes
Jan 26, 2026
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enhance error reporting for evaluation framework
Description
This PR improves error handling and observability in the evaluation framework by introducing structured error tracking and enhanced logging throughout the evaluation pipeline.
Changes
New Components:
ErrorEventdataclass for structured error representation with comprehensive context (timestamp, type, severity, component, message, details, and recovery actions)ErrorTrackerclass for tracking and aggregating errors with rate limiting capabilities and summary statisticsEnhanced Manager Error Handling:
_enqueue_invocation()to capture queue errors with context_worker_loop()to track and log processing failures with structured data_publish_results()without losing evaluation dataNew Features:
get_error_summary()method to Manager for diagnostic access to tracked errorsErrorEventandErrorTrackerfrom public APIBenefits