Skip to content

update error judge#393

Merged
e06084 merged 1 commit intoMigoXLab:devfrom
dt-yy:dev
Apr 17, 2026
Merged

update error judge#393
e06084 merged 1 commit intoMigoXLab:devfrom
dt-yy:dev

Conversation

@dt-yy
Copy link
Copy Markdown
Contributor

@dt-yy dt-yy commented Apr 17, 2026

No description provided.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the LLM document parsing logic by introducing flags to track parsing success and the presence of errors, which are then used to determine the final status. Feedback suggests improving robustness by verifying that the 'errors' field is a list to avoid potential TypeErrors and refining the output labels and reasons for cases where parsing fails or no errors are detected.

Comment on lines 113 to +115
errors = result_data.get("errors", [])
parse_ok = True
errors_nonempty = len(errors) > 0
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The current implementation is not robust against unexpected JSON structures from the LLM. If the errors field is not a list (e.g., a string or null), len(errors) will raise a TypeError, and the subsequent loop will also fail. It's safer to verify that errors is indeed a list before proceeding.

Suggested change
errors = result_data.get("errors", [])
parse_ok = True
errors_nonempty = len(errors) > 0
errors = result_data.get("errors", [])
if isinstance(errors, list):
parse_ok = True
errors_nonempty = bool(errors)
else:
errors = []


result = EvalDetail(metric=cls.__name__)
result.status = False
result.status = (not parse_ok) or errors_nonempty
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The updated logic for result.status correctly identifies issues, but the subsequent code for constructing result.label has two problems:

  1. If no errors are found (status is False), the label becomes ["."] because types and names are empty. It should ideally be ["QUALITY_GOOD"].
  2. If parsing fails (parse_ok is False), the label also becomes ["."], which is not descriptive.

Returning early for the success case and setting a specific label for parsing errors would improve the output quality.

Suggested change
result.status = (not parse_ok) or errors_nonempty
result.status = (not parse_ok) or errors_nonempty
if not result.status:
result.label = ["QUALITY_GOOD"]
result.reason = [json_str] if 'json_str' in locals() else [response]
return result
if not parse_ok:
types, names = ["QUALITY_BAD"], ["ParseError"]

@e06084 e06084 merged commit 1dbb409 into MigoXLab:dev Apr 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants