Skip to content

Guard nvcf 500 error parsing against non-JSON bodies#1945

Open
chuenchen309 wants to merge 1 commit into
NVIDIA:mainfrom
chuenchen309:fix/nvcf-500-nonjson-guard
Open

Guard nvcf 500 error parsing against non-JSON bodies#1945
chuenchen309 wants to merge 1 commit into
NVIDIA:mainfrom
chuenchen309:fix/nvcf-500-nonjson-guard

Conversation

@chuenchen309

@chuenchen309 chuenchen309 commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

On an HTTP 500, NvcfChat._call_model parsed the error body unconditionally:

if response.status_code == 500 and json.loads(response.content)["detail"].startswith("Input value error"):

A 500 body is not guaranteed to be JSON — a gateway/proxy/load balancer in front of the NVCF function commonly returns an HTML or plain-text error page — and even when JSON it may not carry a string detail. In those cases this line raises an uncaught json.JSONDecodeError (a ValueError), KeyError, or AttributeError.

None of those is in the @backoff.on_exception tuple (AttributeError is, but JSONDecodeError/KeyError are not, and the string .startswith on a non-str detail is the AttributeError path), so on a long scan a single such 500 propagates out of _call_model and aborts the whole run instead of being retried or skipped — the top-level handler just logs and exits.

Fix

Extract the check into a defensive _is_input_value_error(response) helper. Anything that can't be confirmed as an input-value error (non-JSON body, missing/non-string detail, or a different message) returns False and falls through to the existing raise_for_status() path — i.e. it's treated as a genuine server error and retried by backoff, exactly as any other unrecognized 5xx already is. Behaviour for a well-formed "Input value error" 500 (skip → [None]) is unchanged.

Verification

  • Added test_nvcf_is_input_value_error_parses_defensively, parametrized over: a well-formed input-value error (→ True), a different message, a non-JSON body, an empty body, JSON without detail, and a non-string detail (all → False). Each of the last four would previously have raised inside the parse. Full tests/generators/test_nvcf.py passes; black --check clean.
  • garak -t <target_type> -n <model_name>not run: this is a unit-level fix and I have no live target configured; verified via the tests above rather than ticking a box I didn't exercise.
  • Supporting configuration such as a generator configuration file — n/a, no config surface changed.

Disclosure: this contribution was prepared with the assistance of an AI agent (Claude Code). The analysis, fix, and tests were reviewed by me before submission.

On an HTTP 500, NvcfChat._call_model ran
json.loads(response.content)["detail"].startswith("Input value error")
unconditionally. A 500 body is not guaranteed to be JSON (a gateway/proxy in
front of the NVCF function can return an HTML/text error page) or to contain a
string 'detail', so this raised an uncaught JSONDecodeError / KeyError /
AttributeError. None of those is in the backoff on_exception tuple, so it
propagated out of _call_model and aborted the whole scan instead of being
retried or skipped.

Extract the check into a defensive _is_input_value_error helper: anything that
can't be confirmed as an input-value error returns False and falls through to
the existing raise_for_status path (a genuine server error, retried by
backoff) rather than crashing. Behavior for a well-formed
'Input value error' 500 (skip -> [None]) is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Andrew Chen <48723787+chuenchen309@users.noreply.github.com>
@jmartin-tech jmartin-tech self-assigned this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants