feat(language_model): add strict_mode option for structured output#50
Open
lelouar wants to merge 1 commit intoSynaLinks:mainfrom
Open
feat(language_model): add strict_mode option for structured output#50lelouar wants to merge 1 commit intoSynaLinks:mainfrom
lelouar wants to merge 1 commit intoSynaLinks:mainfrom
Conversation
Some OpenAI-compatible servers (e.g. TensorRT-LLM's `trtllm-serve`)
reject unknown fields in `response_format` via Pydantic's
`extra_forbidden` validator. When synalinks sends the hardcoded
`"strict": true` field, these servers return:
BadRequestError: [{'type': 'extra_forbidden',
'loc': ('body', 'response_format', 'strict'), ...}]
Add a `strict_mode` constructor argument (default True, preserves
current behavior) that, when False, omits the `strict` field entirely
from the payload sent to the provider. Covers all 5 provider branches
that currently set strict: ollama/mistral, openai/azure, gemini, xai,
hosted_vllm.
Usage:
# Default: strict enforced (unchanged)
lm = LanguageModel(model="openai/gpt-4o-mini")
# For servers that don't accept the `strict` field
lm = LanguageModel(
model="hosted_vllm/my-model",
api_base="http://trtllm-server/v1",
strict_mode=False,
)
Added 5 unit tests covering default=True, strict_mode=False for
hosted_vllm/openai/ollama, and get_config/from_config roundtrip.
Contributor
|
Are you sure that non strict mode is not the json mode ? which doesn't guarantee the format ? |
Author
|
It's seems to depend on model + LLM server + parameters. I can dig more into this to draw a clearer view. I only tried vLLM and trtllm so far but I can do more test next week |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-out for the hardcoded
"strict": truefield thatLanguageModelcurrently injects intoresponse_formatfor structured-output calls.Problem
Some OpenAI-compatible servers reject unknown fields in
response_formatvia Pydantic'sextra_forbiddenvalidator. Notably, NVIDIA's TensorRT-LLM (trtllm-serve) returns:litellm.drop_params = Truedoes not strip this field, so the request fails regardless of thestrictvalue (True or False — the key itself is forbidden).Today, users hitting such a server have no way to use
LanguageModelwith structured output.Change
Add a
strict_modeconstructor argument toLanguageModel:True— preserves current behavior, no breaking change.False— omits thestrictfield entirely from theresponse_formatpayload.Also wired through
get_config()/from_config()for serialization roundtrip.Scope
strictfield. No rename/alias for other servers that might use a different parameter name — that can be a separate change if needed.strictare covered uniformly.LanguageModeloptions.Tests
Added 5 unit tests in
synalinks/src/language_models/language_model_test.py:test_strict_mode_default_true_hosted_vllm— verifies default still emitsstrict: truetest_strict_mode_false_omits_strict_hosted_vllmtest_strict_mode_false_omits_strict_openai(covers the inside-json_schemavariant)test_strict_mode_false_omits_strict_ollamatest_strict_mode_roundtrip_via_get_configAll 10 tests in the file pass (5 existing + 5 new).
Test plan
uv run pytest synalinks/src/language_models/language_model_test.py -vpasses (10/10)strict_mode=Falseunblocks structured output calls🤖 Generated with Claude Code