Issue 282 calibration pipeline update - #292
Merged
Merged
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request moves the calibration training logic to a script and updates the EPR and WEPR factory functions to support training by allowing them to be initialized without a pretrained model path. However, importing from the top-level 'scripts' directory inside the library package will cause packaging issues and lead to a 'ModuleNotFoundError' when installed. Additionally, the 'train_calibration' function should return the fitted pipeline instead of discarding it, process the batch of inputs at once for efficiency, and include input validation checks.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
CharlotteLeBihan18
force-pushed
the
issue-282-Calibration-Pipeline-Update
branch
from
July 8, 2026 07:44
2d2b051 to
81ab329
Compare
chicham
self-requested a review
July 17, 2026 08:15
CharlotteLeBihan18
force-pushed
the
issue-240-Base-Detector
branch
from
July 17, 2026 08:39
3550472 to
149d21e
Compare
Merged
chicham
force-pushed
the
issue-282-Calibration-Pipeline-Update
branch
from
July 17, 2026 14:00
92aa86e to
63392ea
Compare
chicham
requested changes
Jul 17, 2026
chicham
left a comment
chicham
left a comment
Collaborator
There was a problem hiding this comment.
[NOTE] @chicham — after #291 lands and its branch is deleted, GitHub will auto-retarget this PR's base to main; at that moment retarget the branch with the targeted form only:
git fetch origin
git rebase --onto origin/main e77441ce issue-282-Calibration-Pipeline-Update
(e77441c = the issue-240-Base-Detector tip this branch now sits on.) A plain git rebase main would replay the base-detector work too and conflict.
Unplaced comments
- [SUGGESTION] src/artefactual/calibration/train_calibration.py: Chore: deleting this module orphans
scripts/calibration_llm/run_train_calibration.py, which imports it — that script now dies at its import line withModuleNotFoundError(and its CSV-in/JSON-out contract is obsolete now that serialization is deferred). Suggest deleting that script in this PR too; a new CLI can come back with the serialization work.
CharlotteLeBihan18
force-pushed
the
issue-282-Calibration-Pipeline-Update
branch
from
July 24, 2026 09:11
68c3bea to
f15296b
Compare
chicham
force-pushed
the
issue-240-Base-Detector
branch
from
July 27, 2026 09:26
141445d to
67159c3
Compare
chicham
marked this pull request as draft
July 28, 2026 09:19
CharlotteLeBihan18
marked this pull request as ready for review
July 30, 2026 16:03
chicham
force-pushed
the
issue-282-Calibration-Pipeline-Update
branch
4 times, most recently
from
July 31, 2026 13:39
dd27e9b to
dc139e0
Compare
train_calibration took a CSV of precomputed uncertainty scores and judgments and wrote a weights JSON. It now takes the raw completion responses and their binary labels, fits the full parser/entropy/classifier pipeline, and returns the fitted detector -- so a calibration is trained on the same pipeline that scores with it, rather than on scores computed by a separate path. The pipeline is assembled directly rather than through epr()/wepr(): those only ever hand back a calibrated detector, and an unfitted one must not escape the library. The orphaned run_train_calibration.py CLI goes away with the contract it wrapped.
Two notebooks walk through the epr and wepr detectors end to end -- loading a calibration, scoring a response, reading sequence and token level probabilities -- against JSON response fixtures committed alongside them. Paths and thresholds sit in a configuration cell so they can be pointed at other data. They replace epr_usage_demo.py and wepr_demo.ipynb, which are deleted along with their fixture. .gitignore lets examples/*.json through the repo-wide **/*.json rule, and ruff stops flagging print statements in example notebooks.
The trainer was a module in src/artefactual/calibration/ that nothing could run, and the copy under scripts/ was dead: it called epr()/wepr() with no weight path, which raises UncalibratedModelError since #307. No test covered scripts/, so CI never caught it. Training a calibration is an offline, one-off operation, so it lives in scripts/ now -- the same place the skops weight migration (#256) will live. The package copy and its re-export go away, along with its tests: the script is a thin caller, and the pipeline, parser and entropy reduction it relies on are covered in tests/scoring/ and tests/preprocessing/. It also gains the command line it never had: python scripts/train_calibration.py --responses r.json --labels y.json --reduction epr Two guards are dropped. 'Length of X' called len() on whatever the caller passed; LogProbParser accepts a single response payload carrying several generations, and len() on that dict counts its keys, so one real response reported 'Length of X (33) must match length of y (2)'. The class-count guard went with it. The pipeline checks both after parsing, against the real number of sequences, and reports them accurately. The reduction guard stays, because EntropyTransformer only reports a bad reduction once the whole batch is parsed, but now accepts the callables it always supported. scripts/calibration_llm/calibration_script.py is deleted: it already failed at its first import on main, and this branch removes the module its step 3 needed. README documents the new flow and points the demo links at the notebooks that replaced the ones deleted here.
chicham
force-pushed
the
issue-282-Calibration-Pipeline-Update
branch
from
July 31, 2026 13:41
dc139e0 to
11b2e73
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #282