fix: correct off-by-one and tie handling in od p-value computation - #959
Open
shivamlalakiya wants to merge 1 commit into
Open
fix: correct off-by-one and tie handling in od p-value computation#959shivamlalakiya wants to merge 1 commit into
shivamlalakiya wants to merge 1 commit into
Conversation
od/pytorch/base.py and od/sklearn/base.py compute the outlier p-value as
(1 + (scores[:, None] < self.val_scores).sum(-1)) / len(self.val_scores)
The sibling computation in od/pytorch/ensemble.py:114 does the same count
but divides by len(self.val_scores) + 1, which is what keeps that value
a valid p-value in [0, 1]. With the base.py denominator, a test score
below every validation score returns (n+1)/n, e.g. 1.25 at n=4, in the
documented p_value output field. The same expression's strict '<' also
collapses to 1/n instead of 1 when the test score ties every validation
score.
This changes both files to '<=' over (len(self.val_scores) + 1),
matching ensemble.py's denominator and closing the tie gap. No test in
od/pytorch/tests or od/sklearn/tests exercises this expression, so
nothing else needs to change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #958
What
od/pytorch/base.py:199andod/sklearn/base.py:148compute the outlierp-value as
od/pytorch/ensemble.py:114, same author, same subpackage, computes thesame "add one to the count the test score beats" idea but divides by
len(self.val_scores) + 1:The
base.pydenominator can push the value above 1. Reproduced throughthe public API (0.13.0 wheel, Python 3.11, numpy 1.26.4):
The same line's strict
<separately collapses tied scores toward1/ninstead of
1. Full numbers, and the exact-rational tie-support sweep,are in #958.
Fix
One line changed per file:
<becomes<=, and the denominator becomeslen(self.val_scores) + 1, matchingensemble.py.Verified against the patched files:
I searched
od/pytorch/testsandod/sklearn/testsfor coverage of thisexpression and found none, so this is a pure two-line change with nothing
else to update.
ensemble.py:114is untouched; its denominator is alreadycorrect and its strict comparator is a separate, package-wide question
I've left out of this PR (noted in #958).