Skip to content

Add PrecisionAtFixedRecall#69

Closed
ablaom wants to merge 51 commits intodevfrom
precision-given-recall
Closed

Add PrecisionAtFixedRecall#69
ablaom wants to merge 51 commits intodevfrom
precision-given-recall

Conversation

@ablaom
Copy link
Copy Markdown
Member

@ablaom ablaom commented Feb 10, 2026

Partly addresses #68.

This PR is ready for a review after:

on which it is based.

This PR adds a core function Function.precision_at_fixed_recall and, based on that, a new measure constructor PrecisionAtFixedRecall.

Note to self: Rebased off b56bbbf

docstring
  PrecisionAtFixedRecall(; recall_threshold=0.95)

  Return a callable measure for computing the precision at fixed recall. Aliases:
  precision_at_fixed_recall.

  m(ŷ, y)

  Evaluate some measure m returned by the PrecisionAtFixedRecall constructor (e.g., m =
  PrecisionAtFixedRecall()) on predictions ŷ, given ground truth observations y. It is
  expected that ŷ be a vector of distributions over the binary set of unique elements of
  y; specifically, ŷ should have eltype <:UnivariateFinite from the
  CategoricalDistributions.jl package.

  This metric is useful, in applications such as toxicity detection, anomaly detection,
  and screening for disease markers, if one wants a cap on the proportion of positives
  that are misclassified (one minus the recall) while minimizing the rate of false alarms
  (one minus the precision).

  More precisely, the measure:

    1. Determines all values of the recall, as one varies the probability threshold
       for a positive outcome over all predicted probabilities for that class.

    2. Among these recalls, finds the smallest one that exceeds or equals
       recall_threshold.

    3. Returns the corresponding precision for that recall.

  In the event there are multiple precisions for the same recall, the mean precision is
  returned. In the event no recall is found in Step 2, a precision of 0 is returned.

  Core implementation: Functions.precision_at_fixed_recall.

  Generally, an observation obs in MLUtils.eachobs(y) is expected to satisfy
  ScientificTypes.scitype(obs)<:ScientificTypesBase.OrderedFactor{2}.

  See also precision_recall_curve.

  For a complete dictionary of available measures, keyed on constructor, run measures().

  Traits
  ≡≡≡≡≡≡

  consumes_multiple_observations = true
  can_report_unaggregated = false
  kind_of_proxy = LearnAPI.Distribution()
  observation_scitype = ScientificTypesBase.OrderedFactor{2}
  can_consume_tables = false
  supports_weights = false
  supports_class_weights = false
  orientation = StatisticalMeasuresBase.Score()
  external_aggregation_mode = StatisticalMeasuresBase.Mean()
  human_name = precision at fixed recall

ablaom added 27 commits January 21, 2026 11:23
oops, forgotten figure
tweak the implementation to exclude pr-curve "limit point"
tweak again
@ablaom ablaom marked this pull request as draft February 10, 2026 21:44
@ablaom
Copy link
Copy Markdown
Member Author

ablaom commented Feb 12, 2026

cc @DilumAluthge

ablaom and others added 22 commits March 6, 2026 08:37
tweak the implementation to exclude pr-curve "limit point"
tweak again
@ablaom
Copy link
Copy Markdown
Member Author

ablaom commented Mar 8, 2026

Closed in favour of #73.

@ablaom ablaom closed this Mar 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant