Skip to content

Add average precision#70

Closed
ablaom wants to merge 47 commits intodevfrom
average-precision
Closed

Add average precision#70
ablaom wants to merge 47 commits intodevfrom
average-precision

Conversation

@ablaom
Copy link
Copy Markdown
Member

@ablaom ablaom commented Feb 12, 2026

Partly addresses #68.

This PR is ready for review after:

as that PR is the base of this one.

Note to self: Rebased off b56bbbf

Docstring
  AveragePrecision()

  Return a callable measure for computing the average precision. Aliases:
  average_precision.

  AveragePrecision()(ŷ, y)

  Evaluate AveragePrecision() on predictions ŷ, given ground truth observations y. It is
  expected that ŷ be a vector of distributions over the binary set of unique elements of
  y; specifically, ŷ should have eltype <:UnivariateFinite from the
  CategoricalDistributions.jl package.

  Average precision is the area under the empirical precision-recall curve, understood as
  a step function. This is to be contrasted with other measures going under the name "area
  under the precision-recall curve", in which the step function is usually replaced by a
  piece-wise linear approximation. Generally, differences between the two are only obvious
  when the number of observations is small, but it is faster to compute average precision.

  Reference: Wikipedia entry, Average precision
  (https://en.wikipedia.org/w/index.php?title=Information_retrieval&oldid=793358396#Average_precision)

  Definition
  ≡≡≡≡≡≡≡≡≡≡

  Adopting each distinct predicted probability p_1, p_2, \ldots, p_k for the positive
  class as a soft probability threshold for predicting an actual class, and assuming these
  thresholds are arranged in decreasing order, we obtain corresponding recalls R_1, R_2,
  \ldots, R_k (monotonically increasing) and precisions P_1, P_2, \ldots, P_k. Adding an
  extra recall, R_{k+1} = 1, the average precision implemented here is defined as

  \sum_{j=1}^k P_j (R_{j+1} - R_j)

  In some other implementations, such as scikit-learn
  (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score),
  P_j is replaced by P_{j+1}. However, this requires the definition of a precision for
  unit recall, in the case the predicted positive class probabilities exclude 1.0, and
  this is avoided here.

  Core implementation: Functions.average_precision.

  Generally, an observation obs in MLUtils.eachobs(y) is expected to satisfy
  ScientificTypes.scitype(obs)<:ScientificTypesBase.OrderedFactor{2}.

  See also precision_recall_curve.

  For a complete dictionary of available measures, keyed on constructor, run measures().

  Traits
  ≡≡≡≡≡≡

  consumes_multiple_observations = true
  can_report_unaggregated = false
  kind_of_proxy = LearnAPI.Distribution()
  observation_scitype = ScientificTypesBase.OrderedFactor{2}
  can_consume_tables = false
  supports_weights = false
  supports_class_weights = false
  orientation = StatisticalMeasuresBase.Score()
  external_aggregation_mode = StatisticalMeasuresBase.Mean()
  human_name = average precision

@ablaom ablaom marked this pull request as draft February 12, 2026 20:59
@ablaom
Copy link
Copy Markdown
Member Author

ablaom commented Feb 12, 2026

cc @DilumAluthge

Co-authored-by: Okon Samuel <39421418+OkonSamuel@users.noreply.github.com>
ablaom and others added 19 commits March 6, 2026 08:38
Co-authored-by: Okon Samuel <39421418+OkonSamuel@users.noreply.github.com>
Co-authored-by: Okon Samuel <39421418+OkonSamuel@users.noreply.github.com>
Co-authored-by: Okon Samuel <39421418+OkonSamuel@users.noreply.github.com>
Co-authored-by: Okon Samuel <39421418+OkonSamuel@users.noreply.github.com>
Co-authored-by: Okon Samuel <39421418+OkonSamuel@users.noreply.github.com>
Co-authored-by: Okon Samuel <39421418+OkonSamuel@users.noreply.github.com>
Co-authored-by: Okon Samuel <39421418+OkonSamuel@users.noreply.github.com>
Co-authored-by: Okon Samuel <39421418+OkonSamuel@users.noreply.github.com>
Co-authored-by: Okon Samuel <39421418+OkonSamuel@users.noreply.github.com>
@ablaom ablaom mentioned this pull request Mar 8, 2026
@ablaom
Copy link
Copy Markdown
Member Author

ablaom commented Mar 8, 2026

Closed in favour of #72.

@ablaom ablaom closed this Mar 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant