Evaluation, Reproducibility, Benchmarks Meeting 46

Minutes of Meeting 46

Date: 26th August, 2026

Present

• Carole
• Annika
• Oliver

Updates

• Metrics Reloaded Implementation

  • Carole extended the tutorials for object detection, instance segmentation, image-level classification, and semantic segmentation (binary)
  • Testing almost completed
  • During that, she found an additional pitfall (edge case) related to the choice of the assignment

• CI paper

  • The CI paper revision was circulated
  • Decision trees for which CI method to pick for which metric and sample sizes were included
  • Carole will include the updates in the code

• Systematic review (brain)

  • TBD: Inclusion / exclusion criteria for journals (e.g., stratify by predatory journals, exclude based on quality)
  • Additionally: planned survey for clinicians

• Challenge review paper:

  • Revision was circulated

• Dynamic benchmark:

  • 3 of the MICCAI 2026 challenges agreed to share the data
  • Problem: Data sharing agreement has to be created (waiting for it on DKFZ end)