MSLC Data

Overview

This repository contains data and additional figures associated with the Metric Score Landscape Challenge submitted to the WMT Metrics Shared Task in 2023, 2024 and 2025.

Structure

The repository contains a directory for each MSLC paper: 2023/, 2024/ and 2025/, each of which contains a README file along with the data.

For MSLC25 interactive figures, see: https://nrc-cnrc.github.io/MSLC/2025/interactive/charts.html

For MSLC24 interactive figures, see: https://nrc-cnrc.github.io/MSLC/2024/interactive/charts.html

Licence

The contents of this repository are released under a CC-BY 4.0 licence.

Citing this work

If you choose to use this data, please cite the relevant paper(s):

@inproceedings{knowles-etal-2025-mslc25,
    title = "{MSLC}25: Metric Performance on Low-Quality Machine Translation, Empty Strings, and Language Variants",
    author = "Knowles, Rebecca  and
      Larkin, Samuel  and
      Lo, Chi-Kiu",
    editor = "Haddow, Barry  and
      Kocmi, Tom  and
      Koehn, Philipp  and
      Monz, Christof",
    booktitle = "Proceedings of the Tenth Conference on Machine Translation",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.wmt-1.69/",
    pages = "945--956",
    ISBN = "979-8-89176-341-8"
}

@inproceedings{knowles-etal-2024-mslc24,
  title = "{MSLC}24: Further Challenges for Metrics on a Wide Landscape of Translation Quality",
  author = "Knowles, Rebecca  and
      Larkin, Samuel  and
      Lo, Chi-kiu",
  editor = "Haddow, Barry  and
      Kocmi, Tom  and
      Koehn, Philipp  and
      Monz, Christof",
  booktitle = "Proceedings of the Ninth Conference on Machine Translation",
  month = nov,
  year = "2024",
  address = "Miami, Florida, USA",
  publisher = "Association for Computational Linguistics",
  url = "https://aclanthology.org/2024.wmt-1.34",
  doi = "10.18653/v1/2024.wmt-1.7",
  pages = "475--491",
}

@inproceedings{lo-etal-2023-metric,
  title = "Metric Score Landscape Challenge ({MSLC}23): Understanding Metrics{'} Performance on a Wider Landscape of Translation Quality",
  author = "Lo, Chi-kiu  and
      Larkin, Samuel  and
      Knowles, Rebecca",
  editor = "Koehn, Philipp  and
      Haddow, Barry  and
      Kocmi, Tom  and
      Monz, Christof",
  booktitle = "Proceedings of the Eighth Conference on Machine Translation",
  month = dec,
  year = "2023",
  address = "Singapore",
  publisher = "Association for Computational Linguistics",
  url = "https://aclanthology.org/2023.wmt-1.65",
  doi = "10.18653/v1/2023.wmt-1.65",
  pages = "776--799",
}