Labels and Models not harmonized between languages

[cmahnke]

I try to use spaCy to extract entities from a german text and it's englich translation to compare the results. This is suprisingly hard to do since it appears that neither the models nor the labels are harmonizes across languages.

Models:

These are offered at [Trained Models & Pipelines(https://spacy.io/models):

• English:
  • en_core_web_sm
  • en_core_web_md
  • en_core_web_lg
  • en_core_web_trf
    But there are also models like en_core_news_trf downlaodable, not sure if the can be consider recent.
• German:
  • de_core_news_sm
  • de_core_news_md
  • de_core_news_lg
  • de_dep_news_trf

When is use the install assistent to select models I get this combination (for using accurate models):

pip install -U pip setuptools wheel
pip install -U spacy
python -m spacy download en_core_web_trf
python -m spacy download de_dep_news_trf

From my point of view it's a reasonable expectation that this combination behaves similar. But it doesn't:

• de_dep_news_trfdoesn't work at all
• Using de_core_news_lg works, but has an annoying habit of using different labels then the English model like using PER instead of PERSON and using a MISC label instead of multiple more granular ones from the English model.

Since the main issue hasn't been made clear already:
One of the consequences, is that in German many things are dumped into one category /label which make spacy (or at least de_core_news_lg) almost unusable compared to the enflisch model:

• "Jahr 2024" is labeled MISC in German while in English it would correctly be labeled as DATE, same applies to something like "4.6.2024"
• Strings like "Edge" and "Brave" are also labeled "MISC" in German while they are correctly recognized as "PRODUCT" in English
• There are more examples on this, which make working with German difficult

Ideally things should be harmonized a bit more, like having a similar models use the same labels..

Why is that the case, should it considered to be a bug?