Bug[?]: ImportError after upgrading to transformers==5.0.0 and huggingface-hub==1.3.4

[jblake1965]

File "C:\Users***\venv\Lib\site-packages\spacy_transformers\data_classes.py", line 5, in
from transformers.tokenization_utils import BatchEncoding
ImportError: cannot import name 'BatchEncoding' from 'transformers.tokenization_utils' (unknown location)

• spaCy version: 3.8.11
• Platform: Windows-10-10.0.26100-SP0
• Python version: 3.11.4
• Pipelines: en_core_web_lg (3.8.0)
• spacy-transformers: 1.3.9

[weezymatt]

Hi.

The spacy-transformers package is incompatible with newly released v5.0.0 (transformers); the compatible versions are between [3.4.0, 4.53.3].

The issue is also due to the name change from tokenization_utils -> tokenization_python committed here on Dec 1, 2025.

You should get able fix your error by downgrading the version. As for huggingface-hub I'm unsure if there's any requirements but a downgrade might be necessary also.