Update dependencies to support python 3.10 to 3.13 by gsaint · Pull Request #20 · dataiku/dss-plugin-nlp-preparation

gsaint · 2026-01-27T17:00:42Z

Add Python 3.10 to 3.13 support by introducing version-conditional dependencies for packages that have breaking changes or lack support across Python versions

Changes

Replace pycld3 with pycld2 for language detection on Python >= 3.10
- Very close API and compatibility
- Reuse pycld3 language ID mapping
Add spacy 3.x support
- Add explicit lemmatizer component initialization for blank language models, which is an API change starting at 3.0
- Use spacy 3.X for python >= 3.10
Fix pandas 2.0 compatibility: iteritems() → items()
Fix NumPy 2.0 compatibility: np.NaN → np.nan
Testing
- Expand unit test coverage to make sure we keep backward compatibility
- Add make command to execute unit tests within debian containers across Python 3.6-3.13
- Add GitHub Actions workflow to execute unit tests across Python 3.8-3.13

Local unit tests

Run unit tests on all supported python versions

make docker-test-all

Execute unit tests on a single python version

make docker-test-py312

Verification

Language detection
- pycld3 and pycld2 are only used for text longer than 140 chars, otherwise existing code using langid is executed
Spell checking and text cleaning
- Verify spaCy tokenization and lemmatization is working on p39 and from p310 to p313

…versions (except p3.6)

gsaint · 2026-01-28T09:24:56Z

python-lib/language_detector.py

        lang_probability = float(language_detection_object[1])
        return (lang_id, lang_probability)

+    def _cld_detection(self, doc: AnyStr) -> (AnyStr, float):


Core change for CLD

gsaint · 2026-01-28T09:25:20Z

python-lib/spacy_tokenizer.py

                nlp = spacy.blank(language)  # spaCy language without models (https://spacy.io/usage/models)
+                # spacy 3.x requires explicit lemmatizer component for blank languages
+                # Not all languages have lookup data, so we wrap in try/except
+                if spacy.about.__version__.startswith("3"):


Core change for spacy

nicolasdalsass

Python3.6 needs fixing

Great job on resurrecting unit tests with a proper setup for both local runs and github actions 👍

nicolasdalsass · 2026-01-29T08:51:54Z

python-lib/spacy_tokenizer.py

+                # spacy 3.x requires explicit lemmatizer component for blank languages
+                # Not all languages have lookup data, so we wrap in try/except
+                if spacy.about.__version__.startswith("3"):
+                    try:
+                        nlp.add_pipe("lemmatizer", config={"mode": "lookup"})
+                        nlp.initialize()
+                    except Exception:
+                        # Language doesn't support lookup lemmatization, continue without it
+                        if "lemmatizer" in nlp.pipe_names:
+                            nlp.remove_pipe("lemmatizer")


I don't understand the logic behind "spacy require a lemmatizer, but on the other hand, we can just remove it if something goes wrong" ?

So it's a bit tricky and was raised by the unit tests.

spacy 2.x: Lemmatization was automatically built into the tokenizer

spacy 3.x: Lemmatization requires an explicit lemmatizer pipe

This is only required for blank languages (languages without specific pre trained model that we can load), or when we decide use_models=False to keep it light weight on the memory.

We use lookup to automatically find a lemmatization for the language, however not all languages have lookup data, and when they don't it's raising an exception, so we fall back to no lemmatizer.

nicolasdalsass · 2026-01-29T08:52:01Z

code-env/python/spec/requirements.txt

+spacy[lookups,ja,th]==3.8.11; python_version >= '3.10'
 symspellpy==6.7.0
-tqdm==4.60.0
+tqdm==4.66.3


The wheel doesn't install for python3.6. Let's add a conditional requirement so that it actually installs properly on 3.6 since we keep supporting it.

a013070

============================= 40 passed in 41.93s ============================== [DONE] Python 3.6 tests completed

Good catch

nicolasdalsass

LGTM

gsaint added 4 commits January 27, 2026 14:37

Simplify unit test run

38eb640

Use pycld2 instead of pycld3 for language detection in python>=3.10

b948deb

Add support for spacy 3.X

e83d0c1

Upgrade dependencies to support up to python 3.13

1993bc2

gsaint added the dependencies Pull requests that update a dependency file label Jan 27, 2026

gsaint added 8 commits January 28, 2026 09:33

Align pandas and numpy requirements with DSS

bdaabb0

Add GitHub action workflow to run unit tests on all supported python …

b7f079f

…versions (except p3.6)

Update GitHub action test python version and fix numpy requirements

0079cc6

Avoid/abort multi unit test CI run when pushing multiple commits

a6798fe

Run CI unit tests at the PR level and/or on main branch

b6e3bec

Fix requirements for python 3.6, 3.7 and 3.8

066cc33

Add unit test utility to execute tests within docker

2c0a6c5

Bump tqdm to 4.66.3

24abbe8

gsaint commented Jan 28, 2026

View reviewed changes

Update plugin category and tags to NLP

9b80f39

gsaint marked this pull request as ready for review January 28, 2026 13:41

v1.4.0

1501c5e

gsaint mentioned this pull request Jan 28, 2026

Bump tqdm from 4.60.0 to 4.66.3 in /code-env/python/spec #18

Closed

gsaint requested a review from nicolasdalsass January 28, 2026 14:09

nicolasdalsass requested changes Jan 29, 2026

View reviewed changes

Fix python 3.6 requirements

a013070

nicolasdalsass self-requested a review January 29, 2026 10:58

nicolasdalsass approved these changes Jan 29, 2026

View reviewed changes

gsaint merged commit 256c39b into main Jan 29, 2026
8 checks passed

gsaint deleted the perso/gsaint/chore/dss14-sc-290422-make-nlp-text-preparation-plugin-python312 branch January 29, 2026 13:09

gsaint restored the perso/gsaint/chore/dss14-sc-290422-make-nlp-text-preparation-plugin-python312 branch January 29, 2026 13:10

nicolasdalsass deleted the perso/gsaint/chore/dss14-sc-290422-make-nlp-text-preparation-plugin-python312 branch January 29, 2026 14:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update dependencies to support python 3.10 to 3.13#20

Update dependencies to support python 3.10 to 3.13#20
gsaint merged 15 commits intomainfrom
perso/gsaint/chore/dss14-sc-290422-make-nlp-text-preparation-plugin-python312

gsaint commented Jan 27, 2026 •

edited

Loading

Uh oh!

gsaint Jan 28, 2026

Uh oh!

gsaint Jan 28, 2026

Uh oh!

nicolasdalsass left a comment

Uh oh!

nicolasdalsass Jan 29, 2026

Uh oh!

gsaint Jan 29, 2026

Uh oh!

nicolasdalsass Jan 29, 2026

Uh oh!

gsaint Jan 29, 2026 •

edited

Loading

Uh oh!

nicolasdalsass left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

gsaint commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Local unit tests

Verification

Uh oh!

gsaint Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

gsaint Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

nicolasdalsass left a comment

Choose a reason for hiding this comment

Uh oh!

nicolasdalsass Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

gsaint Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

nicolasdalsass Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

gsaint Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nicolasdalsass left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gsaint commented Jan 27, 2026 •

edited

Loading

gsaint Jan 29, 2026 •

edited

Loading