Consider using stubgen-pyx for automatic Cython stub generation

[jon-edward]

Overview

I'd like to propose integrating stubgen-pyx to automatically generate Python stub files (.pyi) from spaCy's Cython source code. I am the creator and maintainer of this utility.

Problem Statement

spaCy's Cython modules currently have limited IDE support. IDE tooling cannot access type hints and docstrings embedded in Cython source code, putting the user at a disadvantage if they're accessing a Cython module that doesn't have a handwritten .pyi file or if the .pyi file is out-of-date (or incomplete).

In addition to this, a spaCy developer must keep the .pyi file up-to-date when making changes to the Cython source, which is a time-consuming and error-prone process.

These difficulties might lead to the following user frustrations:

• Missing or incomplete type annotations and Cython-specific constructs
• Incomplete function signatures
• Loss of docstrings
• Poor IDE autocompletion and type hints for Cython modules

Proposed Solution

Integrate stubgen-pyx, a specialized stub generation tool for Cython, which offers:

Key Benefits

• Direct source analysis: Parses .pyx and .pxd files directly using the Cython compiler's AST
• Type information: Preserves type annotations and signatures from Cython source
• Docstring preservation: Maintains docstrings and decorators in generated stubs
• Full Cython support: Handles cdef classes, cpdef functions and other Cython constructs
• Type normalization: Automatically converts Cython types (e.g., bint → bool, unicode → str) to Python equivalents
• Smart import handling: Auto-deduplicates and trims unused imports in generated stubs
• Easy integration: Works via CLI or Python API; can be integrated into build pipelines and CI/CD

Implementation Approach

1. Add stubgen-pyx as a development/build dependency
2. Create a script to generate stubs for all Cython files (e.g., in build pipeline)
3. Include .pyi files in the distributed package for full IDE support
4. Integrate into CI/CD to regenerate stubs on Cython source changes (optional)

Proof of Concept

This is the current state of spacy/matcher/matcher.pyi:

from typing import (
    Any,
    Callable,
    Dict,
    Iterable,
    Iterator,
    List,
    Optional,
    Tuple,
    Union,
    overload,
)

from ..compat import Literal
from ..tokens import Doc, Span
from ..vocab import Vocab

class Matcher:
    def __init__(
        self,
        vocab: Vocab,
        validate: bool = ...,
        fuzzy_compare: Callable[[str, str, int], bool] = ...,
    ) -> None: ...
    def __reduce__(self) -> Any: ...
    def __len__(self) -> int: ...
    def __contains__(self, key: str) -> bool: ...
    def add(
        self,
        key: Union[str, int],
        patterns: List[List[Dict[str, Any]]],
        *,
        on_match: Optional[
            Callable[[Matcher, Doc, int, List[Tuple[Any, ...]]], Any]
        ] = ...,
        greedy: Optional[str] = ...
    ) -> None: ...
    def remove(self, key: str) -> None: ...
    def has_key(self, key: Union[str, int]) -> bool: ...
    def get(
        self, key: Union[str, int], default: Optional[Any] = ...
    ) -> Tuple[Optional[Callable[[Any], Any]], List[List[Dict[Any, Any]]]]: ...
    def pipe(
        self,
        docs: Iterable[Tuple[Doc, Any]],
        batch_size: int = ...,
        return_matches: bool = ...,
        as_tuples: bool = ...,
    ) -> Union[
        Iterator[Tuple[Tuple[Doc, Any], Any]], Iterator[Tuple[Doc, Any]], Iterator[Doc]
    ]: ...
    @overload
    def __call__(
        self,
        doclike: Union[Doc, Span],
        *,
        as_spans: Literal[False] = ...,
        allow_missing: bool = ...,
        with_alignments: bool = ...
    ) -> List[Tuple[int, int, int]]: ...
    @overload
    def __call__(
        self,
        doclike: Union[Doc, Span],
        *,
        as_spans: Literal[True],
        allow_missing: bool = ...,
        with_alignments: bool = ...
    ) -> List[Span]: ...
    def _normalize_key(self, key: Any) -> Any: ...

This is the result of stubgen-pyx:

from __future__ import annotations

from ..tokens.token import Token
from .levenshtein import levenshtein_compare


class Matcher:
    """Match sequences of tokens, based on pattern rules.

    DOCS: https://spacy.io/api/matcher
    USAGE: https://spacy.io/usage/rule-based-matching
    """

    def __init__(self, vocab, validate=True, *, fuzzy_compare=levenshtein_compare):
        """Create the Matcher.

        vocab (Vocab): The vocabulary object, which must be shared with the
        validate (bool): Validate all patterns added to this matcher.
        fuzzy_compare (Callable[[str, str, int], bool]): The comparison method
            for the FUZZY operators.
        """

    def __reduce__(self):
        ...

    def __len__(self):
        """Get the number of rules added to the matcher. Note that this only
        returns the number of rules (identical with the number of IDs), not the
        number of individual patterns.

        RETURNS (int): The number of rules.
        """

    def __contains__(self, key):
        """Check whether the matcher contains rules for a match ID.

        key (str): The match ID.
        RETURNS (bool): Whether the matcher contains rules for this match ID.
        """

    def add(self, key, patterns, *, on_match=None, greedy: str=None):
        """Add a match-rule to the matcher. A match-rule consists of: an ID
        key, an on_match callback, and one or more patterns.

        If the key exists, the patterns are appended to the previous ones, and
        the previous on_match callback is replaced. The `on_match` callback
        will receive the arguments `(matcher, doc, i, matches)`. You can also
        set `on_match` to `None` to not perform any actions.

        A pattern consists of one or more `token_specs`, where a `token_spec`
        is a dictionary mapping attribute IDs to values, and optionally a
        quantifier operator under the key "op". The available quantifiers are:

        '!':      Negate the pattern, by requiring it to match exactly 0 times.
        '?':      Make the pattern optional, by allowing it to match 0 or 1 times.
        '+':      Require the pattern to match 1 or more times.
        '*':      Allow the pattern to zero or more times.
        '{n}':    Require the pattern to match exactly _n_ times.
        '{n,m}':  Require the pattern to match at least _n_ but not more than _m_ times.
        '{n,}':   Require the pattern to match at least _n_ times.
        '{,m}':   Require the pattern to match at most _m_ times.

        The + and * operators return all possible matches (not just the greedy
        ones). However, the "greedy" argument can filter the final matches
        by returning a non-overlapping set per key, either taking preference to
        the first greedy match ("FIRST"), or the longest ("LONGEST").

        Since spaCy v2.2.2, Matcher.add takes a list of patterns as the second
        argument, and the on_match callback is an optional keyword argument.

        key (Union[str, int]): The match ID.
        patterns (list): The patterns to add for the given key.
        on_match (callable): Optional callback executed on match.
        greedy (str): Optional filter: "FIRST" or "LONGEST".
        """

    def remove(self, key):
        """Remove a rule from the matcher. A KeyError is raised if the key does
        not exist.

        key (str): The ID of the match rule.
        """

    def has_key(self, key):
        """Check whether the matcher has a rule with a given key.

        key (string or int): The key to check.
        RETURNS (bool): Whether the matcher has the rule.
        """

    def get(self, key, default=None):
        """Retrieve the pattern stored for a key.

        key (str / int): The key to retrieve.
        RETURNS (tuple): The rule, as an (on_match, patterns) tuple.
        """

    def pipe(self, docs, batch_size=1000, return_matches=False, as_tuples=False):
        """Match a stream of documents, yielding them in turn. Deprecated as of
        spaCy v3.0.
        """

    def __call__(self, doclike: object, *, as_spans=False, allow_missing=False, with_alignments=False):
        """Find all token sequences matching the supplied pattern.

        doclike (Doc or Span): The document to match over.
        as_spans (bool): Return Span objects with labels instead of (match_id,
            start, end) tuples.
        allow_missing (bool): Whether to skip checks for missing annotation for
            attributes included in patterns. Defaults to False.
        with_alignments (bool): Return match alignment information, which is
            `List[int]` with length of matched span. Each entry denotes the
            corresponding index of token pattern. If as_spans is set to True,
            this setting is ignored.
        RETURNS (list): A list of `(match_id, start, end)` tuples,
            describing the matches. A match tuple describes a span
            `doc[start:end]`. The `match_id` is an integer. If as_spans is set
            to True, a list of Span objects is returned.
            If with_alignments is set to True and as_spans is set to False,
            A list of `(match_id, start, end, alignments)` tuples is returned.
        """

class _FuzzyPredicate:
    operators = ('FUZZY', 'FUZZY1', 'FUZZY2', 'FUZZY3', 'FUZZY4', 'FUZZY5', 'FUZZY6', 'FUZZY7', 'FUZZY8', 'FUZZY9')

    def __init__(self, i, attr, value, predicate, is_extension=False, vocab=None, regex=False, fuzzy=None, fuzzy_compare=None):
        ...

    def __call__(self, token: Token):
        ...

class _RegexPredicate:
    operators = ('REGEX',)

    def __init__(self, i, attr, value, predicate, is_extension=False, vocab=None, regex=False, fuzzy=None, fuzzy_compare=None):
        ...

    def __call__(self, token: Token):
        ...

class _SetPredicate:
    operators = ('IN', 'NOT_IN', 'IS_SUBSET', 'IS_SUPERSET', 'INTERSECTS')

    def __init__(self, i, attr, value, predicate, is_extension=False, vocab=None, regex=False, fuzzy=None, fuzzy_compare=None):
        ...

    def __call__(self, token: Token):
        ...

    def __repr__(self):
        ...

class _ComparisonPredicate:
    operators = ('==', '!=', '>=', '<=', '>', '<')

    def __init__(self, i, attr, value, predicate, is_extension=False, vocab=None, regex=False, fuzzy=None, fuzzy_compare=None):
        ...

    def __call__(self, token: Token):
        ...

def unpickle_matcher(vocab, patterns, callbacks, validate, fuzzy_compare):
    ...

# This file was generated by stubgen-pyx v0.2.2 from spacy\matcher\matcher.pyx

The handwritten type annotations (ie. Callable[[str, str, int], bool]) can instead be written into the Cython file, making a single source of truth instead of maintaining two versions.

Configuration Example

# Generate all stubs in one command
stubgen-pyx spacy/

Or via Python API:

from stubgen_pyx import StubgenPyx

stubgen = StubgenPyx()
results = stubgen.convert_glob("spacy/**/*.pyx")

for result in results:
    if not result.success:
        print(f"Failed: {result.pyx_file} - {result.error}")

Expected Outcomes

• Improved IDE support (autocomplete, type hints) for spaCy's Cython modules
• Better developer experience for downstream users
• Faster development cycles
• Automatic documentation of Cython API through type stubs
• Foundation for stricter type checking and validation

Additional Notes

• stubgen-pyx is actively maintained and available on PyPI
• It has comprehensive test coverage and handles edge cases well
• The generated stubs are human-readable and can be manually reviewed/edited if needed
• No breaking changes to existing spaCy API or structure required

Would the spaCy team be interested in exploring this approach? I'd be happy to help with integration or discuss any questions about the tool.

---

Tool References:

• GitHub: https://github.com/jon-edward/stubgen-pyx
• PyPI: https://pypi.org/project/stubgen-pyx/
• Documentation includes CLI options, Python API, and configuration details