[DO-NOT-MERGE][PYTHON][PS] Support numpy 2.5 PEP 695 NDArray type hints in pandas-on-Spark

[HyukjinKwon]

What changes were proposed in this pull request?

as_spark_type in python/pyspark/pandas/typedef/typehints.py only recognized the legacy form of numpy.typing.NDArray, where NDArray[T] is a GenericAlias with __origin__ is np.ndarray and __args__ == (shape, np.dtype[T]).

Starting with NumPy 2.5, NDArray is declared as a PEP 695 type alias (type NDArray[ScalarT: np.generic] = np.ndarray[Any, np.dtype[ScalarT]]). At runtime NDArray[T] is then a GenericAlias whose __origin__ is the NDArray TypeAliasType (not np.ndarray) and whose __args__ is the single scalar type (T,). The old check no longer matched, so the annotation fell through to issubclass(tpe.__origin__, list), which raised TypeError: issubclass() arg 1 must be a class.

This PR:

• adds _extract_ndarray_scalar_type, which detects the new PEP 695 NDArray form (origin is a TypeAliasType whose __value__ expands to np.ndarray) and returns its scalar element type;
• routes that scalar through as_spark_type to build the ArrayType, mirroring the legacy branch;
• guards the existing list branch with isinstance(tpe.__origin__, type) so a non-class __origin__ can never crash issubclass again.

The legacy NumPy (<2.5) path is unchanged.

Why are the changes needed?

On runners with NumPy 2.5 (e.g. Build / MacOS-26, Python 3.12), pyspark-pandas tests that use NDArray[...] type hints — test_apply_batch_with_type, test_apply_with_type, test_transform_batch_with_type, and their *-connect parities — fail with:

TypeError: Cannot interpret 'NDArray[int]' as a data type
TypeError: issubclass() arg 1 must be a class

Evidence: apache/spark run 28064134730 (Build / ... MacOS-26, Python 3.12), module pyspark-pandas / pyspark-pandas-connect.

Does this PR introduce any user-facing change?

No. It restores correct NDArray type-hint handling under NumPy 2.5.

How was this patch tested?

The existing pyspark.pandas.tests.test_typedef.TypeHintTests.test_as_spark_type_pandas_on_spark_dtype already exercises as_spark_type(ntp.NDArray[...]) / pandas_on_spark_type(ntp.NDArray[...]) across many element types; the fix makes it pass under NumPy 2.5 while keeping it green under NumPy 2.4.

Validated locally on Python 3.12 against:

• the real NumPy 2.4 NDArray (legacy form) — no regression;
• a faithful simulation of NumPy 2.5's PEP 695 NDArray (origin is a TypeAliasType) — previously reproduced the exact issubclass() arg 1 must be a class failure; now as_spark_type(NDArray[int]) == ArrayType(LongType()).

NumPy 2.5 is not yet on PyPI for the OSS CI image, so this cannot be reproduced on the SBT fork lane; it reproduces on the NumPy-2.5 MacOS-26 scheduled lane.

Was this patch authored or co-authored using generative AI tooling?

Yes, Generated-by: Claude Code.

CI evidence

• Before (red): https://github.com/apache/spark/actions/runs/28064134730 — Build / ... MacOS-26, Python 3.12, NumPy 2.5.0; pyspark-pandas / pyspark-pandas-connect fail in test_apply_batch_with_type with TypeError: Cannot interpret 'NDArray[int]' as a data type / issubclass() arg 1 must be a class.
• After (fork SBT Build, no-regression): https://github.com/HyukjinKwon/spark/actions/runs/28204821025 — confirms the change does not regress the legacy NumPy (<2.5) path.

Note: the OSS CI image where this fails uses NumPy 2.5.0, which is not yet installable from PyPI for the fork's standard lanes, so the red MacOS-26 lane cannot be re-run green on the fork directly. The fix was validated locally on Python 3.12 against (a) the real NumPy 2.4 NDArray (legacy form, no regression) and (b) a faithful simulation of NumPy 2.5's PEP 695 NDArray (origin is a TypeAliasType), which reproduced the exact failure before the fix and produced as_spark_type(NDArray[int]) == ArrayType(LongType()) after. The existing test test_typedef.TypeHintTests.test_as_spark_type_pandas_on_spark_dtype covers NDArray[...] and will turn green on the NumPy-2.5 lane.

CI evidence

• Before (red): Build / Python-only (Python 3.12, MacOS26) on apache/spark — https://github.com/apache/spark/actions/runs/28064134730 (pyspark-pandas + pyspark-pandas-connect fail: test_apply_batch_with_type → TypeError: Cannot interpret 'NDArray[int]' as a data type, numpy 2.5.0).
• After (green): same job re-run on the fork with this fix (macOS-26, numpy 2.5.0; matrix narrowed to the two affected modules on a validate-only branch) — https://github.com/HyukjinKwon/spark/actions/runs/28205257318 (pyspark-pandas ✅, pyspark-pandas-connect ✅).

[gaogaotiantian]

I think #56757 should work