Skip to content

1.24.1 release for Python 3.14 from PyPI causes segfault in onnxrutime's own test suite #27392

@cbourjau

Description

@cbourjau

Describe the issue

Running the test suite using the latest 1.24.1 PyPI release causes a segfault. I have been able to reproduce this locally on osx-aarch64 (see below), but I believe this is also affecting other platforms, too.

The error appears to stem from the exception testing here.

Are the Python tests run on 3.14 on all platforms?

To reproduce

Running the following in the root folder of this project passes as expected:

$ uvx --python 3.13 -w "onnxruntime ==1.24.1" pytest onnxruntime/test/python/onnxruntime_test_python_mlops.py
=================================== test session starts ====================================
platform darwin -- Python 3.13.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/christian.bourjau/repos/onnxruntime
configfile: pyproject.toml
collected 6 items

onnxruntime/test/python/onnxruntime_test_python_mlops.py ......                      [100%]

==================================== 6 passed in 0.05s =====================================

Running the same for Python 3.14 causes a segfault:

$ uvx --python 3.14 -w "onnxruntime ==1.24.1" pytest onnxruntime/test/python/onnxruntime_test_python_mlops.py
=================================== test session starts ====================================
platform darwin -- Python 3.14.3, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/christian.bourjau/repos/onnxruntime
configfile: pyproject.toml
collected 6 items

onnxruntime/test/python/onnxruntime_test_python_mlops.py .Fatal Python error: Segmentation fault

Current thread 0x000000020213e240 (most recent call first):
  File "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/python3.14/contextlib.py", line 109 in __init__
  File "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/python3.14/contextlib.py", line 305 in helper
  File "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/python3.14/unittest/case.py", line 664 in run
  File "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/python3.14/unittest/case.py", line 725 in __call__
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/_pytest/unittest.py", line 389 in runtest
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/_pytest/runner.py", line 179 in pytest_runtest_call
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/_pytest/runner.py", line 245 in <lambda>
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/_pytest/runner.py", line 353 in from_call
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/_pytest/runner.py", line 244 in call_and_report
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/_pytest/runner.py", line 137 in runtestprotocol
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/_pytest/runner.py", line 118 in pytest_runtest_protocol
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/_pytest/main.py", line 396 in pytest_runtestloop
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/_pytest/main.py", line 372 in _main
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/_pytest/main.py", line 318 in wrap_session
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/_pytest/main.py", line 365 in pytest_cmdline_main
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/_pytest/config/__init__.py", line 199 in main
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/lib/python3.14/site-packages/_pytest/config/__init__.py", line 223 in console_main
  File "/Users/christian.bourjau/.cache/uv/archive-v0/eAsDVCypy8MzTAJHkeABu/bin/pytest", line 12 in <module>
                                                                                            Current thread's C stack trace (most recent call first):
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _Py_DumpStack+0x44 [0x10626c024]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at faulthandler_dump_c_stack+0x54 [0x1062e783c]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at faulthandler_fatal_error+0x16c [0x1062e76ec]
  Binary file "/usr/lib/system/libsystem_platform.dylib", at _sigtramp+0x38 [0x1941e16a4]     Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _PyFunction_Vectorcall+0x2e4 [0x105b8a310]                  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _PyFunction_Vectorcall+0x2e4 [0x105b8a310]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _PyObject_Call_Prepend+0x98 [0x105b06ec4]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at slot_tp_init+0x1c4 [0x105b06ca4]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at type_call+0x94 [0x105c64af4]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _TAIL_CALL_CALL_NON_PY_GENERAL.llvm.4573810571503426424+0x5d8 [0x105c90a68]                                                                             Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _PyFunction_Vectorcall+0x2e4 [0x105b8a310]                  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at method_vectorcall.llvm.7642887947468079067+0x1ac [0x105c40c8c]                                                                                          Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _TAIL_CALL_CALL_FUNCTION_EX.llvm.4573810571503426424+0xabc [0x105a3a248]                                                                                Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _PyFunction_Vectorcall+0x2e4 [0x105b8a310]                  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _PyObject_Call_Prepend+0x128 [0x105b06f54]                  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at slot_tp_call+0x1c4 [0x105c6610c]                            Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at PyObject_Vectorcall+0x200 [0x1059bd020]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _TAIL_CALL_CALL_KW_NON_PY.llvm.4573810571503426424+0xe0 [0x105c8ec38]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _PyFunction_Vectorcall+0x2e4 [0x105b8a310]                  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _PyObject_Call_Prepend+0x128 [0x105b06f54]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at slot_tp_call+0x1c4 [0x105c6610c]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _TAIL_CALL_CALL_FUNCTION_EX.llvm.4573810571503426424+0x798 [0x105a39f24]                                                                                Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _PyFunction_Vectorcall+0x2e4 [0x105b8a310]                  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _PyObject_Call_Prepend+0x128 [0x105b06f54]                  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at slot_tp_call+0x1c4 [0x105c6610c]                            Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at PyObject_Vectorcall+0x200 [0x1059bd020]                     Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _TAIL_CALL_CALL_KW_NON_PY.llvm.4573810571503426424+0xe0 [0x105c8ec38]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _PyFunction_Vectorcall+0x2e4 [0x105b8a310]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _PyObject_Call_Prepend+0x128 [0x105b06f54]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at slot_tp_call+0x1c4 [0x105c6610c]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at PyObject_Vectorcall+0x200 [0x1059bd020]
  Binary file "/Users/christian.bourjau/.local/share/uv/python/cpython-3.14.3-macos-aarch64-none/lib/libpython3.14.dylib", at _TAIL_CALL_CALL_KW.llvm.4573810571503426424+0x2dc [0x1059e18d4]                                                                                         <truncated rest of calls>
                                                                                            Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg (total: 2)

Urgency

This is a regression that causes a segfault in Python when using one of the library's main entry points. I think this renders onnxruntime as broken/unsafe on Python 3.14 for the time being, which seems rather urgent to me.

Platform

Mac

OS Version

15.7.3

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.24.1

ONNX Runtime API

Python

Architecture

ARM64

Execution Provider

Default CPU

Execution Provider Library Version

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions