Python 3.9 Build Error

[ax3l]

Building against CPython 3.9 I see the following errors:

/home/conda/feedstock_root/build_artifacts/adios-python_1602593380049/_build_env/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
adios.cpp: In function 'PyObject* PyInit_adios()':
adios.cpp:52250:30: error: 'PyTypeObject {aka struct _typeobject}' has no member named 'tp_print'; did you mean 'tp_dict'?
   __pyx_type_5adios_softdict.tp_print = 0;
                              ^~~~~~~~
                              tp_dict

Refs.: https://github.com/conda-forge/adios-python-feedstock

cc @jychoi-hpc

[ax3l]

Ping @jychoi-hpc @pnorbert @dmitry-ganyushin @lwan86: do you think this can be easily fixed? :)

[jychoi-hpc]

@ax3l sorry for my late reply. Can you help me to understand the error? Is this because of using new version of numpy? which python version did you use? I tried to build on Cori with python 3.9.7 (numpy 1.20.3, cython 0.29.24) and didn't get an error.

[ax3l]

@jychoi-hpc no worries. I think this is due to a new version of CPython. tp_print is dropped from Python 3.9 onwards.
https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_vectorcall_offset

Maybe Cython of some of the checked-in ADIOS Cython calls/file generations need to be updated?

[ax3l]

I see the problem on conda-forge:
https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_vectorcall_offset
and build like this:
https://github.com/conda-forge/adios-python-feedstock/blob/master/recipe/build.sh

Should I run Cython or something beforehand? :)

[jychoi-hpc]

I just tried again by doing exactly the same thing in build.sh on Cori without any issue. I used Python 3.9.7. It looks like condo-forge is using more recent python. Do you know which version of python on conda-forge is used?

Alternatively, you can try the following command to invoke 'cython' beforehand:

make CYTHON=y python

instead of

make python

[ax3l]

Thanks a lot, let's try this in Spack (spack/spack#28003 / spack/spack#28223) and Conda-Forge (conda-forge/adios-python-feedstock#11).

X-ref: #202

[ax3l]

Works, fantastic! Thanks for the hint!

I noticed that the makefile does not reliably re-create the .cpp files from the .pyx files. In Spack, I had to remove them beforehand and disable make parallelism (-j) to get it to build: spack/spack#28223

[Yikun]

Looks like we still hit this error with suggested commands:

[  214s] + make CYTHON=y MPI=y python
[  214s] python setup.py build_ext -lrt
[  214s] running build_ext
[  214s] building 'adios.adios' extension
[  214s] creating build
[  214s] creating build/temp.linux-aarch64-3.9
[  214s] gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -D_GNU_SOURCE -fPIC -fwrapv -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -I/opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include -I/home/abuild/rpmbuild/BUILD/adios-1.13.1/src/public -L/home/abuild/rpmbuild/BUILD/adios-1.13.1/src -I/opt/ohpc/pub/mpi/openmpi4-gnu12/4.1.4/include -fPIC -D_NOMPI -I/opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/adios/1.13.1/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/netcdf/4.9.0/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/hdf5/1.10.8/include -I/usr/include -I/usr/include/python3.9 -c adios.cpp -o build/temp.linux-aarch64-3.9/adios.o -Wno-uninitialized -Wno-unused-function
[  214s] In file included from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
[  214s]                  from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
[  214s]                  from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
[  214s]                  from adios.cpp:540:
[  214s] /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
[  214s]    17 | #warning "Using deprecated NumPy API, disable it with " \
[  214s]       |  ^~~~~~~
[  215s] adios.cpp: In function ‘PyObject* PyInit_adios()’:
[  215s] adios.cpp:52250:30: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52250 |   __pyx_type_5adios_softdict.tp_print = 0;
[  215s]       |                              ^~~~~~~~
[  215s] adios.cpp:52261:26: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52261 |   __pyx_type_5adios_file.tp_print = 0;
[  215s]       |                          ^~~~~~~~
[  215s] adios.cpp:52287:31: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52287 |   __pyx_type_5adios_blockinfo.tp_print = 0;
[  215s]       |                               ^~~~~~~~
[  215s] adios.cpp:52300:25: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52300 |   __pyx_type_5adios_var.tp_print = 0;
[  215s]       |                         ^~~~~~~~
[  215s] adios.cpp:52306:26: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52306 |   __pyx_type_5adios_attr.tp_print = 0;
[  215s]       |                          ^~~~~~~~
[  215s] adios.cpp:52312:27: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52312 |   __pyx_type_5adios_group.tp_print = 0;
[  215s]       |                           ^~~~~~~~
[  215s] adios.cpp:52338:31: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52338 |   __pyx_type_5adios_smartdict.tp_print = 0;
[  215s]       |                               ^~~~~~~~
[  215s] adios.cpp:52343:28: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52343 |   __pyx_type_5adios_writer.tp_print = 0;
[  215s]       |                            ^~~~~~~~
[  215s] adios.cpp:52348:30: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52348 |   __pyx_type_5adios_attrinfo.tp_print = 0;
[  215s]       |                              ^~~~~~~~
[  215s] adios.cpp:52353:29: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52353 |   __pyx_type_5adios_varinfo.tp_print = 0;
[  215s]       |                             ^~~~~~~~

Any suggestion?

[1] https://build.openeuler.org/package/live_build_log/home:huangtianhua:ohpc/adios-gnu12-openmpi4/standard_aarch64/aarch64