Inconsistent wrapping of before-build/before-test for Rosetta on macOS

[rgommers]

With the deprecation of the GHA macos-13 runner images, which are the last ones to support Intel Macs, it will become more common for people to try to build x86-64 wheels on macOS arm64. When trying this for NumPy, I found that the before-build, before-test, test-command, etc. hooks are inconsistently wrapped for Rosetta usage (i.e., prepending -arch x86_64`):

• before_build is not wrapped, see macos.py#L451-L454
• before_test is wrapped, see macos.py#L665-L670
• test_command is also wrapped, see macos.py#L732

This is a problem, as a pip install somepkg in before-build may happily succeed while installing an arm64 package, which will cause a problem in next steps by either failing or silently doing the wrong thing. Here is a passing CI job for a numpy cp311-macosx_x86_64 wheel where the before-build phase installs scipy-openblas64 for arm64 which then goes missing from the final wheel (the build linked Accelerate instead).

I'd think that the expected behavior is for before-build and before-test to be wrapped in the same way.

Alternatively, it should be documented that this isn't done. One could then manually work around it and install an x86-64 package like so:

pip install scipy-openblas64 --platform macosx_10_13_x86_64 --only-binary :all: --target $(python -c "import os; print(f'{os.path.dirname(os.__file__)}/site-packages')")

[mayeut]

The doc is clear about the expectation for tests but indeed does not explicitly say anything about build.
Builds are always done from the native architecture for now so you're effectively cross-building when you target x86_64 on arm64 rather than doing a "native x86_64 Rosetta2" build.

I started some branch that was dealing with the venv architecture some time ago (even though not going up to wrapping before_build itself yet) in order to do "native x86_64 Rosetta2" builds, it might be time to restore this branch and add the missing piece to run the before_build step with Rosetta2.

That being said, I don't know what the expectation would be once Rosetta2 is removed from macOS (somewhere between 2 to 4 years from now, from a CI point of view, might be reworded as "once all builders are macOS>=28"), go back to cross-build only with no tests (i.e. what's currently done for build on arm64) or just drop "x86_64 on arm64" altogether ? This could impact how a change in the implementation for "x86_64 on arm64" build should look like to ease the next transition.

[rgommers]

This could impact how a change in the implementation for "x86_64 on arm64" build should look like to ease the next transition.

That's a good point. It would be best if that transition didn't change the build indeed, only removed the ability to run tests.

Builds are always done from the native architecture for now so you're effectively cross-building when you target x86_64 on arm64 rather than doing a "native x86_64 Rosetta2" build.

Thanks, I didn't have that 100% clear. That does make sense to me as well. The thing I was then running into - and what lots of people are going to run into - is that there isn't a great way to set up the non-native env if you need to link against a shared library. Maybe that's the thing that should be spelled out then. Something like this should do it if one has reasonable support for cross-compiling already:

$ mkdir host-env
$ pip install <all-host-deps> --platform macosx_10_13_x86_64 --only-binary :all: --target ./host-env
$ pip install pkgconf
$ export PKG_CONFIG_PATH=$PWD/host-env

Most users of the NumPy C API will also be able to get away with linking against an installed arm64 numpy package, because the headers are (modulo some corner cases) platform-independent.

Let me try a bit more to get this to work as a proper cross build.

[rgommers]

I got that to work (see numpy/numpy#29756), but there are some footguns. NumPy is already set up for cross compiling, and only has one host dependency that needs linking. Packages that rely on running the interpreter to introspect dependencies are going to have issues.

Keeping it as a cross build seems reasonable; changing to a native x86-64 Rosetta2 build would be too. For most packages both will work out of the box I suspect.

[joerick]

Ah, I hadn't considered that we could run the whole build under rosetta emulation! It's quite a nice idea, given how awkward cross-builds can be, and how fast Rosetta 2 seems to be.

That being said, I don't know what the expectation would be once Rosetta2 is removed from macOS

I suppose we could return to the cross-build method then, or, yeah, we could drop x86_64 entirely. I don't have a strong instinct on it at the moment... Apple has said that macOS 26 is the last that supports x86_64. But I will say I'm surprised by how much traffic PyPI still gets from x86_64 python-

It might be a lot of older CI images? Hard to say.

Without a strong signal that we'd return to cross-build, I think it'd be okay to declare YAGNI on the cross build method, I'm sure we can return to it in a few years time if we need to.

[henryiii]

Apple has also said macOS 27 is the last to fully support Rosseta 2 (save for some bits needed for games, it's going away). x86 machines are still around (mine is one).

We need to improve cross-compiles eventually, maybe it makes sense to introduce a control for cross-compile vs. emulation that we can use for Linux eventually? (Maybe Windows too?)

[joerick]

Apple has also said macOS 27 is the last to fully support Rosseta 2

Interesting.

We need to improve cross-compiles eventually, maybe it makes sense to introduce a control for cross-compile vs. emulation that we can use for Linux eventually?

I can see the logic here, but I'm a little concerned how complex things would end up with a switch here, maintaining the two code paths.

This is sorta an aside from the macOS conversation, but our previous forays into linux cross-compile switches weren't successful, mostly because the complexity they brought didn't seem worth it. If something like the cross-compile PEP was agreed, so we had an agreed way to express this to build backends, and a fixed strategy for dealing with the venvs, I'd feel a lot more comfortable with it.

[rgommers]

If something like the cross-compile PEP was agreed, so we had an agreed way to express this to build backends

Note that that PEP is Informational only, it's a summary. The actual relevant specification change is PEP 739 and that was accepted and is becoming available in CPython 3.14. Build backend support for that is coming along very slowly - I have a working cross compilation CI job with SciPy at least (not yet merged though).

What is still missing from that is a way to discover reliably where the build-details.json file is located (but cibuildwheel can know that, and at least for CPython it's a fixed location), and perhaps later a standardized hook to say "do a cross build".

and a fixed strategy for dealing with the venvs

That is the much more annoying part. Setting up build and host venvs and ensuring that the correct cross compilers/toolchain is installed is quite tricky. Not an issue on macOS, but on Linux it's quite fiddly.

[joerick]

Thank you for these pointers @rgommers! My PEP knowledge was way out-of-date, I was thinking of the draft PEP by @benfogle, which does include a build-backend switch.

So, PEP 739 lets us grab all the sysconfig information. (hmm, would we need two of these for macOS, since it works on two platforms?) I suppose the next step would be to tell a build backend - this is the platform you're building for, perhaps by passing a path to a build-details.json.

I have a working cross compilation CI job with SciPy at least (not yet merged though)

Could you link that? I'm curious how it looks...

[joerick]

Also, I did have a thought, if you want to choose to use Rosetta 2 for the compilation, you could invoke cibuildwheel itself with arch -x86_64 cibuildwheel - I'm pretty sure once we're in Rosetta, everything just continues as if it were on an x86_64 machine.

[rgommers]

Could you link that? I'm curious how it looks...

Sure: scipy/scipy@main...rgommers:scipy:ci-cross. Note that since I last touched that, PEP 739 support got merged in Meson master, so that dev install is no longer needed. meson-python support still isn't merged.

The essence of it is to set up build and host envs, then pass the paths to build-details.json and to a cross file:

python -m build -wnx -Csetup-args=--cross-file=$PWD/ci/cross_aarch64.ini -Csetup-args=-Dpython.build_config=$PWD/ci/build-details-py313-aarch64.json

[mokagio]

For what is worth, I pushed a test that should highlight the arch inconsistency here mokagio#1 - as expected, it currently fails.