Generate sdist out of the source tree

[raulcd]

At the Apache Arrow project we are trying to move to scikit-build-core as our build backend for PyArrow on the following PR:

• GH-36411: [Python] Use scikit-build-core as build backend for PyArrow and get rid of setup.py apache/arrow#49259

This has been great so far but we've found a small issue and wanted to get some feedback on what we could do or whether there's something we could do to accommodate our use case.

The PyArrow project builds bindings for Apache Arrow from the Arrow C++ implementation. We have a monorepo setup and we want to keep a single source for our LICENSES.txt and NOTICE.txt at the root for the project as this will be shipped for other language bindings and implementations and we don't want to keep several copies that can get out of sync.

Currently with setuptools we have a custom command for sdist where we copy the files on the release tree:
https://github.com/apache/arrow/blob/c9dfb3ee8701f0c8049c950558c87037e252f9a5/python/setup.py#L443-L469

The above is done out of the source tree.

We wanted to do something similar with scikit-build-core but when building the sdist it always uses an in source build from what we can see:

scikit-build-core/src/scikit_build_core/build/sdist.py

Lines 153 to 161 in 49b26c4

	paths = sorted(
	each_unignored_file(
	Path(),
	include=settings.sdist.include,
	exclude=settings.sdist.exclude,
	build_dir=settings.build_dir,
	mode=settings.sdist.inclusion_mode,
	)
	)

The way we've found to copy our Licenses at the moment is to build a custom build backend wrapping scikit-build-core but this is all currently done on the source tree, possibly polluting our source directory with files we don't want to keep.
https://github.com/apache/arrow/pull/49259/changes#diff-2aa6b17a1ee43ca6e6821bef231d956c4cabb71d8ab47ed427b3e2a3ee710392R49

Is there any possibility of doing source distributions out of the source tree? Is this a feature that could be interesting for the project?

[henryiii]

Yes, I'd like to support both pre-processors better (I think awkward-array could use that too), and collecting external files. I think we could use hatchling's system of force-include; I'm pretty sure we were starting to work on that a while back, need to see what the problem was. Though quick question, how might this interact with license-files from PEP 639? From what I can tell reading https://peps.python.org/pep-0639/#add-license-file-field, it seems like it's not allowed? Might be something to note for the new license PEP being worked on?

• The specified relative path MUST be consistent between project source trees, source distributions (sdists), built distributions (Wheels) and installed projects.
• Parent directory indicators (..) MUST NOT be used.

I think I might remember the issue with force-include, actually, it was hard to implement with editable installs (though I think we can handle it due to our custom .pth addition, while hatchling IIRC does not). Though for license files not supporting editable installs isn't a big issue.

[henryiii]

Ahh, here it is: #776

[LecrisUT]

• Parent directory indicators (..) MUST NOT be used.

I think this was to be somewhat consistent with rust which has the same restriction. You have an obvious workaround for that to just use symlinks and that is especially fine for license-files. It doesn't really matter if it fails for the windows user when using the git repo since that will be developing environment only, while production cases you would have an sdist where that symlink is already expanded.

Can we expand the same logic for other files, not sure, and it's not PEP-ified so it's mostly up to us. We could just say it's up to the projects if they are willing to use symlink method or not.

Re: force-include, definitely.