nfpm.native_libs: new backend to generate pkg deps for nfpm packages

[cognifloyd]

This PR introduces a new backend: pants.backend.npm.native_libs
Initially, the backend will be availble as: pants.backend.experimental.nfpm.native_libs

I proposed this new backend (originally named bindeps) in this discussion: #22396

This backend inspects elf bin/lib files (like lib*.so) in packaged contents (for now, only in pex_binary targets) to identify package dependency metadata and inject that metadata on the relevant nfpm_deb_package or nfpm_rpm_package targets. Effectively, it provides an approximation of the native packagers features:

• rpm: rpmdeps + elfdeps
• deb: dh_shlibdeps + dpkg-shlibdeps (These substitute ${shlibs:Depends} in debian control files have)

Goal: Platform-agnostic package builds

This pants backend is designed to be platform-agnostic, like nFPM.

Native packaging tools are often restricted to a single release of a single distro. Unlike native package builders, this new pants backend does not use any of those distro-specific or distro-release-specific utilities or local package databases. This new backend should be able to build deb and rpm packages anywhere that pants can run (MacOS, rpm linux distros, deb linux distros, other linux distros, docker, ...).

To achieve the platform-agnostic goal, the scripts in this new backend use pure-python deps (elfdeps and pyelftools) to search the elf bin/lib files for provided and/or required SONAMEs. It also uses the official remote package search API for debian and ubuntu to map SONAMEs to the package that provides it. Thanks to pants' caching, and the stability of official package names for a given distro-release, these API calls are readily cacheable, which should minimize the traffic sent to these services.

Code Overview

pants.backend.experimental.nfpm.native_libs

Approximate logic flow (including rule calls):

• inject_native_libs_dependencies_in_package_fields:

  • deb_depends_from_pex:

    • extract the wheels from the pex
    • elfdeps_analyze_pex_wheels: Inspect each wheel, searching for .so (ELF libs) and executable files, and analyzing any ELF metadata to collect SONAMEs (ELF library names, generally a .so file name) that are required.
    • deb_search_for_sonames: Lookup packages containing the required SONAMEs.

  • rpm_depends_from_pex:

    • extract the wheels from the pex
    • elfdeps_analyze_pex_wheels: rpm only needs the SONAMEs in requires, so they do not need to be mapped to package names like deb requires. rpm also tracks which SONAMEs are provided, not just those required.
      w

  • Inject depends (and provides) field(s) on the nfpm_deb_package or nfpm_rpm_package targets. This is used when generating the config passed to nFPM so that nFPM includes the package dependency metadata in the built system package.

pants.backend.experimental.nfpm.native_libs.efldeps

elfdeps is a subsystem (with a default elfdeps.lock file) that allows configuring an alternate version of elfdeps. This might be useful if an alternate version (or a fork?) has fixes/features that apply to libraries contained in the wheels. As with all other backends, we only test with one version in CI, so users may encounter issues with other versions, which they are welcome to report on GitHub or in Slack.

Approximate logic flow (including rule calls):

• elfdeps_analyze_pex_wheels:
  • Run pex3 repository extract to create a directory with all of the wheels in the pex.
  • subsystem.setup_elfdeps_analyze_wheels_tool: Prepare a pex venv using elfdeps.lock
  • Run script pants.backend.nfpm.native_libs.elfdeps.analyze_wheels in pex venv:
    • iterates over contents each wheel as a zip file,
    • analyzes the contents with elfdeps
    • collects SONAMEs in requires and provides ELF metadata
    • returns collected results as JSON

pants.backend.experimental.nfpm.native_libs.deb

The script in nfpm.native_libs.deb requires aiohttp, aiohttp-retries, and beautifulsoup4. This script is treated as "internal" pants code for which users cannot change the dependency versions.

I couldn't think of any reasons to deal with the added complexity of yet another subsystem and lockfile. So, I added these deps to 3rdparty/python/requirements.txt, and then pull those package versions from the pants venv when running the script that needs them.

construct a pex in rule code that constrains the dependencies to locked versions present in the pants venv.

Approximate logic flow (including rule calls):

• deb_search_for_sonames:
  • create a pex venv that:
    • includes the search_for_sonames script
    • includes script dependencies constrained to the locked versions present in the pants venv.
  • Run script pants.backend.nfpm.native_libs.deb.search_for_sonames in pex venv:
    • Issue HTTPS package search API calls which returns results in HTML
      (the HTML-based API design predates JSON and other web API technologies).
    • Retry API calls up to 5 times with jitter + exponential backoff
    • Use beautifulsoup4 to extract the package names from the table in the HTML API response
      (this data structure has been consistent for many years, so script updates/maintenance should be minimal).

Note

The search_for_sonames script uses asyncio (and therefore does not use synchronous requests) because it could involve many IO-bound API calls (all of which should be cached using standard pants caching). As the dep is included with pants, I chose to use aiohttp because it's smaller than httpx. But, httpx has a more modern feature set, so if something like HTTP/2 proves to be an important optimization, the script could be refactored to use it instead.

TODO:

• tests for all new rules

• complete PR description

• find smaller refactor PRs to split off of this PR

  • nfpm: add get_package_field_sets_for_nfpm_content_file_deps rule #22863
    Add nfpm util rule to help collect packages from nfpm_file_content target deps of an nfpm_*_package

    • docs/notes/2.31.x.md
    • src/python/pants/backend/nfpm/util_rules/contents*.py
    • src/python/pants/backend/nfpm/rules.py

  • nfpm: support chaining multiple InjectNfpmPackageFieldsRequests (sorted w/ simple priority integer) #22864
    support chaining multiple InjectNfpmPackageFieldsRequests (sorted w/ simple priority integer)

    • docs/notes/2.31.x.md
    • src/python/pants/backend/nfpm/util_rules/inject_config*.py

  • nfpm.native_libs: new backend for nfpm pkg deps (only elfdeps subsystem) #22873
    introduce nfpm.native_libs backend with pex+elfdeps analysis support

    • 3rdparty/python/: BUILD, requirements.txt, user_reqs.lock*
    • build-support/bin/generate_builtin_lockfiles.py
    • docs/notes/2.31.x.md
    • src/python/pants/backend/experimental/nfpm/native_libs/: BUILD, __init__.py, register.py
    • src/python/pants/backend/nfpm/native_libs/: BUILD, __init__.py, rules.py
    • src/python/pants/backend/nfpm/native_libs/elfdeps/: BUILD, __init__.py, rules*.py, analyze_wheels*.py, subsystem.py, elfdeps.lock*

  • nfpm.native_libs: Add RPM package depends from packaged pex_binaries #22899
    add rpm+pex+elf rules to nfpm.native_libs backend

    • docs/notes/2.31.x.md
    • src/python/pants/backend/nfpm/native_libs/: BUILD, rules*.py

  • nfpm.native_libs: regen elfdeps.lock and fix plugin registration #22903

  • nfpm.native_libs: deb search_for_sonames script to find pkg deps #22904
    add deb/search_for_sonames script to nfpm.native_libs backend

    • docs/notes/2.31.x.md
    • 3rdparty/python/: requirements.txt, user_reqs.lock*
    • src/python/pants/backend/nfpm/native_libs/: rules.py
    • src/python/pants/backend/nfpm/native_libs/deb/: BUILD, __init__.py, rules.py, search_for_sonames*.py

  • nfpm.native_libs: Add DEB package depends from packaged pex_binaries #22905
    add deb+pex+elf support to nfpm.native_libs backend

    • docs/notes/2.31.x.md
    • src/python/pants/backend/nfpm/native_libs/: rules*.py, target_types.py
    • src/python/pants/backend/nfpm/native_libs/deb/: utils.py
    • src/python/pants/backend/nfpm/native_libs/python_ldap-3.4.4-cp311-cp311-linux_x86_64.whl