Ambiguous lock file when two simultaneously-installable extras require different versions of a shared transitive dependency

[carissableker]

Description

When two optional-dependency groups (extras) that can legitimately be
installed together each pull in a different version constraint for the
same transitive package, poetry lock writes two separate lock entries
for that package, distinguished only by extra == "..." markers whose
conditions overlap (both are true when the two extras are installed
together). Since only one physical version of a package can ever be
installed, this makes the lock file's outcome for that package
non-deterministic: repeated poetry install runs with the same
pyproject.toml, the same poetry.lock, and the same --extras flag
install different versions of the package across runs, purely depending on
the order in which the installer processes the (ambiguous) list of planned
operations.

My case:

Two extras (networks, sbml) can be installed together. networks pins
igraph==0.11.9 directly; sbml transitively depends on py4cytoscape,
which declares an unconstrained igraph dependency. poetry lock resolves
this into two lock entries for the same package, with overlapping markers:

[[package]]
name = "igraph"
version = "0.11.9"
markers = "extra == \"networks\" or extra == \"all\" or extra == \"sbml\""

[[package]]
name = "igraph"
version = "1.0.0"
markers = "extra == \"sbml\" or extra == \"all\""

Both conditions are true when sbml is requested, so the lock file doesn't
actually specify a single install outcome. Which version gets installed
depends on the order the installer processes its plan. Running
poetry install --extras "networks sbml" repeatedly, from a fresh
virtualenv each time, with the exact same pyproject.toml/poetry.lock,
installs different igraph versions across runs:

poetry lock
for i in 1 2 3; do
  rm -rf "$(poetry env info --path 2>/dev/null)"
  poetry install --extras "networks sbml" >/dev/null 2>&1
  poetry run python3 -c "import igraph; print(igraph.__version__)"
done

1.0.0
0.11.9
1.0.0

Confirmed on both Poetry 2.3.3 and the latest release, 2.4.1.

For comparison, pip install .[networks,sbml] against the same
pyproject.toml resolves deterministically to 0.11.9 every time, since it
resolves the actually-requested extras fresh at install time rather than
precomputing a lock file for every possible extras combination.

Workarounds

Adding the exact same explicit pin (igraph==0.11.9) directly to the
sbml extra as well (even though nothing in sbml imports igraph
directly) makes the pin visible in every marker branch that could otherwise
pull in the unconstrained transitive version, and resolves cleanly.

Poetry Installation Method

install.python-poetry.org

Operating System

Ubuntu 22.04.5 LTS

Poetry Version

Poetry (version 2.3.3), confirmed still present on Poetry (version 2.4.1),
the latest release at time of writing.

Poetry Configuration

cache-dir = "~/.cache/pypoetry"
data-dir = "~/.local/share/pypoetry"
installer.max-workers = null
installer.no-binary = null
installer.only-binary = null
installer.parallel = true
installer.re-resolve = false
keyring.enabled = true
python.installation-dir = "{data-dir}/python"
requests.max-retries = 0
solver.lazy-wheel = true
system-git-client = false
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"
virtualenvs.prompt = "{project_name}-py{python_version}"
virtualenvs.use-poetry-python = false

Example pyproject.toml

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "poetry-extras-igraph-conflict-mwe"
version = "0.0.0"
requires-python = ">=3.12"
dependencies = []

[project.optional-dependencies]
networks = ["networkx", "igraph==0.11.9"]
sbml = ["python-libsbml>=5.19", "tabularqual"]
all = ["python-libsbml>=5.19", "tabularqual", "networkx", "igraph==0.11.9"]

[tool.poetry]
package-mode = false

Poetry Runtime Logs

poetry-runtime.log

Excerpt from poetry install --extras "networks sbml" -vvv (fresh
virtualenv), showing the resolved lock file's two conflicting igraph
entries producing two operations for the same package within a single
install plan:

- Installing igraph (0.11.9): Skipped for the following reason: Already installed
...
- Updating igraph (0.11.9 -> 1.0.0)

A separate run (same lock file, same command, fresh virtualenv) instead
produced:

- Downgrading igraph (1.0.0 -> 0.11.9)
- Installing igraph (1.0.0): Skipped for the following reason: Already installed