fix(pypi): skip malformed wheel anchors in SimpleAPI response#3907
Closed
kriscfoster wants to merge 3 commits into
Closed
fix(pypi): skip malformed wheel anchors in SimpleAPI response#3907kriscfoster wants to merge 3 commits into
kriscfoster wants to merge 3 commits into
Conversation
Some index servers (e.g. proxies of PyPI) occasionally emit anchors like `<a href=".../foo.whl#sha256=...">foo.whl</a>` whose filename does not parse as a valid PEP 427 wheel name (fewer than 4 `-` separators). Previously these would crash later in `parse_whl_name`. Skip them at parse time and warn via the repo logger so operators can spot the bad index entry. Wire the existing `pypi:simpleapi` logger through `simpleapi_download` so `parse_simpleapi_html` can emit the warning, and update the unit tests to pass a logger.
Add a regression test for the previous fix: an anchor whose filename does not parse as a valid PEP 427 wheel name (fewer than 4 `-` separators, e.g. `six.whl`) is skipped, and a valid wheel in the same response is still returned.
The `read_simpleapi` mocks in simpleapi_download_tests.bzl need to
accept the new `logger` keyword that `simpleapi_download` now threads
through, otherwise they error with:
read_simpleapi() got unexpected keyword argument: logger
Contributor
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Some SimpleAPI index servers (in particular certain PyPI proxies) occasionally
emit anchors whose filename is not a valid PEP 427 wheel name, for example:
The filename
six.whlhas only one-separator instead of the required four(
{distribution}-{version}(-{build})?-{python}-{abi}-{platform}.whl), so whenparse_whl_namelater runs on this entry it fails and aborts the wholeresolve.
This change skips such anchors at parse time in
parse_simpleapi_htmlandwarns via the existing
pypi:simpleapirepo logger, so operators can see andreport the bad index entry to the index maintainer.
To make the warning possible, the existing logger already built in
simpleapi_downloadis threaded through_get_dist_urls/_read_simpleapi/_read_index_result/parse_simpleapi_htmlin the samestyle as the logger passed to
select_whl.What happens downstream when an anchor is skipped
Skipping only removes the malformed anchor from
whls; it does not hide arequired wheel. In
parse_requirements.bzl:wheel (the common case in practice — proxies typically emit the wheel
twice, once well-formed and once with a stripped filename), the resolver
finds the well-formed entry and everything proceeds normally.
logs
Could not find a whl or an sdist with sha256=...and, since nocandidate remains for the target platform, returns
None, True— i.e.falls back to letting
pipresolve that one package normally. The builddoes not fail silently and the dependency is not dropped.
Either way the behavior is strictly better than the previous crash in
parse_whl_name.Test plan
test_malformed_whl_anchor_is_skippedintests/pypi/parse_simpleapi_html/parse_simpleapi_html_tests.bzlthatpasses a mixed response (one
six.whl-shaped anchor, one valid wheel)and asserts the malformed one is skipped while the valid one survives.
parse_simpleapi_htmlandhub_builderunit tests updatedto pass the now-required
loggerand continue to pass.six.whlanchor: the warn fires with the exact filename/href, andbazel build @pip//:requirements.bzlsucceeds.