Remove the experimental fast-deps feature

[pradyunsg]

This feature did not yield significant speed-ups, possibly due to insufficient optimisation of the calls being performed.

Adding a cross reference to #7049 (comment), which links to other issues that are relevant for making a decision on what to do with this feature.

FWIW, we did reuse nearly all of the code for PEP 658 though, so the effort put toward this feature was certainly not in vain. It might just be that our implementation of lazy_wheel could've been optimised further, but given that PEP 658 is on the horizon, I reckon it's OK to trim this functionality since we're going to get the similar semantics with the dist-info-metadata.

[pradyunsg]

I'm not sure yet if this is the route we should go down, FWIW, but I'm filing a PR to facilitate the discussion around this and simplify it to a go/no-go decision for us. :)

[dholth]

Something to time: a lazier lazy wheel #11481

[casassg]

Given #10748 (comment) it may be good to keep. If we can make --dry-run --report avoid downloading wheels for report generation, then we would be able to have a download-less dependency closure generation via pip install --use-feature=fast-deps --dry-run --ignore-installed --report - . If we remove this feature before PEP 658 is merged, then we will be back to requiring to download wheels until the feature is implemented on PyPi (or for that matter implemented in internal index services like JFrog Artifactory which some companies use to serve internal wheel repositories).

[jbylund]

Agree with @casassg

Warehouse doesn't yet support pep 658, and many people (myself included) use different package repositories which may not support pep 658 even if warehouse does. So I think it's at least worth profiling/looking over the existing implementation to see how it can be improved before removing it entirely.

Speaking of... anyone have a good testcase?

[ddelange]

Hi 👋

xref #11512 (comment) the warehouse PR is now merged

is the plan still to remove fast-deps? it is still an opt-in feature right? do I need to enable it since #11111 mentions it 'builds on top of it'?

[cosmicexplorer]

@ddelange what I meant by that was that in #11111, pip extends the code paths and classes that we initially developed for the purposes of fast-deps, to represent "virtual" dists which do not correspond to any concrete place on the filesystem and contain only the metadata necessary for the resolver to make progress. I meant that in the sense @pradyunsg describes in the OP of this PR, to underline that the research effort served a valuable purpose even if it didn't get the eye-popping numbers we wanted without some more effort. I did not mean that the fast-deps feature itself was load-bearing in any way, so thanks for asking so I could clarify that.

Analogy: optimizing --layout zipapp PEX files

In pex-tool/pex#2175 my goal for pex was to reduce i/o and computation from compressing very large amounts of third-party code we do not control. I figured out a hack to reduce that effort, but as @jsirois points out, it's often more robust and maintainable in the long term to develop standard protocols that are designed for the task at hand than rely on clever hacks to optimize existing ones. In that case, while I found it was possible to cache traditional --layout zipapp pex files with a clever hack, I learned that the --layout packed format had been designed from the ground up to enable that cacheability. Analogously, @uranusjr spent the time to develop PEP 658 to avoid the need for us to use clever hacks like fast-deps to get the information pip requires. In both cases, we were able to leverage features of the zip file format to put off standardization for a while, but we're in a better place now that more thoughtful infrastructure has been laid down.

Notes on Cacheability in Package Resolution

Regarding cacheability: there remains one further avenue to investigate from #7819: amortizing/memoizing certain subproblems across separate invocations of the resolver. While I was able to identify one compartmentalized improvement in #7729, in general I still think pip (and all other package managers) currently spend most of their time recomputing a lot of the same constraints against each other over and over again for most hot-path use cases; for example, after identifying the subset of wheels that are compatible with a given interpreter, that information can be recorded and cached indefinitely by the local pip client to immediately jump to the subset of compatible wheels in future resolves.

spack currently performs an analogous process, deserializing and processing all of the dependency constraints annotated in spack package recipes upon each spack invocation (cc @tgamblin @alalazo)). In contrast, spack's main package repository is maintained as part of the spack codebase itself, so it can always expect the luxury of loading everything from the local filesystem. Still, it demonstrates that amortizing dependency graph traversal while incorporating updates from the source of truth is an intrinsic issue to package resolution, as opposed to the problem solved by fast-deps which was a temporary workaround to slowly-evolving packaging standards.

Local indices to amortize resolves: learning from Cargo

My vision is for pip to eventually perform most resolves entirely locally, only calling out to pypi when it's unable to find a satisfying set and it determines that a newer release of some package might be able to do so. I think this should be feasible without needing to store anything close to the magnitude of all the package contents on pypi, because cargo does that right now, and has some useful comments (https://github.com/rust-lang/cargo/blob/263d24baca913d7ace29e2f59a3254e47272fd6b/src/cargo/sources/registry/index.rs#L25-L30):

//! One important aspect of the index is that we want to optimize the "happy
//! path" as much as possible. Whenever you type `cargo build` Cargo will
//! *always* reparse the registry and learn about dependency information. This
//! is done because Cargo needs to learn about the upstream crates.io crates
//! that you're using and ensure that the preexisting `Cargo.lock` still matches
//! the current state of the world.

As detailed in that file, cargo essentially just serializes a list of the dependency relationships to json in order to publish their index, which is exactly how I implemented the persistent dependency resolution cache in #7819 that multiplied the effect of the fast-deps technique alone (https://github.com/cosmicexplorer/pip/blob/40cec94281aa9165c689dd4bb69f953e2fd5fb06/src/pip/_internal/resolution/legacy/resolver.py#L171-L304).

fast-deps (and now PEP 658) gives you the dependency relationships needed to iterate the resolver over a single candidate without actually downloading the many-megabyte zip file, but pip still currently has to pay the cost of network calls to move the resolver forward to evaluate each candidate, and if the candidate is discarded it has to do that again until the resolver finds a satisfying dependency set. In this file, cargo describes generating an index over another index to amortize the graph traversal performed during a cargo resolve while also being able to incorporate any updates to the index that have occurred since the last resolve. There's nothing special cargo is doing here with rust that we can't do in python; as they note:

//! Consequently, Cargo "null builds" (the index that Cargo adds to each build
//! itself) need to be fast when accessing the index. The primary performance
//! optimization here is to avoid parsing JSON blobs from the registry if we
//! don't need them. Most secondary optimizations are centered around removing
//! allocations and such, but avoiding parsing JSON is the #1 optimization.

Assumptions and Requirements

Pip shares Cargo's assumption that each individual release of any package is immutable, so the dependency relationships that fast-deps/PEP 658 obtains from each wheel can therefore be cached and reused indefinitely after they are first computed. My hypothesis is that we can improve resolve performance by:

1. (short term) mirroring every lookup of package==version#checksum => [dependency requirements] that pip extracts from pypi during a normal resolve into a local database of some form, to be reused whenever pip evaluates that installation candidate in future resolves.
   • As noted in consume packed wheel cache in zipapp creation pex-tool/pex#2175, every new cache implies a tradeoff of speedup/disk space. I suspect this will be a small enough dataset to avoid that concern, though, especially if we restrict ourselves to just indexing the packages pip has seen before instead of trying to contain all of pypi at once.
   • As with cargo, there are some dependency relationships that are not immutable, like editable installs or the most recent release of a package, and we would need to be able to incorporate that kind of information too.
2. (longer term) optimizing the local dependency index and the new resolver in tandem.
   • Maybe this is the hubris of a beginner, but I am under the impression pip would be able to perform most resolutions in the blink of an eye if it didn't need to intersperse filesystem and/or network i/o between every step of the resolver's graph walk.
   • The reason PEP 658 works is that it cuts out the ceremony required to access the dependency relationships for a given installation candidate--I think if we continue to pare down the stuff we cache and retain from entire wheels to just the information we need for a resolve (similarly to discussions of what to include in pip install --report), we can turn this into more and more of a compute-bound rather than IO-bound problem.

Conclusion

I guess what I'm saying is: I'm really hopeful that pip resolves can be further improved, and I think that will directly leverage the fast-deps work, but you're right that PEP 658 should obviate the feature itself. However, as per the vague sketch above, I think that we might be introducing multiple other ways to virtualize the information we get from pypi, and fast-deps and LazyZipOverHTTP might not look that out of place in the end result.

That said, the claimed performance improvement leveraging fast-deps remains to be demonstrated, and if it doesn't end up going anywhere then it seems like pip would best serve its users by removing the experiment.

[cosmicexplorer]

Just created #12184 to move forward on all the stuff in that comment instead of taking up space here.

[cosmicexplorer]

Also take a look at #12208, which I hope makes a convincing enough case for instead turning on --use-feature=fast-deps by default.

[sbidoul]

I'm +1 for removing fast-deps. When PyPI will have backfilled PEP 658 metadata (tracked in pypi/warehouse#8254) it will not be useful anymore and removing it from the codebase will simplify reasoning about further optimizations.

[uranusjr]

Personally I feel like flag would be useful as a transition to the ”new” range fetch approach (mentioned above by @cosmicexplorer). After we merge the entire addition, we can use the usual use-feature/use-deprecated mechanism to enable it by default.

[sbidoul]

the ”new” range fetch approach

But do we need that in a world where PyPI has backfilled PEP 658 metadata?

[dholth]

This feature has new energy, and we don't live in a world where PyPI has backfilled PEP 658 metadata. Also, it can work in a simpler non-pypi index.

[pfmoore]

Do we have any indications from PyPI of the timescales for backfilling? Also, as @dholth mentions, fast-deps would be useful for indexes that don't support PEP 658. The pytorch binaries, for example, are served from their own index - does that support PEP 658? I just checked https://download.pytorch.org/whl/cu117, and it doesn't seem to...

[dstufft]

We don't have a set schedule for backfilling, but we were talking about it last Friday and I believe we hope to do it soon. I think we may be waiting until a new version of packaging is released and we can pull in the packaging metadata API to validate the metadata at the same time (obviously invalid metadata we can't do anything about, but we can at least record which metadata is valid or invalid).

[pradyunsg]

I think we may be waiting until a new version of packaging is released and we can pull in the packaging metadata API to validate the metadata at the same time (obviously invalid metadata we can't do anything about, but we can at least record which metadata is valid or invalid).

That came out last weekend, so this should be unblocked from that perspective I think. :)

[pelson]

Given we have backfilling, and we have a released pip which includes a true metadata-only (PEP-658) resolve (for pip install, pip wheel etc.), I think it would make sense to remove fast-deps. I think it is a feature that has been hugely impactful, and has ultimately influenced a faster experience in pip core.

I did a quick side-by-side comparison of how resolve works out in pip install --dry-run and it is pretty impressive how quick the resolve stage is nowadays.

I'd be happy to pick it up if there is consensus, and nobody else has the bandwidth to do so.

[cosmicexplorer]

I personally implemented the PEP 658 support in pip #11111 because as you say it is indeed superior, if we can expect it. I have had this PR open for multiple years #12208 conclusively describing the problem space and solution, with initial investigation from @dholth which found pypi had suddenly stopped supporting negative http range requests. @radoering provided immensely helpful performance analysis from his adaptation of the technique to poetry, after I gave a lengthy conference talk on the subject which I believe you should find invigorating: https://web.archive.org/web/20250425181258/https://cfp.packaging-con.org/2023/talk/hpuhu7/

PEP 777 correctly describes the situation: https://peps.python.org/pep-0777/#backwards-compatibility

This PEP relies on resolvers being able to efficiently acquire package metadata, usually through PEP 658. This might present a problem for users of package indices that do not serve PEP 658 metadata. However, today most installers fall back on using HTTP range requests to efficiently acquire only the part of a wheel needed to read the metadata, a feature most storage providers and servers include. Furthermore, future improvements to wheels such as compression will make up performance losses due to inspecting files in the wheel.

[cosmicexplorer]

I have described repeatedly and at great length the rationale to accept the corrected version of fast-deps: #12208

While PEP 658 is the standards-compliant solution and metadata from there is already preferred when available, --use-feature=fast-deps avoids downloading wheels against --find-links repos and any pypi index not supporting PEP 658 yet. Most non-pypi indices will be either of these, because it's very easy to expose those to pip with a simple file server, so improving fast-deps (and turning it on by default) is necessary to extend the benefits from the recent metadata resolve work to most users hosting their own index, especially corporations running pip in their internal CI.

Furthermore, upon request I have described at great length how the metadata resolution process can be abstracted and extended: #12921

The result makes pip take 1-2 seconds for most resolves and supports very efficient parallel downloads as well. There is no interest in accepting those PRs I spent several years repeatedly polishing to professional standards suitable for pip.

I would be very interested in following up with anyone who can help me get these changes merged. I am also available to call/chat/pair on this topic.

[cosmicexplorer]

As I described in the conference talk, these years of work I did were intended to help the whole python community by making everything instant. I got exactly one statement from pypi about whether reducing bandwidth costs would be helpful to them: #12257 (comment) I know I personally benefit from using my pip fork because I have a metered internet connection with a monthly limit. I have provided very clear and direct benchmarks for speed and bandwidth for every PR in the metadata workstream.

I have also now further indulged in a much greater rewrite of pip's package_finder.py and related code—now it avoids filtering while downloading from pypi, and is able to cache even more. This results in pip becoming so instant (again, around 1 second for very large resolves) that I found its biggest blockers remaining were:

1. parsing JSON from pypi
2. getting pypi to return less data (e.g. with a standard to retrieve only packages after a certain date in the simple repository API)
3. the subprocess execution it performs upon every startup: Enable overriding undocumented telemetry identifier with PIP_TELEMETRY_USER_AGENT_ID #13560

[cosmicexplorer]

Given that I have been unable to get any statement from pypi for years and that PEP 658 metadata took so many years to backfill again without any feedback, I don't know who I could contact to discuss e.g. a PEP to augment the Simple Repository API with the ability to filter results by upload time.

There are further approaches we could consider too, in particular filtering by complex predicates like python version, etc. But that would require pypi and other repos to support python-specific logic, which would be a step change from mostly serving cache entries. Filtering by upload time alone should be unambiguous and easy enough to codify in any server backend.

Since you mentioned PEP 658 support, and seemed particularly interested in removing functionality from pip, I wanted to emphasize that while it indeed is strange that pip has kept the fast-deps flag unchanged for so long, that is not because there has been no work on the feature. Instead, the feature has been developed into a metadata subsystem by a professional packaging engineer over several years, and neither pip nor pypi are interested.

[cosmicexplorer]

Furthermore, the metadata API from PEP 658 should also generate JSON METADATA instead of the unspecified email format which is vulnerable to text injection attacks (this can for example introduce hidden dependencies to download and install). This format was codified in PEP 566 and is available through pip install --report, standardized here: https://pip.pypa.io/en/latest/reference/installation-report/. Again, I spent several years on this work, but PEP 751 actively denies that my work exists at all and the packaging library actively refuses to implement PEP 566 (so we do it ourselves in pip).

I have not yet named my project forking pip yet. I began with string operations (url quoting, xml/html encoding, json parsing) in a rust module. I am currently implementing zstd in rust, because as PEP 777 describes, there are immense opportunities available for compression we haven't yet standardized: https://codeberg.org/cosmicexplorer/corporeal

I have been drafting a tentative specification document for a new wheel format based on the above approach, which produces very high compression ratios on practical inputs in testing. This would of course reduce the burden on pypi and on user bandwidth.

I would love to not be alone in this. Please let me know if you're interested.

[notatallshaw]

@cosmicexplorer I appreciate you have a lot of context that you want to use to back your statements, but please try and keep PR comments short and to the point. Long posts, and along series of posts, are difficult to tackle because it takes some super-linear time to review, understand, and respond to posts based on the length (or at least number of different points being brought up). Especially in PRs, where comments and reviews are ideally about the implementation, no the validity of the idea (which we should have an issue for).

I'm going to be honest, I haven't spent any time yet looking at what the core idea of fast-deps is compared to regular functionality, but so far it hasn't caused us any maintenance burden and, I think it's fair to say, this PR is very out of date. As such I'm going to close this PR, I'm sure @pradyunsg won't mind, without making any judgement on whether fast-deps should be removed or improved. Starting with a clean PR will almost certainly be easier.

I will put it on my 2026 to do list to start to understand the core concept of fast-deps and then have more informative opinions on it.