Use the new XetSession API

[seanses]

Summary

Migrates all xet upload/download code from the old function-based hf_xet API (upload_files, download_files, global MULTITHREADED_RUNTIME) to the new session-based XetSession API introduced in hf-xet >= 1.5.0.

Upload:

session = get_xet_session()  # global singleton, fork- and SIGINT-safe

# Strip auth from custom_headers; keep full headers for token refresh
xet_headers = headers.copy()
xet_headers.pop("authorization", None)

with session.new_upload_commit(
    token_refresh_url=refresh_url,    # URL to refresh the xet write token
    token_refresh_headers=headers,    # full headers (incl. auth) for token refresh
    custom_headers=xet_headers,       # headers without auth for xet storage requests
    progress_callback=on_progress,    # optional; receives (group_report, item_reports)
) as commit:
    commit.start_upload_file(path, sha256=sha256_hex)   # from disk
    commit.start_upload_bytes(data, sha256=sha256_hex)  # from memory
# commit() is called automatically on clean exit of the with-block

Download:

with session.new_file_download_group(
    token_refresh_url=xet_file_data.refresh_route,
    token_refresh_headers=headers,
    custom_headers=xet_headers,
    progress_callback=on_progress,
) as group:
    group.start_download_file(XetFileInfo(file_hash, size), dest_path)
# finish() is called automatically on clean exit of the with-block

Architecture:

• utils/_xet.py — XetSessionHolder is a lightweight wrapper around XetSession with three safety guarantees:
  • SIGINT handling: sigint_abort() cancels the active session and clears the reference so the next get() creates a fresh one (notebook-friendly — subsequent cells still work after Ctrl-C)
  • Fork safety: get() checks os.getpid() against the PID that created the session; on mismatch the inherited session is discarded and a fresh one is created for the child process (prevents deadlock when multiprocessing workers inherit a dead Tokio runtime)
  • Thread safety: a threading.Lock guards all state mutations, important for free-threaded Python (3.14t) where multiple threads can race on get() or sigint_abort() without the GIL serialising them
• _GLOBAL_XET_HOLDER = XetSessionHolder() is a module-level singleton (analogous to get_session() for HTTP). All upload/download call sites share the same session via get_xet_session() — repo commits, hf_hub_download, bucket uploads/downloads, and snapshot_download all reuse the same underlying Tokio runtime
• All blocking call sites are wrapped with except KeyboardInterrupt: _GLOBAL_XET_HOLDER.sigint_abort(); raise to abort the in-flight Rust operation and allow clean re-use of xet in the same process
• xet_connection_info_refresh_url() centralises token-refresh URL construction: handles the bucket repo-type (revision omitted) and the /None revision for PR write-token requests (where the final revision is not yet known)
• _xet_progress_reporting.py — progress callback receives (group_report, item_reports) matching the new API contract
• setup.py — minimum hf-xet version bumped to >= 1.5.0
• Tests mock hf_xet.XetSession; fork-safety unit and integration tests added to test_xet_utils.py

Note on upload_large_folder and Ctrl-C: xet work there runs inside background worker threads. Python only delivers KeyboardInterrupt to the main thread, and PyErr_CheckSignals() is a no-op in non-main threads — so Ctrl-C responsiveness for upload_large_folder requires a separate fix (shared stop event + worker cooperation) and is out of scope for this PR.

TODO

• Refactor upload_large_folder to use the XetSession API directly: register each file to the upload commit object immediately after SHA-256 computation (no batching needed — xet handles concurrency internally), then call commit() once at the end. This removes the artificial batch size tuning (UPLOAD_BATCH_SIZE_XET) and makes the xet path in upload_large_folder consistent with the rest of the codebase.

---

Note

Medium Risk
Touches the primary large-file transfer path across commits, hub downloads, and buckets with a dependency floor bump; behavior changes around token refresh, KeyboardInterrupt, and forked workers warrant careful regression testing.

Overview
Migrates all Hub Xet uploads and downloads from the legacy hf_xet helpers (upload_files, upload_bytes, download_files) to the XetSession API (new_upload_commit, new_file_download_group), with hf-xet bumped to >=1.5.1.

A shared get_xet_session() singleton in utils/_xet.py (fork-safe, thread-safe, abort_xet_session() on Ctrl-C) backs repo commits (_upload_xet_files), xet_get / hf_hub_download, and bucket batch upload/download. Auth is now token_refresh_url + token_refresh_headers instead of pre-fetching endpoints/tokens and passing token_refresher callbacks; xet_connection_info_refresh_url() centralizes refresh URL building.

XetProgressReporter follows the new (group_report, item_reports) progress contract; XetAuthorizationError / XetRefreshTokenError are removed. Tests mock XetSession and add fork-safety coverage for XetSessionHolder.

^{Reviewed by Cursor Bugbot for commit 44aec7a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[codecov]

Codecov Report

❌ Patch coverage is 80.27211% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.40%. Comparing base (1daa48b) to head (44aec7a).
⚠️ Report is 408 commits behind head on main.

Files with missing lines	Patch %	Lines
src/huggingface_hub/utils/_xet.py	75.60%	10 Missing ⚠️
src/huggingface_hub/hf_api.py	82.97%	8 Missing ⚠️
...c/huggingface_hub/utils/_xet_progress_reporting.py	78.26%	5 Missing ⚠️
src/huggingface_hub/_commit_api.py	85.71%	3 Missing ⚠️
src/huggingface_hub/file_download.py	80.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4116      +/-   ##
==========================================
+ Coverage   75.00%   76.40%   +1.39%     
==========================================
  Files         145      173      +28     
  Lines       13978    20349    +6371     
==========================================
+ Hits        10484    15547    +5063     
- Misses       3494     4802    +1308     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Wauplin]

Thanks for working on that @seanses! I've left some high level comments on the API design. Overall a very nice addition!

[Wauplin]

This looks very solid now! Extensively tested it locally (mainly the CLI) and works as smoothly as before 👍

[Wauplin]

I've asked Claude code to review the PR and it commented a few things. Nothing critical IMO

---

3. XetAuthorizationError / XetRefreshTokenError are now dead code

Files: src/huggingface_hub/errors.py:453-457, removed imports in _commit_api.py and hf_api.py

The old code raised XetAuthorizationError on 401 responses and XetRefreshTokenError on failed token refresh. The new session-based API delegates auth to hf_xet (which handles token refresh internally via the URL), so these exceptions are no longer raised anywhere.

They're still defined in errors.py but never imported or used. They're not in __init__.py so they're not public API.

Suggestion: Delete XetAuthorizationError, XetRefreshTokenError, and possibly XetError from errors.py if nothing else uses them, or add a TODO comment. Dead error classes are confusing for future readers.

My opinion (Wauplin): fine to remove them and mention the breaking change in the PR description (it's a breaking change but I don't expect downstream users to have actually raised from it)

---

5. No unit tests for xet_connection_info_refresh_url

This is a new public-ish function with non-trivial logic (bucket vs git-based repo types, the "/None" revision for PR tokens). There are no dedicated tests for it. A few parametrized test cases would be cheap and valuable:

• repo_type="model", revision="main" → .../models/{id}/xet-write-token/main
• repo_type="model", revision=None → .../models/{id}/xet-write-token/None
• repo_type="bucket", revision=None → .../buckets/{id}/xet-write-token (no revision)
• repo_type="bucket", revision="some-rev" → .../buckets/{id}/xet-write-token/some-rev

My opinion (Wauplin): quick and easy to test so let's do it (even if not introduced in this PR)

---

6. progress.close(False) always passes False — is _success meaningful?

Files: _commit_api.py:632, hf_api.py:13193

XetProgressReporter.close(_success) is always called with False (in the finally block), even on successful completion. The _success parameter is currently unused in close() — it just closes bars regardless. If _success is never meaningful, the parameter should be removed. If it's intended for future use, the call site should pass the correct value.

This existed before this PR but worth flagging since the progress reporter was rewritten.

My opinion (Wauplin): let's keep it as-is except if you really care about it^^

---

8. Header stripping pattern repeated

The xet_headers = headers.copy(); xet_headers.pop("authorization", None) pattern appears in 4 call sites:

• _commit_api.py:602-603
• file_download.py:537-538
• hf_api.py:13151-13152
• hf_api.py:13397-13398

Consider a small helper like _strip_auth_headers(headers) if you want, but it's fine as-is — it's 2 lines and very clear.

My opinion (Wauplin): why not but low prio

[seanses]

Is this expected? The xet_connection_info_refresh_url function logic was entirely extracted from the previous fetch_xet_connection_info_from_repo_info function, and the logic implies this. But the comment # On "bucket" repo type, the revision never needed => don't use it suggests otherwise. @Wauplin

[hanouticelina]

thanks a lot @seanses for working on this! and very sorry for the delay. I left a couple of comments and tested the PR as well (with with hf_xet==1.5.0),
let me know if I'm missing something.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 910d691. Configure here.}

[hanouticelina]

Looks good to me, I tested the PR against a locally built wheel from xet-core PR huggingface/xet-core#856 and all good on my side, thanks a lot @seanses for working on this!
I'll let @Wauplin take a look and give the green light

[Wauplin]

Made another pass on the PR. I think we are very close to getting this merged (once hf_xet==1.5.1 gets released).

About the progress bars I like the idea of reporting both the "download" and "reconstruction" progress for better UX. Left 2 comments about it but I think it can be addressed in a separate PR. Let me know what you think

[Wauplin]

Pre-approving, only remaining item is bumping hf_xet version in setup.py 🎉

[Wauplin]

🎉

[huggingface-hub-bot]

This PR has been shipped as part of the v1.19.0 release.