Skip to content

utils/download: raise a clear error instead of crashing on missing ETag - #774

Open
mykaul wants to merge 1 commit into
scylladb:masterfrom
mykaul:fix-download-etag-fallback
Open

utils/download: raise a clear error instead of crashing on missing ETag#774
mykaul wants to merge 1 commit into
scylladb:masterfrom
mykaul:fix-download-etag-fallback

Conversation

@mykaul

@mykaul mykaul commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Motivation

get_url_hash() tries an S3 HeadObject first and falls back to a plain
HTTP HEAD request if that fails. Both paths did .get('ETag')[1:-1]
unconditionally, so a response with no ETag header crashed with a
confusing TypeError: 'NoneType' object is not subscriptable instead of
surfacing what actually went wrong. In practice this hid the real cause:
an S3 403 (access denied) is swallowed by the bare
except botocore.client.ClientError, execution falls through to the HTTP
fallback, that also has no ETag, and the resulting TypeError gives no
indication that the original problem was an S3 permissions/access issue.
This was found while investigating CI failures on two unrelated PRs
(#772, #773) that were both blocked by this exact crash pattern.

Change

  • Both the S3 and HTTP-fallback paths now check for a missing ETag
    and raise a clear RuntimeError naming the URL, instead of crashing
    with an unrelated TypeError.
  • When the HTTP fallback also fails, the error chains in the original
    S3 exception (raise ... from e), so the real root cause (e.g. a 403)
    is visible instead of being silently discarded.

Test

  • New tests in tests/test_caching_hash.py:
    • test_get_url_hash_raises_clear_error_when_http_fallback_has_no_etag:
      S3 fails (403) and the HTTP fallback has no ETag → clear
      RuntimeError mentioning both the S3 failure and the missing ETag.
    • test_get_url_hash_raises_clear_error_when_s3_has_no_etag: S3
      succeeds but returns no ETag → clear RuntimeError instead of an
      uncaught TypeError.
  • All existing tests in tests/test_caching_hash.py still pass (25
    passed, 3 skipped).
  • Confirmed no regressions: the same 7 pre-existing, network-dependent
    failures in tests/test_scylla_repository.py (real S3 calls hitting
    live data, unrelated to this change) occur identically on unmodified
    origin/master.

get_url_hash() tries S3 HeadObject first and falls back to a plain HTTP
HEAD request if that fails (e.g. access denied). Both paths did
`.get('ETag')[1:-1]` unconditionally, so a response with no ETag header
crashed with a confusing `TypeError: 'NoneType' object is not
subscriptable` instead of surfacing what actually went wrong -- e.g. an
S3 403 whose real error message was already being swallowed by the
bare `except botocore.client.ClientError` before falling through to a
broken HTTP fallback.

Raise a RuntimeError naming the URL and, when the HTTP fallback also
fails, chain in the original S3 error too.

Signed-off-by: Yaniv Michael Kaul <yaniv.kaul@scylladb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant