utils/download: raise a clear error instead of crashing on missing ETag - #774
Open
mykaul wants to merge 1 commit into
Open
utils/download: raise a clear error instead of crashing on missing ETag#774mykaul wants to merge 1 commit into
mykaul wants to merge 1 commit into
Conversation
get_url_hash() tries S3 HeadObject first and falls back to a plain HTTP
HEAD request if that fails (e.g. access denied). Both paths did
`.get('ETag')[1:-1]` unconditionally, so a response with no ETag header
crashed with a confusing `TypeError: 'NoneType' object is not
subscriptable` instead of surfacing what actually went wrong -- e.g. an
S3 403 whose real error message was already being swallowed by the
bare `except botocore.client.ClientError` before falling through to a
broken HTTP fallback.
Raise a RuntimeError naming the URL and, when the HTTP fallback also
fails, chain in the original S3 error too.
Signed-off-by: Yaniv Michael Kaul <yaniv.kaul@scylladb.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
get_url_hash()tries an S3HeadObjectfirst and falls back to a plainHTTP
HEADrequest if that fails. Both paths did.get('ETag')[1:-1]unconditionally, so a response with no
ETagheader crashed with aconfusing
TypeError: 'NoneType' object is not subscriptableinstead ofsurfacing what actually went wrong. In practice this hid the real cause:
an S3 403 (access denied) is swallowed by the bare
except botocore.client.ClientError, execution falls through to the HTTPfallback, that also has no
ETag, and the resultingTypeErrorgives noindication that the original problem was an S3 permissions/access issue.
This was found while investigating CI failures on two unrelated PRs
(#772, #773) that were both blocked by this exact crash pattern.
Change
ETagand raise a clear
RuntimeErrornaming the URL, instead of crashingwith an unrelated
TypeError.S3 exception (
raise ... from e), so the real root cause (e.g. a 403)is visible instead of being silently discarded.
Test
tests/test_caching_hash.py:test_get_url_hash_raises_clear_error_when_http_fallback_has_no_etag:S3 fails (403) and the HTTP fallback has no
ETag→ clearRuntimeErrormentioning both the S3 failure and the missing ETag.test_get_url_hash_raises_clear_error_when_s3_has_no_etag: S3succeeds but returns no
ETag→ clearRuntimeErrorinstead of anuncaught
TypeError.tests/test_caching_hash.pystill pass (25passed, 3 skipped).
failures in
tests/test_scylla_repository.py(real S3 calls hittinglive data, unrelated to this change) occur identically on unmodified
origin/master.