Cap worker-lock retry interval to fix int-str overflow crash (fixes #19315 #18221)#19726
Open
George3d6 wants to merge 3 commits intoelement-hq:developfrom
Open
Cap worker-lock retry interval to fix int-str overflow crash (fixes #19315 #18221)#19726George3d6 wants to merge 3 commits intoelement-hq:developfrom
George3d6 wants to merge 3 commits intoelement-hq:developfrom
Conversation
…rowth that trips python 3.11+s 4300-digit str() limit (fixes#19315, element-hq#18221)
|
|
|
|
||
|
|
||
| logger = logging.getLogger(__name__) | ||
| _LOCK_RETRY_CAP_SECS = Duration(minutes=10).as_secs() |
Contributor
There was a problem hiding this comment.
There is already a PR to address this kind of thing: #19394
🤔 I think we can continue with the other PR which is a bit more fleshed out and is currently stuck at the stage of the review being addressed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cap
WaitingLock/WaitingMultiLockretry interval at 10 minutes.Without a cap,
max(5, next * 2)silently promotesself._retry_intervalto aninton the second call (Python'smaxpreserves the larger operand's type, and5is an int), and it doubles without bound. Once it crosses the existing 10-minute "getting excessive" warning threshold, the warning log line callsstr()on the int and eventually trips Python 3.11+'s 4300-digitint_max_str_digitslimit (PEP 644), raisingValueErrorfrom inside the warning itself.__aenter__'s broadexcept Exception: pass(intended to absorbTimeoutErrorfromtimeout_deferred) swallows theValueErrorbefore the retry loop sleeps, so the loop re-raises as fast as Python can emit tracebacks until the process is restarted.The cap is hoisted into a module-level
_LOCK_RETRY_CAP_SECSso both classes share it, and the excessive-timeout warning is restructured to fire exactly once per waiter (on the iteration that first reaches the cap) instead of either spamming every iteration or being silently suppressed.Fixes #19315 and #18221.