fix: budget retained regex by compiled source length - #92
Open
robertsLando wants to merge 1 commit into
Open
Conversation
MongoDB refuses a regex pattern over 16384 bytes (error 51091). The batch budget counted raw filter lengths, but the server compiles `RegExp.source`, where V8 escapes every `/` as `\/`. MQTT filters are slash-delimited, so a deep topic tree inflates ~10%: a batch measured at 14917 raw bytes arrived as 16484 compiled and was rejected, killing the retained stream mid-subscribe. Measure the compiled source instead, and keep exact filters out of the regex entirely: they go through `$in`, wildcard filters become `^`-anchored prefixes. Both keep the `topic` index scan bounded, where the previous unanchored alternation walked the whole index. A prefix too long to fit on its own is now truncated instead of passed through to fail: the regex only pre-filters, `matcher.match` decides the real matches. Co-authored-by: moltnet-bot <275661071+moltnet-bot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
createRetainedStreamCombijoins subscription filters into one alternation regex. The batch budget added in #89 (MAX_TOTAL_PATTERN_LENGTH) counted raw filter lengths — but what MongoDB compiles isRegExp.source, where V8 escapes every/as\/. MQTT filters are slash-delimited, so a deep topic tree inflates ~10% on top ofescape-string-regexp's own expansions (-becomes\x2d, four bytes for one).MongoDB's limit is 16384 bytes and the old budget left 1384 (9.2%) of headroom, so a slash-heavy topic tree still overshoots it. Captured from the mongod slow-query log on a production broker:
The throwing generator tears down the retained stream mid-iteration, so the subscribing client gets no retained messages for that batch or for any batch after it. It reproduces on every broker start for that deployment.
The fix
new RegExp(branch).source.length), never the raw filter length.$in.^-anchored literal prefixes.matcher.matchdecides the real matches — so a shorter prefix stays correct, it just yields more candidates to discard.Both filter shapes also keep the
topicindex scan bounded, where the unanchored alternation had to walk the whole index. Measured on 500 retained topics, one match:^-anchored prefix$inexactTests
retained messages batching: slash-heavy patterns stay under the MongoDB regex limitfails onmasterwith the exact production error and passes here. The fixture asserts its own regime — raw length within the old budget, compiled source over 16384 — so it cannot silently drift out of the case it is meant to cover.A second test covers exact,
#and+filters mixed in one call, and the no-duplicates guarantee.68/68 pass, lint clean.