Skip to content

fix: budget retained regex by compiled source length - #92

Open
robertsLando wants to merge 1 commit into
masterfrom
fix/retained-regex-compiled-size
Open

fix: budget retained regex by compiled source length#92
robertsLando wants to merge 1 commit into
masterfrom
fix/retained-regex-compiled-size

Conversation

@robertsLando

Copy link
Copy Markdown
Member

The bug

createRetainedStreamCombi joins subscription filters into one alternation regex. The batch budget added in #89 (MAX_TOTAL_PATTERN_LENGTH) counted raw filter lengths — but what MongoDB compiles is RegExp.source, where V8 escapes every / as \/. MQTT filters are slash-delimited, so a deep topic tree inflates ~10% on top of escape-string-regexp's own expansions (- becomes \x2d, four bytes for one).

MongoDB's limit is 16384 bytes and the old budget left 1384 (9.2%) of headroom, so a slash-heavy topic tree still overshoots it. Captured from the mongod slow-query log on a production broker:

ns          mqttFreeboard.retained
branches    193 subscribe filters
raw bytes   14917   <- under the 15000 budget, so batching considered it safe
compiled    16484   <- what mongod actually parses
            -> Location51091 "pattern string is longer than the limit set by the application"

+1567 = 1341 slashes (+1 each) + 11 hyphens (+3) + 1 '$' (+1) + 192 separators

The throwing generator tears down the retained stream mid-iteration, so the subscribing client gets no retained messages for that batch or for any batch after it. It reproduces on every broker start for that deployment.

The fix

  • Budget against the compiled pattern (new RegExp(branch).source.length), never the raw filter length.
  • Filters with no wildcard skip the regex entirely and go through $in.
  • Wildcard filters become ^-anchored literal prefixes.
  • A single prefix too long to fit on its own is truncated instead of being passed through to fail (previously documented as "MongoDB will handle it or fail with a clear error"). The regex is only a pre-filter — matcher.match decides the real matches — so a shorter prefix stays correct, it just yields more candidates to discard.
  • Deduplication now applies whenever more than one filter is produced. Previously it only ran inside the batching branch, so an exact filter overlapping a wildcard one could yield the same packet twice.

Both filter shapes also keep the topic index scan bounded, where the unanchored alternation had to walk the whole index. Measured on 500 retained topics, one match:

filter keysExamined docsExamined
^-anchored prefix 2 1
unanchored alternation (before) 500 1
$in exact 4 2

Tests

retained messages batching: slash-heavy patterns stay under the MongoDB regex limit fails on master with the exact production error and passes here. The fixture asserts its own regime — raw length within the old budget, compiled source over 16384 — so it cannot silently drift out of the case it is meant to cover.

A second test covers exact, # and + filters mixed in one call, and the no-duplicates guarantee.

68/68 pass, lint clean.

MongoDB refuses a regex pattern over 16384 bytes (error 51091). The batch
budget counted raw filter lengths, but the server compiles `RegExp.source`,
where V8 escapes every `/` as `\/`. MQTT filters are slash-delimited, so a deep
topic tree inflates ~10%: a batch measured at 14917 raw bytes arrived as 16484
compiled and was rejected, killing the retained stream mid-subscribe.

Measure the compiled source instead, and keep exact filters out of the regex
entirely: they go through `$in`, wildcard filters become `^`-anchored prefixes.
Both keep the `topic` index scan bounded, where the previous unanchored
alternation walked the whole index. A prefix too long to fit on its own is now
truncated instead of passed through to fail: the regex only pre-filters,
`matcher.match` decides the real matches.

Co-authored-by: moltnet-bot <275661071+moltnet-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant