perf(dictBuilder): skip the merge scans that cannot match

[teerthsharma]

Summary

ZDICT_tryMerge walks the whole dictItem table twice per call, and
ZDICT_insertDictItem calls it once per candidate plus once per round of its
merge fixpoint. Both loops return at the first match, so a loop only runs to
completion when nothing matches: the quadratic term is spent proving absence.

Each dictItem is an interval in the sample buffer, and the merge fixpoint keeps
those intervals pairwise disjoint. This change adds three counters that make the
absence proof O(1):

• span[p] low nibble, the number of dictItems whose interval holds p,
  answers the front positional predicate
• span[p] high nibble, the number of dictItems beginning at p, answers the
  tail predicate
• prefix[h], the number of dictItems whose 8-byte prefix hashes to h,
  answers the content predicate

Disjointness bounds the overlap depth, which is what allows nibble-width
counters to track coverage exactly. The counters only ever guard a decision to
skip a loop and are conservative: they never report absence when a match exists.
Every merge is still selected by the original predicates.

Two further consequences of the same structure:

• the content test is skipped when no dictItem can carry the tested key, which
  removes a random read into the sample buffer per entry visited;
• when exactly one dictItem spans elt.pos and no content match is possible,
  the positional test has a single solution, so the loop returns the same entry
  whichever direction it is walked. It is walked downward there, which reaches
  the entry sooner because a merge partner has just been inserted and still sits
  at the low-savings end of the table.

dictItem gains a cached hash of the 8 bytes at its position so the content
pre-test reads the table entry already being walked rather than the sample
buffer. The exact MEM_read64 comparison still decides every merge.

There is no public API change, no dictionary format change, and no change to
the ranking heuristic, the savings computation, ZDICT_analyzePos, or the
cover/fastcover trainers. Only ZDICT_trainFromBuffer_legacy and the CLI's
--train-legacy reach the modified code.

Built on #4724, which establishes the pairwise disjointness this relies on.

Byte-identity

Six corpora, -O3 -DNDEBUG, dictionary capacity 112,640 B,
selectivityLevel 9. The checksum is over the full emitted dictionary.

corpus	bytes	base	this change
zstd lib/*.c	2,375,034	94e7afcb	94e7afcb
zstd lib/*.h	858,183	a29ac6f4	a29ac6f4
zstd programs/*	468,328	9948b1d1	9948b1d1
zstd tests/ + doc/	1,207,601	1a95763c	1a95763c
python sources	2,067,845	b14b4cd4	b14b4cd4
rust sources	2,460,916	3c9698f4	3c9698f4

6 of 6 identical, 0 differing. End to end through the CLI,
zstd --train-legacy over 49 headers emits a 55,860 B dictionary on both trees.

Instruction counts

Callgrind, deterministic and identical between runs, -O2 -fno-inline -g so
that per-function attribution is exact. gcc 15.2.0, Ubuntu 26.04 LTS under
WSL2. Base is topo/zdict-h0-interval-merge at fc2b6a358, so the measurement
attributes only this change.

corpus	base Ir	this change	whole run	base tryMerge	this change	tryMerge
zstd lib/*.c	4,239,867,437	2,421,375,251	1.751x	2,204,100,449	642,779,694	3.429x
zstd lib/*.h	1,313,475,337	604,626,894	2.172x	849,993,823	238,932,154	3.557x
zstd programs/*	454,313,794	273,956,640	1.658x	213,642,180	56,398,520	3.788x
zstd tests/+doc/	2,185,612,807	1,324,003,263	1.651x	1,001,029,699	261,314,415	3.831x
python sources	6,261,690,578	2,356,302,683	2.657x	4,475,868,941	1,146,505,852	3.904x
rust sources	5,940,161,178	4,287,058,981	1.386x	1,977,757,434	555,226,008	3.562x

On an -O3 -DNDEBUG build the whole-run ratios are 1.660x, 1.971x, 1.585x,
1.596x, 2.382x and 1.361x.

The counters cost 16,300,000 Ir out of 2,952,302,176, or 0.55% of a training
run.

Validation

From PR head 8e1109a22, built with the project's own test flags:

check	result
tests/fuzzer -i3000	3000/3000 completed, exit 0
tests/zstreamtest -i1000	1001/1001 completed, exit 0
zstd --train-legacy over 49 headers, then compress and decompress each against the resulting dictionary	49/49 byte-identical

lib/dictBuilder/zdict.c compiles with no new diagnostics under
-Wall -Wextra -Wconversion -Wcast-qual -Wstrict-prototypes -Wdeclaration-after-statement.

Rejected alternatives

approach	measured	why it was dropped
fixed 32,768-slot prefix table	1.498x / 1.723x / 1.455x	undersized against dictListSize; scaling it to dictListSize * 64 moved the ratio by under 0.03x, showing collisions were not the residual
masking the raw low bits of the 8-byte read as the hash	—	collapses on text, where the first two bytes carry little entropy
a covers bitmap used for the tail predicate as well	34% of visits removed	the tail predicate needs starts, not spans; separating them took it to 68%
selecting walk direction with u = down ? tableSize-i : i	1.302x / 1.435x / 1.318x	destroyed the loop's induction variable and cost more than the iterations it saved; re-expressed as its own loop instead

Limits

The whole-run ratio varies because the merge phase is a variable share of
training. On the rust corpus ZDICT_analyzePos is 40.83% and ZDICT_count
20.87% of the run, so removing ZDICT_tryMerge entirely would still cap that
corpus at 1.73x; that phase is a different algorithm and is untouched here. The
change adds one byte per sample byte for the span counters, the same footprint
as doneMarks, plus a prefix table sized from dictListSize. dictItem grows
from 12 to 16 bytes. All measurements come from a single machine, and no 32-bit
target was exercised.

Fixes #4725.