Stream objects are re-parsed on every `Get`, allocating the whole stream each time

[KHDHaDi]

Stream objects are re-parsed on every Get, allocating the whole stream each time

Summary

PdfTokenScanner.Get(IndirectReference) caches resolved objects — except stream
objects, which are deliberately excluded:

// We don't cache StreamToken as this would keep
// the attached raw bytes (can be large)
if (found.Data is not StreamToken)
{
    objectLocationProvider.Cache(found);
}

The reasoning is sound, but the current cost is that every single resolution of a
stream object seeks, re-tokenises and copies the whole stream again. Measured
below: allocation per resolution equals the size of the stream.

For consumers that visit the same stream repeatedly — content streams, embedded
font programs, ICC profiles, CMaps, XMP packets — this dominates. Resolving one
75 KB font program 500 times allocates 36.8 MB.

I am not proposing to simply drop the exclusion; retaining raw bytes for every
stream in a large document would trade one problem for another. This is a request
to make the trade-off adjustable rather than fixed at "never".

What is already fixed in master

d67d792 ("Improve caching: found tokens in token scanner, resolved resources and
XObject forms", 2026-08-18) added the Cache(found) call shown above. In the
released v0.1.15 the successful lookup path calls Cache not at all, so
non-stream objects are re-parsed too:

// v0.1.15
var found = (ObjectToken)CurrentToken!;

if (found.Number.Equals(reference))
{
    return found;
}

So half of this is already solved and only awaiting a release. This issue is about
the half that master keeps: StreamToken.

Reproduction

Works against any PDF that contains a reasonably large stream object. The program
picks the largest one it can reach, then resolves it repeatedly.

using System.Diagnostics;
using UglyToad.PdfPig;
using UglyToad.PdfPig.Core;
using UglyToad.PdfPig.Tokens;

using var document = PdfDocument.Open(args[0], new ParsingOptions { UseLenientParsing = true });
var scanner = document.Structure.TokenScanner;

IndirectReference? largest = null;
var size = 0;
for (var number = 1; number < 400; number++)
{
    var reference = new IndirectReference(number, 0);
    ObjectToken? token;
    try { token = scanner.Get(reference); } catch { continue; }
    if (token?.Data is StreamToken stream && stream.Data.Length > size)
    {
        size = stream.Data.Length;
        largest = reference;
    }
}

Console.WriteLine($"largest stream: {largest}, {size:N0} raw bytes");

// Two consecutive resolutions of the same reference.
Console.WriteLine(ReferenceEquals(scanner.Get(largest!.Value), scanner.Get(largest.Value))
    ? "cached"
    : "re-parsed");

const int Rounds = 500;
var before = GC.GetAllocatedBytesForCurrentThread();
var watch = Stopwatch.StartNew();
for (var i = 0; i < Rounds; i++)
{
    _ = scanner.Get(largest.Value);
}
watch.Stop();
var allocated = GC.GetAllocatedBytesForCurrentThread() - before;

Console.WriteLine($"{Rounds} resolutions: {watch.Elapsed.TotalMilliseconds:N0} ms, "
    + $"{allocated / 1024d / 1024d:N1} MB");
Console.WriteLine($"per resolution: {allocated / (double)Rounds / 1024:N1} KB, "
    + $"{watch.Elapsed.TotalMicroseconds / Rounds:N0} us");

Measurements

PdfPig 0.1.15, .NET 10, Release, Windows 11, on a PDF 1.4 file with a classic
cross-reference table (no object streams involved).

Largest stream, an embedded CID-keyed CFF font program:

largest stream: 11 0, 75,501 raw bytes
re-parsed
500 resolutions: 9 ms, 36.8 MB
per resolution: 75.3 KB, 18 us

Per-reference, same document:

Reference	cached	Type	Bytes / resolution	us / resolution
7 0	no	DictionaryToken	2,160	5.1
6 0	no	StreamToken	4,216	6.8
9 0	no	StreamToken	2,480	6.0

The DictionaryToken row is the part d67d792 addresses; the StreamToken rows are
not, and the allocation there tracks the stream size rather than any fixed
overhead.

Why this matters

Any analysis pass that walks a document more than once resolves the same stream
objects again and again: a font program is consulted per glyph requirement, a
content stream per page visit, an ICC profile per colour space that names it. The
work is invisible to the caller — the API looks like a lookup, so callers treat it
as one and loop over it.

Callers can work around it with a cache of their own, keyed by
IndirectReference. That is what we ended up doing, and it is measurable: across
our validation workload it moved the medium test document from 10.69 ms to 1.01 ms
and from 8.0 MB to 262 KB of allocation. But every consumer with this access
pattern has to discover the problem and rebuild the same thing, and a cache
outside the library cannot know when PdfPig itself considers an object stale.

Possible directions

Any of these would solve it; the choice is yours.

1. A byte budget. Cache stream tokens too, but track the retained raw bytes and
   stop caching (or evict) once a configurable ceiling is reached. Bounded memory,
   no configuration needed for the common case.
2. A size threshold. Cache streams below n bytes, never above. Simplest, and
   it already covers the many-small-streams case that hurts most.
3. Opt-in via ParsingOptions. Something like CacheStreamObjects, defaulting
   to today's behaviour. Analysis workloads that reread the same document opt in;
   one-pass extraction is unaffected.
4. Cache the token without the bytes. Keep the StreamToken's dictionary and
   the offset, re-read the raw bytes on demand. Saves the re-tokenisation without
   retaining anything large — though it is the most invasive of the four.

Happy to prepare a pull request for whichever direction you prefer.

Environment

• PdfPig 0.1.15 (latest release; measurements above)
• master read at d67d792 for the code quoted from it
• .NET 10, Release, Windows 11