Super-speed caching for static module directory trees

[smprather]

Hi @xdelaruelle --

After a lot of back-n-forth with my GPT-friend, I would like to propose a new opt-in feature: a user commitment to a static module dir tree (no virtual modules, .modulerc dynamic behaviour changes, etc), which gets you big performance gains (>100x). As you know by now, I have a script that generates 15k (15k today, more coming soon) modules with no dynamic features. And no problem taking an arbitrarily long up-front cache build time.

Do you agree this is something you can accept? The basic contract is that user gives up dynamic features for big performance.

We can talk about what to do if the contract is broken. Fallback to old? Hard fail? User chooses?

In the next comment, I'll copy/paste the full chat log for more details.

Example config names GPT threw out.

  1. Add an explicit opt-in mode
     Something like a config knob:
  - static_cache
  - frozen_modulepath
  - trust_cache_fully

Example of what we have achieved so far:

Benchmarks, clean baseline modules-pr613 vs new worktree modules-avail-opt:
  - full module avail --terse >/dev/null 2>&1
      - before: 5.991 s ± 0.157 s
      - after: 105.9 ms ± 1.4 ms
  - module avail --terse vendor1 >/dev/null 2>&1
      - before: 3.750 s ± 0.086 s
      - after: 203.7 ms ± 0.8 ms
  - module avail --terse vendor1/product1 >/dev/null 2>&1
      - before: 3.367 s ± 0.086 s
      - after: 142.7 ms ± 3.1 ms

  Sanity checks:
  - module avail --redirect --terse vendor1 | wc -l -> 50001
  - module avail --redirect --terse vendor1/product1 | wc -l -> 10001
  - module avail --redirect --terse vendor1/product1/0000* | wc -l -> 10

[smprather]

This is a chat-log in which I asked for module avail performance improvement.
Consider this an Appendix. Skip to the next comment.

Codex in normal. My prompts in bold italics.

---

<snip...>

Code changed there:

• modules-avail-opt/Makefile:98
• modules-avail-opt/tcl/cache.tcl.in:23
• modules-avail-opt/tcl/subcmd.tcl.in:1899

Benchmarks, clean baseline modules-pr613 vs new worktree modules-avail-opt:

• full module avail --terse >/dev/null 2>&1
  • before: 5.991 s ± 0.157 s
  • after: 105.9 ms ± 1.4 ms
• module avail --terse vendor1 >/dev/null 2>&1
  • before: 3.750 s ± 0.086 s
  • after: 203.7 ms ± 0.8 ms
• module avail --terse vendor1/product1 >/dev/null 2>&1
  • before: 3.367 s ± 0.086 s
  • after: 142.7 ms ± 3.1 ms

Sanity checks:

• module avail --redirect --terse vendor1 | wc -l -> 50001
• module avail --redirect --terse vendor1/product1 | wc -l -> 10001
• module avail --redirect --terse vendor1/product1/0000* | wc -l -> 10

Important caveat:

• this fast path is still avail-specific
• it is not per-query caching
• it currently only activates when the modulepath is simple enough to precompute safely from names alone: plain modulefiles, no extra alias/tag/variant metadata to report
• otherwise it falls back to normal Modules logic

That is probably the right intermediate shape. Next move: port this from modules-avail-opt into modules-main, then widen the index so it can cover richer real trees without dropping to fallback.

› just curious. all of your improvements are cache based. is all the juice squeezed out of the cacheless (or cache-as-it-is-today) code? is there anything in the tcl codebase to speed up this particular hot loop?

• I’m checking a few concrete hot spots before I answer. The big question is whether the normal Tcl path is fundamentally doing too much work per entry, or whether it’s just missing a better index.

• I have enough. Short answer is: no, the non-indexed path is not “fully squeezed.” There are still real gains in the Tcl path, but they’re smaller and riskier than building a better index. I’m pulling one more code slice so I can point to the exact place where today’s cache still does too much work.

• No. There is still headroom in the normal Tcl path.

But the important distinction is:

• cacheless: still fundamentally expensive
• cache as it exists today: still doing too much work after loading the cache
• indexed cache: where the big wins come from

What I think from reading the code and timing it:

module avail today is not slow because puts is slow. I checked that indirectly: sending output to /dev/null barely changed runtime. The time is mostly CPU spent building and post-processing the result set.

The main hot spots are visible in the current code:

1. Full-tree walk and per-entry filesystem/string work

• modules-pr613/tcl/modfind.tcl.in:2957 findModulesFromDirsAndFiles
• modules-pr613/tcl/modfind.tcl.in:3055 findModules
• modules-pr613/tcl/modfind.tcl.in:2901 __getFilesInDirectory

This path does a lot of:

• glob twice per directory
• file tail
• file dirname
• file split
• file isdirectory
• checkValidModule
• getFileMtime

Across 100k files, that adds up fast.

2. Today’s .modulecache is not really an index

• modules-pr613/tcl/cache.tcl.in:377 findModulesInCacheFile

It still:

• loads the whole cached modulepath into arrays
• iterates array names cache_arr
• recomputes file split, file tail, file dirname
• filters after materializing everything

That means the current cache avoids filesystem reads, but it does not avoid the “touch the whole universe” shape.

3. Output prep is still heavy per module

• modules-pr613/tcl/report.tcl.in:1438 reportModules

For every entry it may call:

• getVariantList
• getVersAliasList
• getTagList
• abbrevTagList
• formatListEltToDisplay

Then it sorts the full display list:

• modules-pr613/tcl/report.tcl.in:1595

On a simple tree like your mock modules, most of that metadata work is wasted.

4. Query fanout is still broad

• modules-pr613/tcl/modfind.tcl.in:3659 getMatchingAnyModules

Even for module avail vendor1/product1, the normal path still builds a large candidate set and filters it late.

So, is all the juice squeezed out of cacheless / current-cache code?

No.

What is still available without “cache exact query answers”:

• Better filtering structure inside the existing cache:
  bucket by prefix / directory / root name instead of replaying the whole cache and filtering afterward.
• Delay metadata work:
  do not call tag / variant / alias helpers unless output config actually needs them.
• Avoid repeated path parsing:
  stop doing file split / file dirname / file tail in the innermost loops.
• Avoid resorting giant lists when traversal/index order is already usable.
• Push more scan logic into libtclenvmodules.so for the cold path.

What I do not believe:

• I do not think the cacheless path can be made truly fast for 100k modules just by polishing Tcl code.
• If each query still has to walk or reconstruct the whole tree, you are fighting the algorithm, not just interpreter overhead.

My practical read:

• cacheless: maybe there is a meaningful 20-50% left, possibly more with C help, but not an order-of-magnitude miracle.
• cache-as-it-is-today: yes, there is much more than 20% left, because it is still not an actual search index.
• Real big wins come from changing the shape of the work: preindexed names/prefixes, then cheap filtering.

If you want, next step is I can try a non-cheaty middle ground in the real codebase:

• no pre-rendered exact answers
• no one-cache-per-command
• just improve the existing .modulecache path so prefix queries like avail foo and avail foo/bar stop replaying the entire module universe.

› well, i think, and correct me if i'm wrong, that "replay the module universe" thing is triggered by things like virtual (dynamic) modules. BUT, by creating a cache, aren't we signing a contract to not be dynamic? now that i think about it, what i'm really asking for is a "i agree to be 100% static, and agree to pay any up-front caching time, if you give me big performance improvement". as long as we create that contract for signing, and have a way to sign it, then we should be able to pass all tests, including virtual module testing by not using cache improvements in that case. sound correct?

• Yes. That is the right framing.

You are basically defining a stronger cache contract:

• tree is static between cache rebuilds
• module visibility/shape is static between cache rebuilds
• no expectation that dynamic module behavior is reflected live
• user accepts upfront cache build cost for much lower query cost

Under that contract, much more becomes legal.

And yes, that should let us keep upstream semantics intact by making it opt-in. In other words:

• default behavior:
  current semantics, dynamic-safe, all existing tests still relevant
• static-fast behavior:
  stronger cache mode, only enabled when explicitly configured

That is the clean way to make both true.

Your key point is correct:
current Modules behavior keeps “replaying the universe” partly because some semantics are computed at query time, including things like:

• modulerc / .version effects
• hidden/forbidden state
• aliases / virtual modules
• variant metadata
• anything influenced by runtime state or evaluation-time logic

If you declare “I do not need live dynamism,” then the cache can become an actual index or even a precomputed view.

That suggests a good design:

1. Add an explicit opt-in mode
   Something like a config knob:

• static_cache
• frozen_modulepath
• trust_cache_fully

2. Define exactly what it means
   For example:

• module tree contents are authoritative from cache
• no filesystem walk on avail / show lookup paths when cache exists
• no live reevaluation for aliases/virtuals unless encoded into cache
• cache rebuild required after any tree change

3. Keep it narrow at first
   Start with:

• avail
• exact module resolution for show / load by fully qualified name

Those are your biggest wins.

4. Make fallback explicit
   If static-fast mode is off:

• current behavior
  If on and cache missing/incompatible:
• either hard fail with “rebuild cache”
• or fall back, depending on how strict you want it

For your use case, hard fail is better. It preserves the contract.

5. Tests
   Yes, this is how you pass the normalYes. That is basically correct.

Current Modules behavior treats cache as:

• an acceleration artifact
• but not a semantic narrowing of what the module tree can do

So even with .modulecache, the runtime still preserves dynamic features:

• modulerc effects
• aliases / virtuals
• hidden / forbidden logic
• variants
• metadata-sensitive output
• limited-access handling
• anything that can change meaning without just “the file exists”

That is why today’s cache still “replays the universe” after loading it. It is not a strict static index contract.

Your proposed model is different and much cleaner:

• user explicitly opts into a static-tree mode
• cache build becomes authoritative for selected operations
• runtime is allowed to skip dynamic semantics outside that contract
• tests still pass because default behavior stays unchanged
• static fast paths only activate when the contract is enabled

That is the right shape if your real goal is upstreamable performance without breaking general Modules semantics.

What this contract would mean technically:

1. Opt-in switch
   Something like:

• config option
• env var
• cachebuild flag
• modulepath marker

Example shape:

• module cachebuild --static-index
• writes extra static index artifacts
• runtime only trusts them if config says static mode is enabled

2. Semantic narrowing
   In static mode, you explicitly give up:

• dynamic virtuals/aliases beyond what was captured at cachebuild time
• live modulerc reevaluation
• runtime-sensitive hidden/forbidden changes
• tree mutation without cache rebuild

3. Runtime payoff
   Then we can safely do:

• exact-name index for show
• prefix index for avail foo
• pre-sorted terse display lists
• directory/root buckets
• maybe even direct binary or dict-backed indexes

4. Test story
   Default mode:

• unchanged
• all existing semantics and tests apply

Static mode:

• new targeted tests
• existing dynamic tests should continue to run in default mode only
• performance features are gated, so they do not disturb virtual-module behavior unless explicitly enabled

So yes: if the contract is explicit, the semantic risk becomes manageable.

The only correction I’d make is this:
creating a cache alone does not currently mean “I agree to be static.” That is your proposed new contract, not the current one. And I think that distinction is exactly what makes an upstreamable design
possible.

Best upstream framing would be:

• keep current cache semantics untouched
• add an optional static cache/index mode
• document exactly what guarantees the user is trading away
• make fast paths contingent on that mode

That is much easier to defend than trying to smuggle static assumptions into the existing cache model.

If you want, next step is I can sketch the contract precisely:

• user-facing option names
• what semantics are disabled
• which subcommands get accelerated first
• what cache files get written
• what tests should exist for default vs static mode.

[smprather]

Codex was not pleased with my chat copy/paste. It wanted a more polished comment 😃.

---

The important distinction is between:

1. cache as an acceleration artifact, and
2. cache as an explicit static-tree contract.

Today, .modulecache is in category 1. It reduces filesystem I/O, but it
does not mean the module tree has agreed to give up dynamic behavior. Because
of that, runtime still has to preserve semantics for things like aliases,
virtuals, modulerc effects, hidden/forbidden state, variants, and other
metadata-sensitive behavior.

That is why even cached lookups can still end up replaying a large part of the
module universe.

I think there is room for a new explicit mode where the administrator says:

• this module tree is static
• I will rebuild indexes when it changes
• Modules may answer covered queries from prebuilt indexes instead of dynamic
  runtime reconstruction

If that contract is explicit, then much more aggressive indexing becomes
reasonable without changing default behavior.

That could look like:

• module cachebuild --static
• extra index artifacts written next to .modulecache
• runtime fast paths enabled only when static mode is explicitly turned on

This should let default behavior and existing dynamic tests stay intact, while
opening the door to much faster implementations of:

• module avail
• module avail foo
• module avail foo/bar
• exact module show foo/bar/1.2

The main point is that the optimization boundary becomes semantic and
intentional, not implicit.

[smprather]

Actually, we came up with a much more sane configuration interface. If a directory tree contains the new cache information, then that's the implicit static-mode opt-in for that directory tree. That way you can mix/match base dirs in the MODULEPATH. Codex is working on the implementation and testing now.

[xdelaruelle]

Many thanks @smprather for your proposal. I am currently trying to see how this feature idea could be integrated.

Existing .modulecache are listing all the files contained in the modulepath, even non-modulefile files. And for modulefiles and modulercs, their content is recorded in the cache file.

In situation like yours, it could lead to very large cache file that makes module quite unresponsive as the whole cache needs to be evaluated to proceed to the command.

So an alternate kind of cache, as you propose, seems interesting to add. I call this new kind of cache index.

This is an initial specification that came to mind (with still open questions to discuss):

Introduce a .moduleindex file under each modulepath.

This file will list all the modulefiles and modulercs contained in the modulepath.

Each line represents a valid modulefile or modulerc with the following information separated by colon character:

• path name, relative to modulepath root directory
• file last modification date (as a Unix epoch time)
• file module header

open question: do we separated these information with colon character or do we parse the file as a Tcl script, as done with .modulecache file.

open question: to generate .moduleindex files, should we have dedicated sub-commands (indexbuild, indexclear) or command-line option onto the cache sub-commands?

As planned for .modulecache (see #563), a new option should be added to be able to list sub-commands that should ignore the .moduleindex file.

open question: could the .moduleindex and .modulecache file be both used for a given sub-command?

open question: based on the answer to the previous question, if .moduleindex and .modulecache have no benefit for being both used, could index be an alternative content for .modulecache