Isolate unit-test runs from the installed Harper root (per-PID system database, config, and logs)

[kriszyp]

Unit-test runs no longer borrow the installed Harper root. Every mocha process now runs against its own per-PID root (unitTests/envDir/<pid>): its own config file, log, keys, and every database — system included. Suites that need the system tables seed them there the way an install would. This removes the exclusive-LOCK contention that made unit runs fail whenever a running Harper (or a leaked prior test process) held the installed root's RocksDB system LOCK — the failure mode that once broke every local unit run for three weeks, and that currently blocks a pre-push changed-tests check. It also makes the suites runnable on a machine with no installed Harper at all.

Refs the unit-system-db-isolation dispatch task.

What changed

• unitTests/mocha.init.js now builds a complete per-PID root before any Harper module loads: the standard directories, plus a config file generated from static/defaultConfig.yaml with the same path fields the installer's validation resolves (the synthesis lives in the new unitTests/perPidRoot.js, so seeding can re-materialize the root after a mid-run tearDownMockDB()). It exports ROOTPATH=<pid dir> — config resolution (configUtils.getConfigFilePath) and the logger's file stream bind to it at first initialization, and it reaches worker threads and spawned processes — and deletes any ambient STORAGE_PATH/SCHEMAS_DATA_PATH, which outrank every configured path in getDatabases(). The apiTests carve-out (which existed only to preserve the installed system path) is gone.
• Per-PID roots are actually reclaimed now. The old exit hook called an async teardown from process.on('exit'), so its removal was scheduled and then dropped by process.exit() — it had silently never worked, which matters more once each root contains a seeded system database and a private key. There was a second reason it never fired: preTestPrep() prepends its own 'exit' listener that calls process.exit(), and node skips every listener after that one, so the ~98 suites using preTestPrep were never reached; that handler now removes the root itself. The hook removes the root synchronously; mocha.init.js additionally wipes a recycled PID's stale root (guarded on isMainThread — worker threads re-run the preload under the same PID) and sweeps envDir roots that are both over an hour stale and whose PID is gone, reclaiming runs killed before the hook (SIGKILL, OOM, cancelled CI). The age floor is load-bearing: kill(pid, 0) alone is not a liveness signal across PID namespaces (a bind-mounted checkout probed from inside a container reads every host run as dead), so materializePerPidRoot() heartbeats the root's mtime — every suite's setupTestDBPath() refreshes it, keeping a live run inside the floor for as long as it makes progress.
• setupTestDBPath() and setTestPath() in unitTests/testUtils.js drop the "preserve the installed system path" logic and assert the same paths as mocha.init.js. The old "really confuses the system" warning was about repointing an already-open system; since mocha.init.js configures these exact paths before any module can open it, the path never changes after open.
• New ensureSystemTables() in unitTests/testUtils.js seeds the per-PID system database exactly as an install does: mountHdb (the system tables), a super_user role with an active admin user (authorizeLocal/getSuperUser() require one to authorize local requests), and self-signed certificates via generateCertsKeys() (the TLS listeners — e.g. MQTT's secure port — need them). The generated key is renamed to unitTestPrivateKey.pem and the per-PID config's tls.privateKey is repointed at it, so cert records, config, and the key watcher agree — the name privateKey.pem is what loadCertificates() registers for an ambient config's key in the in-process privateKeys cache, which would shadow the generated key and pair the fresh certificates with an unrelated one (ERR_OSSL_X509_KEY_VALUES_MISMATCH, zero TLS contexts). The seed is guard-keyed on its final artifacts (key file, tables, a usable admin, the config repoint), and every state the guard rejects it can also repair: alterUser reactivates/relinks an existing admin, alterRole restores a mangled super_user permission — so a broken identity can't strand the guard false and re-run certificate generation against a live server on every call. It decides whether to mount from the database on disk rather than the module cache: after a tearDownMockDB() the cache still answers for a database whose files are gone, and seeding through those handles wrote to unlinked files and reported success (the re-seeded system directory came out empty). That ordering can't be repaired from test code — reopening the handles recurses inside databases.table() — so it now throws a message naming the fix instead.
• unitTests/apiTests/setupTestApp.mjs replaces the getDatabases() preload of the installed root with ensureSystemTables().
• The two MQTT mTLS tests resolve the client key the way the server does — by the certificate record's private_key_name under <root>/keys — instead of reading the tls.privateKey config path.
• Three test files that specifically exercise boot-props-based resolution (which a ROOTPATH env var shadows) clear the variable — and the noBootFile() memo derived from it — for their own scope: configUtils.test.js, installation.test.js, and harper_logger.test.js's initLogSettings describe. The logger's stdio-capture tests now stub writeToLogFile like they already stub the rest of the config surface, since a fresh module copy's initLogSettings can fail after an earlier test's tearDownMockDB() removed the per-PID config.
• New unitTests/isolation.test.js locks the contract directly. The load-bearing test spawns a fresh child with hostile STORAGE_PATH/SCHEMAS_DATA_PATH exported and asserts — inside the child, where no earlier suite can have repaired state — that the env vars are neutralized and the resolved Harper root, storage path, log path, and system database root all land inside the child's own per-PID directory (not merely somewhere under envDir, which the parent's inherited root would also satisfy). It also re-runs initTestEnvironment() inside the child and asserts the storage layout survives it — that assertion was checked the only way worth trusting: it fails when the layout fix is reverted and passes when it is restored. In-process assertions cover the same contract as a fast signal.
• Outside unitTests/: initTestEnvironment() in utility/environment/environmentManager.ts (a documented test-only function) loses its copy of the installed-system-path preservation, and its storage.path now follows the same <pid>/database layout the preload pins — preTestPrep() stubs initSync with this function, so a mid-run re-init previously repointed storage at the PID dir itself and detached the seeded system database (verified: seed → re-init → resetDatabases() now keeps the system tables). utility/common_utils.ts gains a two-line resetNoBootFileCache() test-support export. Production database resolution is untouched.

Why seeding rather than a snapshot

A consistent snapshot of the installed system database is impossible in exactly the scenario this change exists to fix: a RocksDB checkpoint must be taken by the process that has the database open, and when a running Harper holds the LOCK the test process cannot open it at all (nor can a recursive file copy be made consistent from outside the engine). Seeding a fresh minimal system database is the only primitive that works while the installed root is contended — and it is also more deterministic than inheriting whatever users/roles/certs a developer's install happens to contain. Seeding runs the real install code paths (mountHdb, addRole/addUser, generateCertsKeys) rather than copying in a checked-in fixture root: a fixture drifts silently as install internals evolve, while this fails loudly.

harper.js stop stays in test:unit:apitests

With full root isolation the stop is no longer needed for the LOCK, but it is still needed for ports: a running Harper binds *:9925/9926/1883 (wildcard), so the integration-testing loopback pool rejects every 127.0.0.x candidate ("still in use by another Harper node (port 9925 bound); skipping to avoid an SO_REUSEPORT co-bind") and apiTests setup loops indefinitely. Moving apiTests off the default ports would fix that, but is a separate change.

For the human reviewer

Sixteen cross-model rounds ran pre-push (codex graded throughout; Harper-domain adjudication most rounds; gemini where its adapter cooperated). One note on the footer below: the final round was a receipt-only pass after a comment trim, so it ran the graded leg alone and the field reads 4 for want of adjudication — the substantive rounds on the identical code adjudicated to 3, and the per-SHA lock means a re-run cannot lower it. Read it as 3 with an unadjudicated last round, not as a new concern. Later rounds hardened the harness itself: the exit-hook cleanup that had silently never worked (async teardown in process.on('exit')), the dead-PID sweep and its cross-namespace liveness hazard, the self-repairing seed guard, the child-probe isolation test replacing assertions earlier suites could repair, and initTestEnvironment()'s storage path brought onto the <pid>/database layout so a stubbed re-init can no longer detach the seeded system database. Dismissed with rationale along the way: gemini majors that adjudication refuted by code trace (e.g. getHdbBasePath() is set before any module loads; initLogSettings reads ROOTPATH directly, never noBootFile()), ROOTPATH-unset-on-crash, and the seeded admin/password user (per-PID database, loopback-only listeners, removed at exit — the same shape a dev-profile install creates). The decision ledger, with where each landed:

• Process-wide ROOTPATH export vs. injection: exported for the whole run. The transient variant (bind config/logger, then drop) was built and measured first: worker threads spawned mid-run and tests that force config/logger re-init re-resolved boot props once the variable was gone, and kept appending to the installed root's hdb.log. Permanent export closes that; the cost is three scoped test-file shims, which are also the isolation's remaining carve-out (a fresh module copy taken inside those windows resolves the installed root) — stated in AGENTS.md rather than papered over.
• resetNoBootFileCache() on shipped core: kept. noBootFile() memoizes a ROOTPATH-derived true; the shims must be able to un-poison it, and re-requiring the module can't reach the copy configUtils closed over. It is two lines, documented as test support.
• Seeding by running install code vs. fixture root: install code, per the section above. Corollary the reviewers flagged: ensureSystemTables() pays RSA keygen per apiTests process and couples the harness to the security stack's real behavior — deliberate, for fidelity.
• Synthesized config vs. running the installer: the harness writes harper-config.yaml from static/defaultConfig.yaml and fills the path fields install validation would. Drift between this and the real installer would surface as confusing suite failures rather than a diff.
• Key-rename workaround vs. product fix: the in-process privateKeys cache is keyed by file name, which is what forces the unitTestPrivateKey.pem rename. That aliasing could equally bite two roots in one production process; fixing the cache to key by absolute path in security/keys.ts is the root-cause change, left out of scope here (noted in dispatch findings).
• Reclaiming other runs' roots by PID liveness: the sweep only removes roots that are both ESRCH and over an hour stale, because PID liveness is not comparable across namespaces — and live runs heartbeat their root's mtime through setupTestDBPath(), so a run only becomes sweepable if it makes no progress for an hour and its PID reads as gone. The residual (a run wedged inside one suite for over an hour, probed from another namespace) is accepted.
• PID-only root identity across namespaces: retained for this PR. Two containers sharing the checkout can have the same namespace-local PID, so a second run could collide with the first run’s root; adding a per-run namespace token would close that residual but would change the stated per-PID contract and path expectations. This remains the explicit human-review decision from the final independent review.
• Always wiping the root at exit vs. keeping failed-run state for post-mortem: wiped. Debugging a flaky apiTest now means reproducing under a modified harness — before this branch the (broken) exit hook incidentally left state behind. If post-mortem state proves valuable, an env-var opt-out is a two-line follow-up.
• Refusing the teardown-then-reseed ordering vs. repairing it: refused, with an error naming the fix. No suite does this today; making it work would mean reopening database handles whose files were removed, which recurses inside product code. The alternative — carrying on — is what produced a silently empty system database, so failing loudly is the safer default for a harness.
• A pre-existing wedge this branch surfaced, and what it does about it. CI's Node 24 unit job sat nine minutes at unitTests/config/rootConfigWatcher.test.js before being cancelled, and it reproduces locally about one run in six (once for thirteen hours). The cause is in code this branch does not touch: RootConfigWatcher emits ready only from handleChange(), which returns silently when the read comes back empty and swallows read/parse errors, while handleError() drops repeat exhaustion errors without reopening — so .ready can stay pending forever, and .mocharc.json's timeout: 0 means nothing breaks the deadlock. There is no local baseline to compare against, because main cannot run this suite in a dev checkout at all: it dies at load reaching into the installed root (Cannot read properties of undefined (reading 'toJSON') opening database .../database/system.mdb) — the very borrowing this PR removes. Acceptance criterion 5 confines this change to test infrastructure, so the product paths are not touched here; the suite gets a timeout so the flake fails visibly instead of wedging the run, and the product fix is written up for a follow-up issue. Verified in situ: the guard caught a real occurrence as a 30s failure with the suite completing normally, and four further runs were clean.
• One test fails in this checkout on base and branch alike: configValidator "does not warn when a relative rootPath resolves within the limit" — it resolves relative/root/operations-server against the checkout path, and any checkout path over ~76 bytes crosses the 107-byte domain-socket limit. Environmental, pre-existing; noted in the dispatch findings.
• The one red CI shard is a filed Windows product bug on main, not this change. Integration Tests 6/6 (Windows, Node.js v24) fails on integrationTests/apiTests/configuration.test.mjs with set_configuration returning 500 behind EPERM: operation not permitted, rename harper-config.yaml.<pid>.0.<hash>.tmp -> harper-config.yaml at server/operationsServer.ts:345. That is #2313 — filed from a main run (32809913865) on the same test and the same signature, so it predates this head. It reproduces here rather than flaking: a re-run of the shard at the identical head failed again, 3 failures instead of 1. This branch cannot reach that path — the last all-green Integration run on it was fbe41dc33, and the two commits since touch only unitTests/isolation.test.js and unitTests/testUtils.js, which integration tests never load.

Verification

All runs on a scratch Harper install (prod profile) under a fake $HOME, with storage.path made absolute to reproduce the leak shape; "running" means that instance is started and holding its system LOCK:

Check	Base	This change
test:unit:resources, Harper running	fails at startup: IO error: While lock file: .../database/system/LOCK: Resource temporarily unavailable	1674 passing, 0 failing
test:unit:security, Harper running	663 passing (masked by preTestPrep's STORAGE_PATH override — file-order luck)	663 passing, 0 failing
apiTests (direct mocha, Harper stopped)	194 passing	194 passing
Full matrix: main / resources / security / dataLayer / utility / bin / config / logging + apiTests + LMDB-engine resources/apitests/bin	—	all passing (main carries only the pre-existing path-length failure above)
Matrix re-run after each review-fix round (resources 1674, security 663, apiTests 194, LMDB resources 1425, LMDB apiTests 194, main 4858)	—	green, same single pre-existing failure
Seed guard/repair paths (healthy, deactivated admin, mangled role permission, deleted role, teardown + re-seed) and sweep behavior (old+dead swept, fresh+dead kept, old+live kept, backdated-live refreshed by heartbeat)	—	probed live, all correct
Installed-root mtime comparison across full runs	log appends + config rewrites (pre-existing leak)	byte-identical (untouched)
Suites with no installed Harper at all (empty $HOME)	fail (hdb_boot_properties.file ENOENT at server boot)	pass

test:integration:all: 187 passing; the only failures are environmental or pre-existing — the Ollama-backend tests (require a live local Ollama), the QA-269 TTL canary (a deliberate product-behavior probe, untouched by this diff), and one Northwind CSV-upload timeout under heavy box contention that passes cleanly re-run in isolation. Integration tests do not load unitTests/, and the only core edits are the test-only initTestEnvironment branch removal and the additive resetNoBootFileCache export.

test:unit:server and test:unit:components fail at file-load on base and branch alike (fixture files matched by their globs; neither suite is in CI) — dispatch findings. Formatting and lint:required clean.

Feedback-fix verification at b879915f4: npm run build, the focused isolation suite (5 passing), test:unit:apitests (194 passing on the first full run; one unrelated cache-expiration timing failure on the post-review rerun passed immediately in isolation), test:unit:main (4,857 passing with the documented path-length failure plus an inherited GIT_CONFIG_GLOBAL/GIT_PAGER environment failure), test:unit:resources (1,673 passing; randomized HNSW failure passed immediately in isolation), test:integration:all (1,805 passing, six Ollama children cancelled by a pre-existing Node JSON import-attribute error), formatting, and lint. This test-infrastructure-only fix is not separately observable end-to-end.

Complexity: medium

_{Review-Coverage: authored=codex; ran=gemini; adjudicated=domain; declined=claude,cursor-grok,cursor-composer; rounds=2 @ b879915}

_{Human-Review-Need: 3 (decisions: run-root-identity, system-seed-fidelity, key-cache-workaround, test-cache-api) @ b879915}

[gemini-code-assist]

Code Review

This pull request implements robust unit-test isolation by ensuring each test run operates within its own per-PID root directory, preventing interference with any installed Harper root. Key changes include updating the test bootstrap (mocha.init.js) to configure the per-PID environment, introducing a new isolation test suite, and adding ensureSystemTables to seed system databases locally. Feedback on these changes suggests defensively guarding the certificate search and error handling in testUtils.js against null or undefined values to prevent runtime crashes, and ensuring that temporary directories in isolation.test.js are cleaned up before assertions run to avoid resource leaks on failure.

[claude]

Reviewed; no blockers found.

[cb1kenobi]

Reviewed fbe41dc — no issues found. This PR looks good, nice job!

Verification beyond static read: mutation-tested the isolation contract in unitTests/isolation.test.js (disabling the STORAGE_PATH/SCHEMAS_DATA_PATH neutralization in mocha.init.js makes the child-probe test fail immediately — it's not vacuous), and ran unitTests/config/**, unitTests/utility/logging/*, unitTests/utility/installation.test.js, unitTests/security/**, and unitTests/resources/** against both this head and the merge-base (f8a5aa90a) with no live Harper instance running. Pass/pending/fail counts are identical on both sides for every suite (resources: 1674/16/0 both sides, matching the PR's own verification table; security: 661/1/2 both sides, the 2 failures a pre-existing macOS SES/jsLoader realpath issue unrelated to this change). Also confirmed test:unit:apitests remains unsafe to run against a live Harper instance — the isolation removes the RocksDB LOCK dependency, but harper.js stop stays load-bearing for port availability (a wildcard-bound instance still starves the loopback-address pool), exactly as the PR body states.

—
Generated by Barber AI

[cb1kenobi]

Reviewed b879915f4 (up from fbe41dc3) — no issues found. Confirmation pass on top of a prior clean review.

Delta: 2 commits, 2 files (unitTests/isolation.test.js, unitTests/testUtils.js), 34 insertions / 20 deletions — same merge-base as the prior head, so a clean append, not a rebase or merge (git range-diff confirms commits 1-16 identical). Both commits are test-harness hardening: tag the child-probe's stdout so unrelated output can't break JSON.parse, derive cleanup's childRoot from the actual spawned PID instead of the parsed payload and wrap it in try/finally so a failed assertion no longer leaks the child's directory (this also resolves the non-blocking concern raised in an earlier inline comment on this PR), and stop mutating hdb_certificate records while a for await search cursor is still open over the same table (collect matches, finish iterating, then rename).

Re-verified by measurement, not by reading:

• Mutation-tested the isolation contract again at the new head: commenting out the STORAGE_PATH/SCHEMAS_DATA_PATH deletes in mocha.init.js makes isolation.test.js fail immediately (3 failures, /tmp/ambient-storage leaking through) with an ambient env var set; restoring the file returns it to 5/0. Isolation is still real, not vacuous.
• Confirmed resources/databases.ts is byte-identical between fbe41dc3 and b879915f4 (0-line diff), and the mutation run above exercised its live runtime priority of raw STORAGE_PATH/SCHEMAS_DATA_PATH over configured paths — that invariant still holds.
• Grepped both new commits for skip patterns (.skip, it.skip, describe.skip, xit, xdescribe) — none added or removed.
• Re-ran the suites the prior pass measured, at both heads: config/** 248/0/0, utility/logging 139 pass/14 pending, utility/installation 3/0, security/** 661 pass/1 pending/2 fail (same pre-existing macOS SES/jsLoader realpath failures on both heads), isolation.test.js 5/0 — all identical head vs. base. resources/** was 1673 pass/16 pending/1 fail at the base run (the known Caching flake) and 1674/16/0 at head; total executed assertions are identical (1690) on both sides, and neither new commit touches resources/**, so this is the pre-existing flake, not a regression.
• The ports 9925/9926/1883 constraint on test:unit:apitests (documented in the PR body) is unchanged — neither new commit touches setupTestApp.mjs or any port-related code, so that suite remains unsafe to run against a live Harper for the same reason as before.

—
Generated by Barber AI