feat: Update CORTEX design plan and execution plan with status and next steps; enhance routing policy resolution and tests

Author: devlux76

CORTEX-DESIGN-PLAN-TODO.md

@@ -22,6 +22,20 @@ Interpretation rule for this document:
 1. Architectural intent remains here.
 2. Execution order, command contract, and test matrix are governed by `PROJECT-EXECUTION-PLAN.md`.
 
+### 0.1 Status Update (2026-03-11)
+
+Completed since the prior snapshot:
+1. Model-profile source-of-truth layer exists (`core/ModelProfile.ts`, `core/ModelDefaults.ts`, `core/ModelProfileResolver.ts`).
+2. Routing policy dimensions are now derived from profile-owned embedding dimensions (`createRoutingPolicy`).
+3. Guard command exists for model-related hardcoded numeric literals (`npm run guard:model-derived`).
+4. Runtime numeric constants are centralized for backend/storage internals (`core/NumericConstants.ts`).
+5. Runtime helper resolves model metadata and derives routing policy in one call (`resolveRoutingPolicyForModel` in `Policy.ts`).
+
+Next focus:
+1. Wire resolved model profiles into runtime ingest/query entry points.
+2. Build embedding provider resolver + runner modules and associated tests.
+3. Add browser/electron runtime test lanes to match merge-gate policy.
+
 ## 1. Design
 
 ### 1.1 Product contract
@@ -139,11 +153,10 @@ Graceful degradation:
 
 ### 1.9 Current gap analysis from repo snapshot
 Observed blockers in current PoC files:
-1. Interface mismatch: `VectorBackend.ts` method signatures differ from concrete backend implementations.
-2. `BackendKind` and `detectBackend` are referenced in code but only defined in sketch docs.
-3. No project scaffolding (`package.json`, `tsconfig`, test runner, lint config).
-4. Shader and backend files are present but not integrated into an executable runtime.
-5. No concrete OPFS or IndexedDB implementation files yet.
+1. Embedding runtime modules (provider resolver + runner) are still missing.
+2. Ingest/query orchestrators are not yet wired to resolved `ModelProfile` values.
+3. Browser/Electron runtime test lanes are not yet implemented in scripts/CI.
+4. Shader and backend files compile but are not yet integrated into a full vertical runtime path.
 
 These are Phase 0 blockers and should be fixed before feature work.
 
@@ -256,16 +269,16 @@ Priority legend:
 3. P2 = optimization, resilience, polish.
 
 ### P0 (do first)
-1. Create TypeScript project scaffolding (`package.json`, `tsconfig`, test runner).
-2. Define canonical shared types for all entities from sketch and errata.
-3. Resolve `VectorBackend` interface mismatch against all backend classes.
-4. Add `BackendKind` and `detectBackend` in real code module.
-5. Add missing imports/exports so backend factory compiles.
+1. ~~Create TypeScript project scaffolding (`package.json`, `tsconfig`, test runner).~~ ✅ Done
+2. ~~Define canonical shared types for all entities from sketch and errata.~~ ✅ Done
+3. ~~Resolve `VectorBackend` interface mismatch against all backend classes.~~ ✅ Done
+4. ~~Add `BackendKind` and `detectBackend` in real code module.~~ ✅ Done
+5. ~~Add missing imports/exports so backend factory compiles.~~ ✅ Done
 6. Decide and document naming convention: `Metroid` vs `Medoid`.
 7. Add minimal CI workflow for build and tests.
-8. Add lint/format rules for consistent style.
+8. ~~Add lint/format rules for consistent style.~~ ✅ Done
 9. Add deterministic floating-point tolerance helpers for backend parity tests.
-10. Add smoke test that instantiates each backend or cleanly falls back.
+10. ~~Add smoke test that instantiates each backend or cleanly falls back.~~ ✅ Done (`tests/BackendSmoke.test.ts`)
 
 ### P1 (v1 core)
 1. ~~Implement `OPFSVectorStore.appendVector` and `readVector`.~~ ✅ Done (Phase 1, 2026-03-11)

PROJECT-EXECUTION-PLAN.md

@@ -2,22 +2,48 @@
 
 This file is the canonical carry-over plan for implementation sequencing, test gates, and next-session priority. Keep this file updated as work progresses.
 
+## Pass Status (2026-03-11)
+
+Completed in this pass:
+1. Implemented `ModelProfile` source-of-truth layer:
+   - `core/ModelProfile.ts`
+   - `core/ModelDefaults.ts`
+   - `core/ModelProfileResolver.ts`
+2. Added TDD coverage for model defaults/resolution and routing derivation:
+   - `tests/model/ModelDefaults.test.ts`
+   - `tests/model/ModelProfileResolver.test.ts`
+   - `tests/model/RoutingPolicy.test.ts`
+3. Added `createRoutingPolicy(...)` derivation from profile-owned embedding dimensions.
+4. Added runtime numeric constants surface (`core/NumericConstants.ts`) and removed repeated byte/workgroup literals from backends and storage paths.
+5. Added `guard:model-derived` command and scanner (`scripts/guard-model-derived.mjs`) to block hardcoded model-related numeric literals outside approved sources.
+6. Validation gates passed in this workspace state:
+   - `npm run test:unit`
+   - `npm run guard:model-derived`
+   - `npm run build`
+   - `npm run lint`
+7. Added model-profile-to-policy bridge helper so runtime callers can resolve profile and derive routing in one step:
+   - `resolveRoutingPolicyForModel(...)` in `Policy.ts`
+   - integration tests in `tests/model/RoutingPolicy.test.ts`
+
+Open items carried to next pass:
+1. Wire resolved `ModelProfile` into first concrete ingest/query orchestrator path (once those runtime modules are added).
+2. Add embedding provider resolver/runner modules and connect fallback policy to runtime execution.
+3. Add browser/electron runtime scripts and CI lanes for non-Node merge gating.
+
 ## Next Session Highest Priority (P0)
 
-Run a full code pass across the repository before new feature coding.
+Integrate model-profile ownership into runtime flows and start the embedding vertical slice.
 
 Instruction:
-1. Traverse all TypeScript source and tests.
-2. Remove hardcoded model-dependent assumptions.
-3. Classify every numeric constant as one of:
-   - `model-derived`: must come from resolved model metadata.
-   - `runtime-policy`: must come from explicit policy/config objects.
-4. Add or update tests first (Red), then implement (Green), then refactor.
+1. Use `ModelProfileResolver` at runtime boundaries before any policy derivation or embedding execution.
+2. Implement first embedding runner/provider resolver slice with fallback semantics.
+3. Keep strict TDD (Red -> Green -> Refactor).
+4. If a blocker appears, record it in this document under an error log entry and continue with the next actionable slice.
 
 Definition of done for this pass:
-1. No model-dependent domain constants remain in feature code.
-2. A `ModelProfile` contract is the single source of truth for model-derived values.
-3. A guard command fails CI when disallowed hardcoded literals are introduced.
+1. Runtime path resolves model metadata through `ModelProfileResolver` before use.
+2. Embedding provider resolver tests and implementation are present.
+3. Any unresolved blocker is documented with file/symptom/next action.
 
 ## Non-Negotiable Rules
 
@@ -30,11 +56,11 @@ Definition of done for this pass:
 
 1. Lock contracts and TDD workflow.
 2. Lock command contract and CI lanes.
-3. Implement model-profile layer first:
+3. ~~Implement model-profile layer first:~~ ✅ Done (2026-03-11)
    - `core/ModelProfile.ts`
    - `core/ModelProfileResolver.ts`
    - `core/ModelDefaults.ts`
-4. Replace hardcoded model-dependent values with `ModelProfile` lookups.
+4. ~~Replace hardcoded model-dependent values with `ModelProfile` lookups.~~ ✅ Done for current code paths (2026-03-11)
 5. Implement embedding runner with fallback chain and telemetry:
    - `embeddings/EmbeddingRunner.ts`
    - `embeddings/ProviderResolver.ts`
@@ -63,24 +89,27 @@ Definition of done for this pass:
 
 ## Command Contract
 
+Available now:
 1. `npm run test:unit`
 2. `npm run test:unit -- tests/model/ModelProfileResolver.test.ts`
 3. `npm run test:unit -- tests/model/ModelDefaults.test.ts`
-4. `npm run test:unit -- tests/embeddings/ProviderResolver.test.ts`
-5. `npm run test:unit -- tests/embeddings/OnnxEmbeddingRunner.test.ts`
-6. `npm run guard:model-derived`
-7. `npm run test:browser`
-8. `npm run test:electron`
-9. `npm run build && npm run lint`
-10. `npm run test:all`
-11. `npm run benchmark`
+4. `npm run guard:model-derived`
+5. `npm run build && npm run lint`
+
+Planned commands to add in later passes:
+1. `npm run test:unit -- tests/embeddings/ProviderResolver.test.ts`
+2. `npm run test:unit -- tests/embeddings/OnnxEmbeddingRunner.test.ts`
+3. `npm run test:browser`
+4. `npm run test:electron`
+5. `npm run test:all`
+6. `npm run benchmark`
 
 ## Known Hardcoded Hotspots To Clean First
 
-1. `core/types.ts` comments with sample token/dimension values.
-2. `Policy.ts` sample projection dimensions.
-3. `Cortex-sketch.md` and `Cortex-sketch-errata.md` illustrative constants (`2048`, `768`, `128`, `0.68`, `40`, etc.).
-4. Any ingest/query defaults currently assumed without model metadata backing.
+1. ~~`core/types.ts` comments with sample token/dimension values.~~ ✅ Updated to source-owned wording.
+2. ~~`Policy.ts` sample projection dimensions.~~ ✅ Replaced with derived policy implementation.
+3. ~~`Cortex-sketch.md` and `Cortex-sketch-errata.md` illustrative constants (`2048`, `768`, `128`, `0.68`, `40`, etc.).~~ ✅ Clarified as illustrative.
+4. Any ingest/query defaults currently assumed without model metadata backing. (Pending runtime module implementation)
 
 ## Scope Notes
 

Policy.ts

@@ -1,4 +1,9 @@
 import type { ModelProfile } from "./core/ModelProfile";
+import {
+  ModelProfileResolver,
+  type ModelProfileResolverOptions,
+  type ResolveModelProfileInput,
+} from "./core/ModelProfileResolver";
 
 export type QueryScope = "broad" | "normal" | "narrow" | "default";
 
@@ -16,6 +21,17 @@ export interface RoutingPolicy {
   narrow: ProjectionHead;
 }
 
+export interface ResolvedRoutingPolicy {
+  modelProfile: ModelProfile;
+  routingPolicy: RoutingPolicy;
+}
+
+export interface ResolveRoutingPolicyOptions {
+  resolver?: ModelProfileResolver;
+  resolverOptions?: ModelProfileResolverOptions;
+  routingPolicyOverrides?: Partial<RoutingPolicyDerivation>;
+}
+
 export interface RoutingPolicyDerivation {
   broadDimRatio: number;
   normalDimRatio: number;
@@ -112,3 +128,20 @@ export function createRoutingPolicy(
     },
   };
 }
+
+export function resolveRoutingPolicyForModel(
+  input: ResolveModelProfileInput,
+  options: ResolveRoutingPolicyOptions = {},
+): ResolvedRoutingPolicy {
+  const resolver =
+    options.resolver ?? new ModelProfileResolver(options.resolverOptions);
+  const modelProfile = resolver.resolve(input);
+
+  return {
+    modelProfile,
+    routingPolicy: createRoutingPolicy(
+      modelProfile,
+      options.routingPolicyOverrides,
+    ),
+  };
+}

tests/model/RoutingPolicy.test.ts

@@ -1,6 +1,10 @@
 import { describe, expect, it } from "vitest";
 
-import { createRoutingPolicy } from "../../Policy";
+import {
+  createRoutingPolicy,
+  resolveRoutingPolicyForModel,
+} from "../../Policy";
+import { ModelProfileResolver } from "../../core/ModelProfileResolver";
 
 describe("createRoutingPolicy", () => {
   it("derives projection dimensions from embedding dimension", () => {
@@ -37,3 +41,63 @@ describe("createRoutingPolicy", () => {
     expect(policy.narrow.offset).toBe(24576);
   });
 });
+
+describe("resolveRoutingPolicyForModel", () => {
+  it("resolves from metadata and derives policy in one call", () => {
+    const result = resolveRoutingPolicyForModel({
+      modelId: "nomic-embed-text",
+      metadata: {
+        embeddingDimension: 768,
+        contextWindowTokens: 8192,
+      },
+    });
+
+    expect(result.modelProfile.source).toBe("metadata");
+    expect(result.modelProfile.embeddingDimension).toBe(768);
+    expect(result.routingPolicy.broad.dimIn).toBe(768);
+    expect(result.routingPolicy.normal.dimOut).toBe(192);
+  });
+
+  it("resolves from registry when metadata is missing", () => {
+    const result = resolveRoutingPolicyForModel(
+      { modelId: "all-MiniLM-L6-v2" },
+      {
+        resolverOptions: {
+          registry: {
+            "all-minilm-l6-v2": {
+              embeddingDimension: 384,
+              contextWindowTokens: 512,
+            },
+          },
+        },
+      },
+    );
+
+    expect(result.modelProfile.source).toBe("registry");
+    expect(result.routingPolicy.narrow.dimIn).toBe(384);
+  });
+
+  it("supports injected resolver and routing overrides", () => {
+    const resolver = new ModelProfileResolver({
+      registry: {
+        "my-model": {
+          embeddingDimension: 1024,
+          contextWindowTokens: 4096,
+        },
+      },
+    });
+
+    const result = resolveRoutingPolicyForModel(
+      { modelId: "my-model" },
+      {
+        resolver,
+        routingPolicyOverrides: {
+          broadHashBits: 64,
+        },
+      },
+    );
+
+    expect(result.modelProfile.embeddingDimension).toBe(1024);
+    expect(result.routingPolicy.broad.bits).toBe(64);
+  });
+});