Skip to content

Commit 14d2788

Browse files
committed
feat(rd-18): Slice 8b Phase 4 — embed() end-to-end
- Core AssetReader seam: SourceId-keyed (the frontend owns no paths), Uint8Array byte contract, resolvedPath in the ok arm; mint E10205 EmbedPathEscapesRoot - Disk reader (compiler layer): source-file-relative resolution, invalid path-literal rejection, lexical-then-canonical (realpath) project-root containment — a `..` escape rejects without touching the filesystem and a symlink escape rejects on the canonical path — stat-before-read 65536 cap + post-read re-check; wired into analyze() via the interning-time SourceId → absolute-path map - Embed declaration typing in the const pass (the only legal position: module-level const byte-array initializer): E10200 legality everywhere else (incl. an expression-typing arm), E10201/E10202/E10205 reader-result mapping, loud E90001 for the format argument, exact-size check, unsized inference from the file size, absent-reader silent poison - ConstValue gains embed provenance; SemanticModel gains embeddedAssets (FQN → canonical path); lower.ts tags ConstDataEntry "embed" honestly - Engine lengthOf now honors the patched symbol type, so length() folds on embed-initialised consts - Spec tier 16 tests red→green (frontend 14 + codegen 2); impl tier 12 (incl. the chmod-proven stat-first cap and the symlink-escape probe) - Verification: passing (full chain, 17/17 turbo tasks) Ref: codeops/features/blend65-ri/plans/rd-18-slice-8b-strings-embed/99-execution-plan.md Tasks: 4.1.1-4.3.3
1 parent 4364281 commit 14d2788

19 files changed

Lines changed: 867 additions & 23 deletions

File tree

codeops/features/blend65-ri/plans/rd-18-slice-8b-strings-embed/99-execution-plan.md

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@
22

33
> **Document**: 99-execution-plan.md
44
> **Parent**: [Index](00-index.md)
5-
> **Last Updated**: 2026-07-17 19:12
6-
> **Progress**: 31/58 tasks (53%)
5+
> **Last Updated**: 2026-07-17 20:12
6+
> **Progress**: 43/58 tasks (74%)
77
> **CodeOps Skills Version**: 3.8.0
88
99
## Overview
@@ -143,24 +143,24 @@ implement → green → impl tests → full verify). Designs live in the 03-docs
143143

144144
**Reference**: 03-03 · AR-10/11/12/13 · ST-25..35
145145

146-
- [ ] 4.1.1 Write `embed.spec.test.ts` (ST-25..32, ST-34, ST-35; temp-dir fixtures incl. the traversal/absolute/oversize probes) — `packages/frontend/src/semantics/`
147-
- [ ] 4.1.2 Write `lower-embed.spec.test.ts` (ST-33) — `packages/codegen/src/il/`
148-
- [ ] 4.1.3 Run — verify FAIL (red; embed poisons silently today)
146+
- [~] 4.1.1 Write `embed.spec.test.ts` (ST-25..32, ST-34, ST-35; temp-dir fixtures incl. the traversal/absolute/oversize probes) — `packages/frontend/src/semantics/` ⏳ (implemented: 2026-07-17 19:26 — scripted-reader oracle at the frontend seam (the frontend owns no fs under R15); the REAL-fs traversal/absolute/oversize/symlink probes pin the disk reader at 4.3.1 and the full pipeline in the Phase-5 negatives)
147+
- [~] 4.1.2 Write `lower-embed.spec.test.ts` (ST-33) — `packages/codegen/src/il/` ⏳ (implemented: 2026-07-17 19:26 — embed-tagged ConstDataEntry + verbatim bytes; the !byte rows are the emitter's, pinned by the Phase-5 golden)
148+
- [x] 4.1.3 Run — verify FAIL (red; embed poisons silently today) ✅ (completed: 2026-07-17 19:33 — frontend 14/14 red; codegen embed-entry red, the provenance-CONTRAST case passes pre-impl by design (pins shipped array tagging))
149149

150150
### Step 4.2: Implementation
151151

152-
- [ ] 4.2.1 Define `AssetReader` + `AssetReadResult` (`SourceId`-keyed, `Uint8Array` contract, ok arm carries `resolvedPath` — per 03-03) — `packages/core/src/host/asset-reader.ts` (+ barrel)
153-
- [ ] 4.2.2 Mint `EmbedPathEscapesRoot: "E10205"` (additive) — `packages/core/src/diagnostics/diagnostic-codes.ts`
154-
- [ ] 4.2.3 Implement the disk reader (sourceId→path map built during interning, resolve, realpath canonicalization, containment, stat-cap + post-read re-check, invalid-path policy per 03-03) + wire into `analyze()` input — `packages/compiler/src/api/` (`asset-reader.ts` + `run-frontend.ts`)
155-
- [ ] 4.2.4 Implement embed declaration typing (EMB-1..4 legality, E10200/01/02/05 + format E90001, size inference, constValues bytes + `source:"embed"`, `embeddedAssets` map, absent-reader poison) — `packages/frontend/src/semantics/type-check/statement-typing.ts` (+ `packages/core/src/semantics/{semantic-model,const-value}.ts` for the `embeddedAssets` map + `source` provenance types)
156-
- [ ] 4.2.5 Provenance passthrough to `ConstDataEntry.type:"embed"``packages/codegen/src/il/lower.ts` (+ stale cfg.ts comment refresh)
157-
- [ ] 4.2.6 Run Step-4.1 suites — verify GREEN
152+
- [x] 4.2.1 Define `AssetReader` + `AssetReadResult` (`SourceId`-keyed, `Uint8Array` contract, ok arm carries `resolvedPath` — per 03-03) — `packages/core/src/host/asset-reader.ts` (+ barrel) ✅ (completed: 2026-07-17 19:55; impl 19:40)
153+
- [x] 4.2.2 Mint `EmbedPathEscapesRoot: "E10205"` (additive) — `packages/core/src/diagnostics/diagnostic-codes.ts` ✅ (completed: 2026-07-17 19:55; impl 19:40)
154+
- [x] 4.2.3 Implement the disk reader (sourceId→path map built during interning, resolve, realpath canonicalization, containment, stat-cap + post-read re-check, invalid-path policy per 03-03) + wire into `analyze()` input — `packages/compiler/src/api/` (`asset-reader.ts` + `run-frontend.ts`) ✅ (completed: 2026-07-17 19:55; impl 19:40)
155+
- [x] 4.2.4 Implement embed declaration typing (EMB-1..4 legality, E10200/01/02/05 + format E90001, size inference, constValues bytes + `source:"embed"`, `embeddedAssets` map, absent-reader poison) — `packages/frontend/src/semantics/type-check/statement-typing.ts` (+ `packages/core/src/semantics/{semantic-model,const-value}.ts` for the `embeddedAssets` map + `source` provenance types) ✅ (completed: 2026-07-17 19:55; impl 19:40)
156+
- [x] 4.2.5 Provenance passthrough to `ConstDataEntry.type:"embed"``packages/codegen/src/il/lower.ts` (+ stale cfg.ts comment refresh) ✅ (completed: 2026-07-17 19:55; impl 19:40)
157+
- [x] 4.2.6 Run Step-4.1 suites — verify GREEN ✅ (completed: 2026-07-17 19:55 — embed spec 14/14, lower-embed 2/2, whole frontend 886/886; two green-phase fixes: embed-handled consts skip scalar evaluation, engine lengthOf honors the patched symbol type)
158158

159159
### Step 4.3: Impl tests & hardening
160160

161-
- [ ] 4.3.1 Write `asset-reader.impl.test.ts` (byte identity ≥`$80`, stat-cap ordering + post-read re-check, canonical containment incl. a symlink-escape probe) — `packages/compiler/src/api/`
162-
- [ ] 4.3.2 Write `embed.impl.test.ts` (provenance, inference parity) — `packages/frontend/src/semantics/`
163-
- [ ] 4.3.3 Full verify + prior-goldens check
161+
- [x] 4.3.1 Write `asset-reader.impl.test.ts` (byte identity ≥`$80`, stat-cap ordering + post-read re-check, canonical containment incl. a symlink-escape probe) — `packages/compiler/src/api/` ✅ (completed: 2026-07-17 19:55 — 9/9; stat-first proven via a chmod-000 oversize file; post-read re-check is code-guarded, race not unit-reproducible without fs mocks)
162+
- [x] 4.3.2 Write `embed.impl.test.ts` (provenance, inference parity) — `packages/frontend/src/semantics/` ✅ (completed: 2026-07-17 19:55 — 3/3 incl. SourceId keying in multi-file programs)
163+
- [x] 4.3.3 Full verify + prior-goldens check ✅ (completed: 2026-07-17 20:12 — 17/17 turbo tasks from the repo root; goldens byte-exact)
164164

165165
**Verify**: (same command)
166166

packages/codegen/src/il/cfg.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,8 +58,8 @@ export interface ILFunction {
5858
/**
5959
* A blob of constant data emitted into the binary.
6060
*
61-
* Produced for array/struct literals and `embed`ded data. Empty in v1 — const
62-
* and embed lowering arrive with their slices.
61+
* Produced for array/struct literals and `embed`ded data (the embed arm is
62+
* tagged from the const value's provenance).
6363
*/
6464
export interface ConstDataEntry {
6565
/** The ACME label this data is emitted under. */
Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
/**
2+
* Specification tests for embedded-data lowering: an embed-initialised
3+
* const array flows into the lowered program's const-data stream as an
4+
* entry tagged with embed provenance, carrying the file bytes verbatim
5+
* under the standard data label. The labeled byte rows themselves are the
6+
* emitter's job, pinned by the acceptance golden.
7+
*
8+
* Oracles derive from the frozen spec Ch 13 and the recorded decisions;
9+
* exercised through the REAL frontend with a scripted asset reader. Never
10+
* derived from the implementation.
11+
*/
12+
13+
import { describe, expect, it } from "vitest";
14+
import { createDiagnosticBag, DEFAULT_PROFILE } from "@blend65/core";
15+
import type { AssetReader, ProgramNode } from "@blend65/core";
16+
import {
17+
analyze,
18+
lex,
19+
modelToFunctionInfo,
20+
modelToModuleVars,
21+
parse,
22+
planAllocation,
23+
} from "@blend65/frontend";
24+
import type { ILProgram } from "./cfg.js";
25+
import { lowerToIL } from "./lower.js";
26+
27+
const TABLE_BYTES = [0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80];
28+
29+
/** A scripted reader serving the standard 8-byte table fixture. */
30+
const READER: AssetReader = {
31+
readAsset: (_sourceId, relPath) =>
32+
relPath === "table.bin"
33+
? {
34+
kind: "ok",
35+
bytes: Uint8Array.from(TABLE_BYTES),
36+
resolvedPath: "/proj/table.bin",
37+
}
38+
: { kind: "not-found" },
39+
};
40+
41+
/** Lowers one source end-to-end through the real frontend + reader. */
42+
function lowerEmbed(source: string): { il: ILProgram; hasErrors: boolean } {
43+
const bag = createDiagnosticBag();
44+
const { tokens } = lex(1, source, bag);
45+
const { ast } = parse({ tokens, source, sourceId: 1, bag });
46+
const programs: ProgramNode[] = [ast];
47+
const model = analyze({
48+
programs,
49+
bag,
50+
profile: DEFAULT_PROFILE,
51+
assetReader: READER,
52+
});
53+
const plan = planAllocation(
54+
{
55+
functions: modelToFunctionInfo(model),
56+
moduleVars: modelToModuleVars(model),
57+
zpUserVars: [],
58+
upstreamErrors: bag.hasErrors(),
59+
},
60+
DEFAULT_PROFILE,
61+
bag,
62+
);
63+
const il = lowerToIL({ program: programs, model, plan }, bag);
64+
return { il, hasErrors: bag.hasErrors() };
65+
}
66+
67+
describe("Specification: embed lowering to the const-data stream", () => {
68+
it("produces an embed-tagged const-data entry with the file bytes verbatim", () => {
69+
const { il, hasErrors } = lowerEmbed(
70+
'module Main;\nconst D: byte[] = embed("table.bin");\n' +
71+
"function main(): void { poke($C000, D[0]); }\n",
72+
);
73+
expect(hasErrors).toBe(false);
74+
const entry = il.constData.find((e) => e.symbol === "__data_Main_D");
75+
expect(entry, "expected the embed const-data entry").toBeDefined();
76+
expect(entry?.type).toBe("embed");
77+
expect(Array.from(entry?.data ?? [])).toEqual(TABLE_BYTES);
78+
});
79+
80+
it("keeps plain const arrays tagged as arrays (provenance is embed-specific)", () => {
81+
const { il, hasErrors } = lowerEmbed(
82+
"module Main;\nconst A: byte[2] = [1, 2];\n" +
83+
"function main(): void { poke($C000, A[0]); }\n",
84+
);
85+
expect(hasErrors).toBe(false);
86+
const entry = il.constData.find((e) => e.symbol === "__data_Main_A");
87+
expect(entry?.type).toBe("array");
88+
});
89+
});

packages/codegen/src/il/lower.ts

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -219,14 +219,20 @@ export function lowerToIL(input: LowerInput, bag: DiagnosticBag): ILProgram {
219219

220220
// Const aggregates carry fully-evaluated memory images — each becomes an
221221
// in-image data entry under its `__data_<Module>_<name>` label (const
222-
// SCALARS keep inlining as immediates and own no data).
222+
// SCALARS keep inlining as immediates and own no data). Embedded assets
223+
// keep their provenance tag; everything else derives from the type.
223224
const constData: ConstDataEntry[] = [];
224225
for (const [sym, value] of input.model.constValues) {
225226
if (value.bytes === undefined) continue;
226227
constData.push({
227228
symbol: constDataSymbol(sym),
228229
data: value.bytes,
229-
type: sym.type.kind === "struct" ? "struct" : "array",
230+
type:
231+
value.source === "embed"
232+
? "embed"
233+
: sym.type.kind === "struct"
234+
? "struct"
235+
: "array",
230236
});
231237
}
232238

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
/**
2+
* Implementation tests for the disk asset reader: byte identity, the
3+
* stat-before-read size cap, and lexical + canonical (symlink-aware)
4+
* project-root containment — all against the real filesystem in a temp
5+
* tree.
6+
*/
7+
8+
import { afterAll, beforeAll, describe, expect, it } from "vitest";
9+
import {
10+
chmodSync,
11+
mkdirSync,
12+
mkdtempSync,
13+
rmSync,
14+
symlinkSync,
15+
writeFileSync,
16+
} from "node:fs";
17+
import { tmpdir } from "node:os";
18+
import { join } from "node:path";
19+
import { realpathSync } from "node:fs";
20+
import { createDiskAssetReader } from "./asset-reader.js";
21+
import type { AssetReader } from "@blend65/core";
22+
23+
let base: string;
24+
let projectRoot: string;
25+
let sourcePath: string;
26+
let reader: AssetReader;
27+
28+
const SRC_ID = 1;
29+
30+
beforeAll(() => {
31+
base = mkdtempSync(join(tmpdir(), "blend65-asset-"));
32+
projectRoot = join(base, "proj");
33+
mkdirSync(join(projectRoot, "src"), { recursive: true });
34+
sourcePath = join(projectRoot, "src", "main.blend");
35+
writeFileSync(sourcePath, "module Main;\n");
36+
reader = createDiskAssetReader({
37+
sources: new Map([[SRC_ID, sourcePath]]),
38+
projectRoot,
39+
});
40+
});
41+
42+
afterAll(() => {
43+
rmSync(base, { recursive: true, force: true });
44+
});
45+
46+
describe("disk asset reader — bytes", () => {
47+
it("returns high-bit and NUL bytes identically (no text decoding)", () => {
48+
const bytes = Uint8Array.from([0x00, 0x41, 0x80, 0xff, 0x0d, 0x0a]);
49+
writeFileSync(join(projectRoot, "src", "raw.bin"), bytes);
50+
51+
const result = reader.readAsset(SRC_ID, "raw.bin");
52+
expect(result.kind).toBe("ok");
53+
if (result.kind === "ok") {
54+
expect(Array.from(result.bytes)).toEqual(Array.from(bytes));
55+
expect(result.resolvedPath).toBe(realpathSync(join(projectRoot, "src", "raw.bin")));
56+
}
57+
});
58+
59+
it("resolves relative to the CALLING source file's directory", () => {
60+
writeFileSync(join(projectRoot, "top.bin"), Uint8Array.from([1]));
61+
// The source lives in src/, so the asset one level up needs `..` —
62+
// legal as long as it stays inside the project root.
63+
const result = reader.readAsset(SRC_ID, "../top.bin");
64+
expect(result.kind).toBe("ok");
65+
});
66+
});
67+
68+
describe("disk asset reader — size cap", () => {
69+
it("rejects an oversized file by stat, before any read", () => {
70+
const big = join(projectRoot, "src", "big.bin");
71+
writeFileSync(big, new Uint8Array(65537));
72+
// Remove read permission: a read attempt would fail as not-found, so
73+
// getting too-large proves the cap fired on stat alone.
74+
chmodSync(big, 0o000);
75+
76+
const result = reader.readAsset(SRC_ID, "big.bin");
77+
expect(result).toEqual({ kind: "too-large", size: 65537 });
78+
chmodSync(big, 0o644);
79+
});
80+
81+
it("accepts a file of exactly the cap size", () => {
82+
writeFileSync(join(projectRoot, "src", "max.bin"), new Uint8Array(65536));
83+
expect(reader.readAsset(SRC_ID, "max.bin").kind).toBe("ok");
84+
});
85+
});
86+
87+
describe("disk asset reader — containment", () => {
88+
it("rejects a traversal escape whether or not the target exists", () => {
89+
// No ../../outside.bin exists anywhere — the lexical check needs no
90+
// filesystem access, so the rejection must still be outside-root.
91+
expect(reader.readAsset(SRC_ID, "../../outside.bin")).toEqual({
92+
kind: "outside-root",
93+
});
94+
});
95+
96+
it("rejects an inside-the-project symlink that points outside it", () => {
97+
writeFileSync(join(base, "secret.bin"), Uint8Array.from([7]));
98+
symlinkSync(join(base, "secret.bin"), join(projectRoot, "src", "link.bin"));
99+
100+
expect(reader.readAsset(SRC_ID, "link.bin")).toEqual({ kind: "outside-root" });
101+
});
102+
103+
it("rejects absolute and escape-bearing path literals as invalid", () => {
104+
expect(reader.readAsset(SRC_ID, "/etc/hostname").kind).toBe("not-found");
105+
expect(reader.readAsset(SRC_ID, "a\\x2Eb.bin").kind).toBe("not-found");
106+
expect(reader.readAsset(SRC_ID, "").kind).toBe("not-found");
107+
});
108+
109+
it("reports an unknown source id as not-found", () => {
110+
expect(reader.readAsset(999, "raw.bin").kind).toBe("not-found");
111+
});
112+
113+
it("reports a genuinely missing in-root file as not-found", () => {
114+
expect(reader.readAsset(SRC_ID, "nope.bin").kind).toBe("not-found");
115+
});
116+
});
Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
/**
2+
* The disk-backed asset reader for `embed()`.
3+
*
4+
* Resolution is strictly source-file-relative: the embed path resolves
5+
* against the directory of the source file containing the call, then must
6+
* stay inside the project root. Containment is checked twice — lexically
7+
* first, so a `..` escape is rejected without touching the filesystem
8+
* (whether or not the target exists), then on the CANONICAL (symlink-
9+
* resolved) path, so a symlink inside the project pointing outside it
10+
* cannot smuggle bytes in. The 65536-byte cap is enforced by `stat` BEFORE
11+
* the file is read (a size bomb is rejected without reading it) and
12+
* re-checked on the actual bytes afterwards, closing the stat→read race.
13+
*
14+
* Reads are binary end-to-end (`Uint8Array`) — a text decode would corrupt
15+
* bytes at or above `$80`.
16+
*/
17+
18+
import { readFileSync, realpathSync, statSync } from "node:fs";
19+
import { dirname, isAbsolute, resolve, sep } from "node:path";
20+
import type { AssetReader, AssetReadResult, SourceId } from "@blend65/core";
21+
22+
/** The largest embeddable file: nothing larger fits the address space. */
23+
const MAX_EMBED_BYTES = 65536;
24+
25+
/** What the disk reader needs at construction. */
26+
export interface DiskAssetReaderInput {
27+
/** SourceId → the absolute path of that source file (built at interning). */
28+
readonly sources: ReadonlyMap<SourceId, string>;
29+
/** The project root every resolved asset must stay inside. */
30+
readonly projectRoot: string;
31+
}
32+
33+
/** A path literal the language accepts: printable ASCII, no escapes. */
34+
function isValidPathLiteral(relPath: string): boolean {
35+
if (relPath.length === 0 || relPath.includes("\\")) return false;
36+
for (const ch of relPath) {
37+
const cp = ch.codePointAt(0) ?? 0;
38+
if (cp < 0x20 || cp > 0x7e) return false;
39+
}
40+
return true;
41+
}
42+
43+
/**
44+
* Builds the disk-backed reader over the compilation's source-path map.
45+
*
46+
* @param input The source-path map and the project root.
47+
* @returns The reader `analyze()` consumes.
48+
*/
49+
export function createDiskAssetReader(input: DiskAssetReaderInput): AssetReader {
50+
const lexicalRoot = resolve(input.projectRoot) + sep;
51+
52+
return {
53+
readAsset(sourceId: SourceId, relPath: string): AssetReadResult {
54+
// An absolute or escape-bearing/non-printable path literal is
55+
// rejected as invalid before any resolution.
56+
if (!isValidPathLiteral(relPath) || isAbsolute(relPath)) {
57+
return { kind: "not-found" };
58+
}
59+
60+
const sourcePath = input.sources.get(sourceId);
61+
if (sourcePath === undefined) return { kind: "not-found" };
62+
const resolved = resolve(dirname(sourcePath), relPath);
63+
64+
// Lexical containment first — needs no filesystem access, so a `..`
65+
// escape is rejected whether or not the target exists.
66+
if (!resolved.startsWith(lexicalRoot)) return { kind: "outside-root" };
67+
68+
// Canonicalize, then re-check containment on the canonical path — a
69+
// lexical prefix check alone would pass an inside-the-project symlink
70+
// that points outside it.
71+
let canonical: string;
72+
let canonicalRoot: string;
73+
try {
74+
canonical = realpathSync(resolved);
75+
canonicalRoot = realpathSync(resolve(input.projectRoot)) + sep;
76+
} catch {
77+
return { kind: "not-found" };
78+
}
79+
if (!canonical.startsWith(canonicalRoot)) return { kind: "outside-root" };
80+
81+
try {
82+
const size = statSync(canonical).size;
83+
if (size > MAX_EMBED_BYTES) return { kind: "too-large", size };
84+
const bytes = new Uint8Array(readFileSync(canonical));
85+
if (bytes.byteLength > MAX_EMBED_BYTES) {
86+
return { kind: "too-large", size: bytes.byteLength };
87+
}
88+
return { kind: "ok", bytes, resolvedPath: canonical };
89+
} catch {
90+
return { kind: "not-found" };
91+
}
92+
},
93+
};
94+
}

0 commit comments

Comments
 (0)