C64-dev/newLang/planning/claims/E08-S01.md

---

story: E08-S01 owner: codebuff status: done started: 2026-11-10 completed: 2026-11-10 depends-on: [E01-S05, E02-S04]

E08-S01 Claim

Intended change

One harness that runs the verification suites reproducibly on the target machine: each suite gets a fresh build directory (so it creates its own D81), a wall-clock limit that bounds emulated cycles, a captured log, a machine-readable result, and — where the output is stable — a reviewed SHA-256 baseline of the PRG/MAP artifacts. A failing suite names its fixture and the observed mismatch.

File area

• tests/regression/run-regression.ps1 (new): suite loop, child PowerShell per suite, timeout with tree kill, log capture, artifact hashing and comparison, JSON results, baseline update
• tests/regression/suites.ps1 (new): manifest of 24 suites, each pointing at an existing scripts/verify-*-test.ps1 script and a <name>-test build directory
• tests/regression/baseline.json (new): reviewed hashes for the source, lexer and parser artifacts
• tests/regression/harness.tests.ps1 (new): self-tests of the mechanics with no emulator
• docs/decisions.md (D152), planning/claims/E08-S01.md
• generated files under build/regression/ (results, per-suite logs)

Acceptance evidence

• Creates a fresh D81 per test run: every suite runs in build/<name>-test, which the harness deletes before the run unless -NoFresh is passed; each of the three exercised suites builds its own image there. The self-test proves the deletion with a suite that only passes when its directory starts empty.
• Enforces time/cycle limits and captures machine-readable results: a suite exceeding its Timeout seconds is killed with its process tree and recorded as timedOut; results are JSON with startedAt, finishedAt, per-suite status, seconds, exitCode, signature, problems, log and artifacts, plus passed/failed totals.
• Extracts PRG/ERR/MAP outputs and compares expected artifacts: artifacts are hashed with SHA-256 and compared against baseline.json; a rerun reports match, a changed file reports changed, a new file reports new, and a baselined file that disappeared reports missing.
• A failed test identifies the fixture and observed signature: the failed suite's signature is the diagnosis line from its log (the self-test asserts a fixture name and observed value survive), not merely exit code 3.

Coordination notes

• Depends on E01-S05 (container/loader) and E02-S04 (lexer); both are complete. The harness runs the existing scripts/verify-*-test.ps1 scripts unchanged, so it adds automation rather than a second implementation of any check.
• Only the source, lexer and parser suites have a reviewed baseline: their PRG and MAP bytes are stable across runs. The other 21 suites are executed for status, success marker and log, because their artifacts either embed run-specific data or have not been reviewed yet. Adding a baseline entry requires reviewing the bytes first, per E08-S02.
• -NoFresh exists for the self-test only; a normal run always reuses nothing.

Work log

• The manifest was first written as suites.psd1; invoking a .psd1 with & yields nothing, so the harness reported “manifest lists no suites”. It is now suites.ps1.
• The harness stored its result records in $results while the output path parameter was $Results: PowerShell variable names are case-insensitive, so the path became an empty array and New-Item -Path "" failed with “Cannot bind argument to parameter ‘Path’". The parameter is now $ResultsPath with an alias -Results so callers are unchanged.
• -File passes -Filter source,lexer,parser as a single string; the filter now splits on commas as well as accepting a string array.
• The self-test expected exit code 0 from a run that deliberately includes two failing suites. It now asserts the two expected suites pass and that failing suites are reported, which is what the harness guarantees.
• The failing suite's signature was its exit code 3; it now prefers the diagnosis line from the captured output.

Completion

• Tests run: tests/regression/harness.tests.ps1, then run-regression.ps1 -Filter source,lexer,parser -UpdateBaseline, then the same filter without -UpdateBaseline to prove the baseline matches.
• Results: regression harness tests passed; both real runs reported 3 passed, 0 failed (source 18.3 s / 18.5 s, lexer 21.7 s / 23.7 s, parser 35.6 s / 37.9 s). Baseline covers 3 suites: source PRG 4C0FA221…DC6AAC65, lexer PRG 822074D0…97AA4, parser PRG 7A2E1B90…4128FBA.
• Remaining risks: a wall-clock timeout on a heavily loaded host can kill a healthy suite, and the message then reports the timeout rather than the fixture; MAP hashes track the cc65 version, so a toolchain upgrade legitimately changes them and requires a reviewed -UpdateBaseline; the other 21 suites have no pinned artifacts yet, so a silent byte-level change in them would not be caught.