C64-dev/newLang/planning/claims/E02-S02.md

---

story: E02-S02 owner: e02_lexer status: done started: 2026-11-07 depends-on: [E02-S01, E00-S02]

E02-S02 Claim

Intended change

Add the PETSCII lexer (src/compiler/lexer.s) with stable token kinds and error codes (src/common/tokens.inc), and prove it against golden token files that an independent host reference lexer also reproduces.

File area

• src/compiler/lexer.s, src/common/tokens.inc
• tests/lexer/ (fixtures, golden files, reference lexer, x128 test)
• scripts/build-lexer-test.ps1, scripts/verify-lexer-test.ps1, scripts/lib/Nova128Test.psm1 (monitor bank fix)
• docs/language-spec.md 2.2, docs/decisions.md (D119)
• planning/claims/E02-S02.md
• generated files under build/

Acceptance evidence

• Both supported PETSCII letter ranges compare case-insensitively: END Program WHILE wHiLe yields the same keywords; $ffFF (both ranges) is 65535; mixed-case identifiers normalize to $41-$5A. A mutation that disables folding fails every fixture with a capitalized identifier.
• Keywords, identifiers, numbers, strings, operators, comments, EOL, and EOF match the specification: tests/lexer/tokens.n8 covers all 32 keywords used by the grammar, every operator including <>/<=/>=, decimal/hex/binary limits, all string and character escapes, tabs, blank and comment-only lines.
• Invalid bytes and overlong identifiers report exact positions: tests/lexer/errors.n8 has one error per line for all seven LEXERR_* codes (invalid &, ;, leading _, 17-character identifier, ranges, malformed numbers, unterminated string, bad escapes, bad character literals), each at its documented column, with recovery on the next line. The grammar fixture bad_char.n8 gives BAD_CHAR_LITERAL at 3:11, matching its pre-existing .expect sidecar.
• Token dump of a Vim-format fixture matches a golden expected file: the C128 lexer's dump for all 10 fixtures (1,119 tokens) matches tests/lexer/golden/*.tokens, which the host reference lexer also reproduces. Fixtures use Vim's exact PETSCII save format; a live Vim edit is E07-S03.

Coordination notes

• This claim file was created after implementation had started, contrary to planning/agent-workflow.md. No other agent was working in this area.
• Spec 2.2 omitted <=/>= from the punctuation list although section 8 and the grammar fixtures use them; 2.2 was corrected to match.
• The spec defines \n/\r escapes but not their bytes. Both produce $0D for now; recorded as D119 (Proposed) for user approval.
• init and destroy are identifiers, not keywords (spec 6.3 uses them as method names).

Work log

• Golden files were generated by the reference lexer and then reviewed line by line against the fixtures (error columns, escape bytes, EOL columns) before the C128 run.
• Fixed in the test harness: Read-Nova128Memory now reads RAM bank 0 (bank ram); the CPU view after return shows BASIC ROM at $4000, which had corrupted the dump read-back. The lexer test computed its mailbox fields from a private-zero-page pointer after mmu_shutdown; reordered.
• Found that the Bash tool collapses \\ and can turn \n into a newline, corrupting fixtures and docs; rewrote affected files with the Write tool and byte-checked them.

Completion

• Tests run: scripts/verify-lexer-test.ps1 (pass plus case-folding mutation), reference-lexer golden check in build-lexer-test.ps1.
• Results: 10/10 fixtures, 1,119 tokens match; mutation fails 9 fixtures as expected; source restored.
• Files changed: see file area.
• Remaining risks: D119 awaits approval; the lexer reports at most one error per line by design; $A0 (shifted space) and other PETSCII control bytes are invalid outside strings, which a Vim user could type by accident; lexer error codes still need user-facing Exxx IDs (E02-S04).