dcovington
/
C64-dev
1
0
Fork 0
Código Incidencias 0 Pull Requests 0 Lanzamientos 0 Wiki Actividad
Explorar el Código

newLang: add the PETSCII lexer (E02-S02) 

lexer.s turns cached source lines into tokens with stable kinds and error
codes (tokens.inc): 32 keywords, identifiers folded across both PETSCII
letter ranges, decimal/hex/binary numbers to 65535, strings and character
literals with escapes, all operators, and comments. An error reports its
exact column, then the rest of the line is skipped.

An independent host reference lexer and the C128 lexer in x128 both
reproduce the reviewed golden token files: 10 fixtures, 1,119 tokens,
including one fixture per error kind. A case-folding mutation is caught.

Spec 2.2 now lists <= and >=, matching section 8. D119 (proposed): the \n
and \r escapes both produce $0D. The monitor helper now reads RAM bank 0,
because BASIC ROM hides $4000+ after a program returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main
Daniel Covington hace 6 días
padre
5f16994c60
commit
37bf335da9
Se han modificado 22 ficheros con 2604 adiciones y 5 borrados
Dividir vista
Opciones de diferencias
Mostrar estadísticas Descargar archivo de parche Descargar archivo de diferencias
  1. +3
    -1
      newLang/docs/decisions.md
  2. +5
    -2
      newLang/docs/language-spec.md
  3. +85
    -0
      newLang/planning/claims/E02-S02.md
  4. +63
    -0
      newLang/scripts/build-lexer-test.ps1
  5. +6
    -2
      newLang/scripts/lib/Nova128Test.psm1
  6. +55
    -0
      newLang/scripts/verify-lexer-test.ps1
  7. +97
    -0
      newLang/src/common/tokens.inc
  8. +629
    -0
      newLang/src/compiler/lexer.s
  9. +21
    -0
      newLang/tests/lexer/errors.n8
  10. +132
    -0
      newLang/tests/lexer/golden/arrays.tokens
  11. +16
    -0
      newLang/tests/lexer/golden/badchar.tokens
  12. +207
    -0
      newLang/tests/lexer/golden/decls.tokens
  13. +82
    -0
      newLang/tests/lexer/golden/errors.tokens
  14. +180
    -0
      newLang/tests/lexer/golden/exprs.tokens
  15. +104
    -0
      newLang/tests/lexer/golden/forward.tokens
  16. +8
    -0
      newLang/tests/lexer/golden/minimal.tokens
  17. +63
    -0
      newLang/tests/lexer/golden/nulls.tokens
  18. +208
    -0
      newLang/tests/lexer/golden/stmts.tokens
  19. +119
    -0
      newLang/tests/lexer/golden/tokens.tokens
  20. +286
    -0
      newLang/tests/lexer/lexer-test.s
  21. +219
    -0
      newLang/tests/lexer/reference-lexer.ps1
  22. +16
    -0
      newLang/tests/lexer/tokens.n8

+ 3
- 1
newLang/docs/decisions.md Ver fichero

@@ -43,7 +43,9 @@ documented default. `Proposed` means a later change still needs approval.

## Proposed

No proposed product changes are currently pending.
| ID | Proposal | Consequence |
|---|---|---|
| D119 | The string and character escapes `\n` and `\r` both produce PETSCII `$0D`. | Commodore output has one line-end character, so both escapes start a new line. Implemented as `ESC_N_VALUE`/`ESC_R_VALUE` in `src/common/tokens.inc`; changing them later affects only those constants and the lexer golden files. |

## Deferred



+ 5
- 2
newLang/docs/language-spec.md Ver fichero

@@ -60,8 +60,11 @@ this specification defines its syntax, semantics, diagnostics, and tests.
- Character literals use single quotes and contain exactly one PETSCII
character or one supported escape: `\'`, `\\`, `\n`, or `\r`.
- Reserved punctuation is `(`, `)`, `[`, `]`, `,`, `.`, `:`, `=`, `+`, `-`,
`*`, `/`, `<`, `>`, `<>`, and the single quote that delimits character
literals.
`*`, `/`, `<`, `>`, `<>`, `<=`, `>=`, and the single quote that delimits
character literals. (`<=` and `>=` were missing from this list although
section 8 defines them; corrected in E02-S02.)
- `\n` and `\r` both produce PETSCII `$0D`, the Commodore line end (D119,
proposed).
- Boolean operators are the words `and`, `or`, and `not`; `&&` and `||` are
not aliases.



+ 85
- 0
newLang/planning/claims/E02-S02.md Ver fichero

@@ -0,0 +1,85 @@
---
story: E02-S02
owner: e02_lexer
status: done
started: 2026-11-07
depends-on: [E02-S01, E00-S02]
---

# E02-S02 Claim

## Intended change

Add the PETSCII lexer (`src/compiler/lexer.s`) with stable token kinds and
error codes (`src/common/tokens.inc`), and prove it against golden token files
that an independent host reference lexer also reproduces.

## File area

- `src/compiler/lexer.s`, `src/common/tokens.inc`
- `tests/lexer/` (fixtures, golden files, reference lexer, x128 test)
- `scripts/build-lexer-test.ps1`, `scripts/verify-lexer-test.ps1`,
`scripts/lib/Nova128Test.psm1` (monitor bank fix)
- `docs/language-spec.md` 2.2, `docs/decisions.md` (D119)
- `planning/claims/E02-S02.md`
- generated files under `build/`

## Acceptance evidence

- [x] Both supported PETSCII letter ranges compare case-insensitively:
`END Program WHILE wHiLe` yields the same keywords; `$ffFF` (both ranges)
is 65535; mixed-case identifiers normalize to `$41-$5A`. A mutation that
disables folding fails every fixture with a capitalized identifier.
- [x] Keywords, identifiers, numbers, strings, operators, comments, EOL, and
EOF match the specification: `tests/lexer/tokens.n8` covers all 32
keywords used by the grammar, every operator including `<>`/`<=`/`>=`,
decimal/hex/binary limits, all string and character escapes, tabs, blank
and comment-only lines.
- [x] Invalid bytes and overlong identifiers report exact positions:
`tests/lexer/errors.n8` has one error per line for all seven `LEXERR_*`
codes (invalid `&`, `;`, leading `_`, 17-character identifier, ranges,
malformed numbers, unterminated string, bad escapes, bad character
literals), each at its documented column, with recovery on the next line.
The grammar fixture `bad_char.n8` gives `BAD_CHAR_LITERAL` at 3:11,
matching its pre-existing `.expect` sidecar.
- [x] Token dump of a Vim-format fixture matches a golden expected file: the
C128 lexer's dump for all 10 fixtures (1,119 tokens) matches
`tests/lexer/golden/*.tokens`, which the host reference lexer also
reproduces. Fixtures use Vim's exact PETSCII save format; a live Vim edit
is E07-S03.

## Coordination notes

- This claim file was created after implementation had started, contrary to
`planning/agent-workflow.md`. No other agent was working in this area.
- Spec 2.2 omitted `<=`/`>=` from the punctuation list although section 8 and
the grammar fixtures use them; 2.2 was corrected to match.
- The spec defines `\n`/`\r` escapes but not their bytes. Both produce `$0D`
for now; recorded as D119 (Proposed) for user approval.
- `init` and `destroy` are identifiers, not keywords (spec 6.3 uses them as
method names).

## Work log

- Golden files were generated by the reference lexer and then reviewed line
by line against the fixtures (error columns, escape bytes, EOL columns)
before the C128 run.
- Fixed in the test harness: `Read-Nova128Memory` now reads RAM bank 0
(`bank ram`); the CPU view after return shows BASIC ROM at `$4000`, which
had corrupted the dump read-back. The lexer test computed its mailbox
fields from a private-zero-page pointer after `mmu_shutdown`; reordered.
- Found that the Bash tool collapses `\\` and can turn `\n` into a newline,
corrupting fixtures and docs; rewrote affected files with the Write tool
and byte-checked them.

## Completion

- Tests run: `scripts/verify-lexer-test.ps1` (pass plus case-folding
mutation), reference-lexer golden check in `build-lexer-test.ps1`.
- Results: 10/10 fixtures, 1,119 tokens match; mutation fails 9 fixtures as
expected; source restored.
- Files changed: see file area.
- Remaining risks: D119 awaits approval; the lexer reports at most one error
per line by design; `$A0` (shifted space) and other PETSCII control bytes
are invalid outside strings, which a Vim user could type by accident;
lexer error codes still need user-facing `Exxx` IDs (E02-S04).

+ 63
- 0
newLang/scripts/build-lexer-test.ps1 Ver fichero

@@ -0,0 +1,63 @@
# E02-S02: build the lexer test PRG and a fresh D81 holding the fixtures.
# The host reference lexer must reproduce tests/lexer/golden/*.tokens exactly;
# -UpdateGolden rewrites them instead (review the diff before committing).
[CmdletBinding()]
param([switch]$UpdateGolden)

$ErrorActionPreference = "Stop"
$project = Split-Path $PSScriptRoot -Parent
Import-Module (Join-Path $PSScriptRoot "lib\Nova128Test.psm1") -Force
. (Join-Path $project "tests\lexer\reference-lexer.ps1")

$out = Join-Path $project "build\lexer-test"
$golden = Join-Path $project "tests\lexer\golden"
New-Item -ItemType Directory -Force -Path $out, $golden | Out-Null

# Order must match the FIXTURE list in tests/lexer/lexer-test.s.
$grammar = Join-Path $project "tests\grammar"
$fixtures = [ordered]@{
"minimal.n8" = Join-Path $grammar "accepted\minimal.n8"
"decls.n8" = Join-Path $grammar "accepted\decls.n8"
"arrays.n8" = Join-Path $grammar "accepted\arrays.n8"
"exprs.n8" = Join-Path $grammar "accepted\exprs.n8"
"forward.n8" = Join-Path $grammar "accepted\forward.n8"
"nulls.n8" = Join-Path $grammar "accepted\nulls.n8"
"stmts.n8" = Join-Path $grammar "accepted\stmts.n8"
"tokens.n8" = Join-Path $project "tests\lexer\tokens.n8"
"errors.n8" = Join-Path $project "tests\lexer\errors.n8"
"badchar.n8" = Join-Path $grammar "rejected\bad_char.n8"
}

$link = Invoke-Nova128Link -Name "lexer-test" -BuildDir $out `
-Config (Join-Path $project "cfg\compiler.cfg") -Sources @(
(Join-Path $project "src\compiler\mmu.s"),
(Join-Path $project "src\compiler\diskio.s"),
(Join-Path $project "src\compiler\source.s"),
(Join-Path $project "src\compiler\lexer.s"),
(Join-Path $project "tests\lexer\lexer-test.s"))
if ((Get-Nova128Label $link.Labels "test_entry") -ne 0x1C0D) { throw "test_entry is not at `$1C0D" }

$files = @(@{ Name = "lexer-test"; Path = $link.Program; Type = "prg" })
$mismatch = @()
foreach ($name in $fixtures.Keys) {
$bytes = ConvertTo-Nova128Petscii (Get-Content -Raw -LiteralPath $fixtures[$name])
$pet = Join-Path $out ($name -replace '\.n8$', '.pet')
[IO.File]::WriteAllBytes($pet, $bytes)
$files += @{ Name = $name; Path = $pet; Type = "seq" }

$tokens = (Get-ReferenceTokens $bytes) -join "`n"
$goldenPath = Join-Path $golden ($name -replace '\.n8$', '.tokens')
if ($UpdateGolden) {
[IO.File]::WriteAllText($goldenPath, $tokens + "`n")
} elseif (-not (Test-Path -LiteralPath $goldenPath)) {
$mismatch += "$name (no golden file; run with -UpdateGolden and review)"
} elseif (([IO.File]::ReadAllText($goldenPath).TrimEnd("`n")) -ne $tokens) {
$mismatch += "$name (reference lexer differs from golden)"
}
}
if ($mismatch) { throw ("golden check failed:`n " + ($mismatch -join "`n ")) }

$disk = Join-Path $out "lexer-test.d81"
New-Nova128Disk -Path $disk -Files $files
Write-Host ("built {0}" -f $link.Program)
Write-Host ("disk {0} ({1} fixtures); reference lexer matches golden files" -f $disk, $fixtures.Count)

+ 6
- 2
newLang/scripts/lib/Nova128Test.psm1 Ver fichero

@@ -171,9 +171,13 @@ function Invoke-Nova128Monitor {
}

# Read memory through the monitor. Returns byte[] (length = Count).
# Reads RAM bank 0 by default, not the CPU's current view: after a program
# returns to BASIC, $4000-$BFFF shows BASIC ROM in the CPU view.
function Read-Nova128Memory {
param([Parameter(Mandatory)]$Session, [int]$Address, [int]$Count)
$text = Invoke-Nova128Monitor $Session ("m `${0:X4} `${1:X4}" -f $Address, ($Address + $Count - 1))
param([Parameter(Mandatory)]$Session, [int]$Address, [int]$Count, [string]$Bank = "ram")
Invoke-Nova128Monitor $Session "bank $Bank" | Out-Null
$timeout = 5000 + 2 * $Count # large dumps print slowly
$text = Invoke-Nova128Monitor $Session ("m `${0:X4} `${1:X4}" -f $Address, ($Address + $Count - 1)) -TimeoutMs $timeout
$memory = @{}
foreach ($row in [regex]::Matches($text, "(?i)C:([0-9a-f]{4})((?:[ ]{1,2}[0-9a-f]{2}){1,16})")) {
$a = [Convert]::ToInt32($row.Groups[1].Value, 16)


+ 55
- 0
newLang/scripts/verify-lexer-test.ps1 Ver fichero

@@ -0,0 +1,55 @@
# E02-S02: run the C128 lexer over every fixture in x128 and compare its token
# dump with tests/lexer/golden/*.tokens (which the host reference lexer must
# also reproduce; build-lexer-test.ps1 checks that first).
[CmdletBinding()]
param([int]$Port = 6532, [int]$TimeoutSeconds = 180)

$ErrorActionPreference = "Stop"
$project = Split-Path $PSScriptRoot -Parent
Import-Module (Join-Path $PSScriptRoot "lib\Nova128Test.psm1") -Force
. (Join-Path $project "tests\lexer\reference-lexer.ps1")
& (Join-Path $PSScriptRoot "build-lexer-test.ps1")

$out = Join-Path $project "build\lexer-test"
$disk = Join-Path $out "lexer-test.d81"
$screenshot = Join-Path $out "lexer-test-vicii.png"
$names = @("minimal", "decls", "arrays", "exprs", "forward", "nulls", "stmts",
"tokens", "errors", "badchar")

$session = Start-Nova128X128 -Autostart $disk -Drive8Image $disk -Port $Port -Screenshot $screenshot
try {
$mb = Wait-Nova128Mailbox $session -TimeoutSeconds $TimeoutSeconds
$address = $mb[8] + 256 * $mb[9]
$length = $mb[10] + 256 * $mb[11]
$dump = Read-Nova128Memory $session $address $length
} finally {
Stop-Nova128X128 $session
}
[IO.File]::WriteAllBytes((Join-Path $out "lexer-dump.bin"), $dump)

$fixtures = ConvertFrom-TokenDump $dump
$failures = @()
if ($mb[3] -ne 0xA5) { $failures += "C128 reported FAIL (code `$" + $mb[4].ToString("X2") + ")" }
if ($fixtures.Count -ne $names.Count) { $failures += "dump holds $($fixtures.Count) fixtures, expected $($names.Count)" }
for ($i = 0; $i -lt $names.Count; $i++) {
$name = $names[$i]
$golden = (Get-Content -LiteralPath (Join-Path $project "tests\lexer\golden\$name.tokens")) |
Where-Object { $_ -ne "" }
$entry = $fixtures[$i]
if ($null -eq $entry) { $failures += "${name}: missing from dump"; continue }
if ($entry.Load -ne 0) { $failures += "${name}: src_load returned $($entry.Load)"; continue }
$got = @($entry.Tokens)
$diff = Compare-Object -ReferenceObject $golden -DifferenceObject $got -SyncWindow 0
if ($diff) {
$first = 0
while ($first -lt [Math]::Min($golden.Count, $got.Count) -and $golden[$first] -eq $got[$first]) { $first++ }
$failures += ("{0}: token {1} differs: golden '{2}' vs C128 '{3}' ({4} vs {5} tokens)" -f
$name, ($first + 1), $golden[$first], $got[$first], $golden.Count, $got.Count)
Write-Host (" {0,-8} {1,4} tokens FAIL" -f $name, $got.Count)
} else {
Write-Host (" {0,-8} {1,4} tokens match" -f $name, $got.Count)
}
}
if ($failures) { throw ("lexer verification failed:`n " + ($failures -join "`n ")) }
Write-Host ("verified C128 lexer: {0} fixtures, {1} dump bytes, all tokens match golden files" -f $names.Count, $length)
Write-Host "screenshot: $screenshot"

+ 97
- 0
newLang/src/common/tokens.inc Ver fichero

@@ -0,0 +1,97 @@
; tokens.inc - Nova128 token kinds and lexer error codes (E02-S02).
;
; Values are stable: parsers, dumps, and golden files depend on them. Add new
; kinds at the end of their group; never renumber.

.ifndef TOKENS_INC
TOKENS_INC = 1

; ---------------------------------------------------------- basic kinds
TOK_EOF = $00
TOK_EOL = $01
TOK_IDENT = $02 ; tok_text = normalized name ($41-$5A, digits, $5F)
TOK_NUMBER = $03 ; tok_value = 0..65535
TOK_STRING = $04 ; tok_text = decoded bytes
TOK_CHAR = $05 ; tok_value = one PETSCII byte
TOK_ERROR = $06 ; tok_err = LEXERR_*; the rest of the line is skipped

; ---------------------------------------------------------- punctuation
TOK_LPAREN = $10
TOK_RPAREN = $11
TOK_LBRACKET = $12
TOK_RBRACKET = $13
TOK_COMMA = $14
TOK_DOT = $15
TOK_COLON = $16
TOK_EQ = $17 ; = (assignment and equality)
TOK_PLUS = $18
TOK_MINUS = $19
TOK_STAR = $1A
TOK_SLASH = $1B
TOK_LT = $1C
TOK_GT = $1D
TOK_NE = $1E ; <>
TOK_LE = $1F ; <=
TOK_GE = $20 ; >=

; ------------------------------------------------------------- keywords
; Ordered as in the lexer's keyword table (alphabetical).
TOK_KW_FIRST = $40
KW_AND = $40
KW_AS = $41
KW_BOOLEAN = $42
KW_BYTE = $43
KW_CHAR = $44
KW_CLASS = $45
KW_DELETE = $46
KW_ELSE = $47
KW_END = $48
KW_FALSE = $49
KW_FOR = $4A
KW_IF = $4B
KW_IMPLEMENTS = $4C
KW_INTEGER = $4D
KW_INTERFACE = $4E
KW_METHOD = $4F
KW_NEW = $50
KW_NOT = $51
KW_NULL = $52
KW_OR = $53
KW_PRINT = $54
KW_PRIVATE = $55
KW_PROGRAM = $56
KW_PUBLIC = $57
KW_RETURN = $58
KW_SELF = $59
KW_STEP = $5A
KW_TO = $5B
KW_TRUE = $5C
KW_VAR = $5D
KW_WHILE = $5E
KW_WORD = $5F
TOK_KW_LAST = $5F

; ------------------------------------------------------ lexer error codes
LEXERR_INVALID_CHAR = 1 ; byte cannot start any token
LEXERR_IDENT_TOO_LONG = 2 ; more than 16 identifier characters
LEXERR_NUMBER_RANGE = 3 ; value above 65535
LEXERR_BAD_NUMBER = 4 ; $/% without digits, or digits run into letters
LEXERR_UNTERMINATED_STRING = 5 ; end of line inside "..."
LEXERR_BAD_ESCAPE = 6 ; \ followed by an unsupported character
LEXERR_BAD_CHAR_LITERAL = 7 ; '' empty, several characters, or unterminated

; ------------------------------------------------------------- limits
LEX_IDENT_MAX = 16 ; significant identifier characters
LEX_TEXT_MAX = 80 ; tok_text capacity (one source line)

; ------------------------------------------------------- PETSCII bytes
PET_SPACE = $20
PET_TAB = $09
PET_QUOTE = $22
PET_APOS = $27
PET_BACKSLASH = $5C ; ASCII '\' (shown as a pound sign on the C128)
PET_UNDERSCORE = $5F ; ASCII '_' (shown as a left arrow)
ESC_N_VALUE = $0D ; "\n" (D119, proposed)
ESC_R_VALUE = $0D ; "\r" (D119, proposed)

.endif

+ 629
- 0
newLang/src/compiler/lexer.s Ver fichero

@@ -0,0 +1,629 @@
; lexer.s - Nova128 PETSCII lexer (E02-S02).
;
; Pulls lines from source.s and returns one token per lex_next call:
; tok_kind TOK_* / KW_* (src/common/tokens.inc)
; tok_line 1-based line (2 bytes) tok_col 1-based start column
; tok_len raw source length of the token (0 for EOL/EOF/ERROR)
; tok_value NUMBER value, or CHAR byte
; tok_text / tok_text_len IDENT: normalized name; STRING: decoded bytes
; tok_err LEXERR_* when tok_kind = TOK_ERROR
;
; Rules (docs/language-spec.md 2.2):
; - Letters $41-$5A and $C1-$DA are folded to $41-$5A in identifiers and
; keywords. Original spelling is src_line[tok_col-1 .. +tok_len].
; - Every line, including the last one and comment-only or blank lines,
; ends with one TOK_EOL whose column is line length + 1.
; - After a TOK_ERROR the rest of that line is skipped; TOK_EOL follows.
; - Error columns: the token's first character, except BAD_ESCAPE, which
; points at the backslash.

.setcpu "6502"
.include "../common/tokens.inc"

.import src_rewind, src_next_line, src_line, src_line_len, src_line_no

.export lex_reset, lex_next
.export tok_kind, tok_line, tok_col, tok_len, tok_value
.export tok_text, tok_text_len, tok_err

CC_BAD = 0
CC_SPACE = 1
CC_LETTER = 2
CC_DIGIT = 3
CC_UNDER = 4
CC_PUNCT = 5
CC_QUOTE = 6
CC_APOS = 7
CC_DOLLAR = 8
CC_PERCENT = 9

; ------------------------------------------------------------------ CODE
.segment "CODE"

; Start lexing from the first source line.
lex_reset:
jsr src_rewind
lda #$00
sta lx_have_line
sta lx_skip
rts

lex_next:
lda #$00
sta tok_len
sta tok_text_len
sta tok_value
sta tok_value+1
sta tok_err
lda lx_have_line
bne @have
jsr src_next_line
bcc @fresh
lda #TOK_EOF ; line stays at the last line delivered
sta tok_kind
lda src_line_no
sta tok_line
lda src_line_no+1
sta tok_line+1
lda #$00
sta tok_col
rts
@fresh: lda #$01
sta lx_have_line
lda #$00
sta lx_pos
sta lx_skip
@have: lda src_line_no
sta tok_line
lda src_line_no+1
sta tok_line+1
lda lx_skip
bne @eol
ldy lx_pos
@ws: jsr getc
tax
lda cclass,x
cmp #CC_SPACE
bne @start
iny
bne @ws
@start: cpx #$0D ; getc sentinel: end of line
beq @eol
sty lx_start
iny
sty tok_col ; 1-based start column
stx lx_ch
asl a ; dispatch on character class
tax
lda scan_table+1,x
pha
lda scan_table,x
pha
lda lx_ch
rts ; jumps to scan_* with Y = start + 1

@eol: lda #TOK_EOL
sta tok_kind
ldx src_line_len
inx
stx tok_col
lda #$00
sta lx_have_line
rts

scan_table:
.word scan_bad - 1, scan_bad - 1, scan_ident - 1, scan_dec - 1
.word scan_bad - 1, scan_punct - 1, scan_string - 1, scan_char - 1
.word scan_hex - 1, scan_bin - 1

; A = character at Y, or $0D past the end of the line. Preserves X/Y.
getc: cpy src_line_len
bcs @end
lda src_line,y
rts
@end: lda #$0D
rts

; Finish a good token: tok_kind = A, lx_pos = Y, tok_len = Y - start.
finish: sta tok_kind
sty lx_pos
tya
sec
sbc lx_start
sta tok_len
rts

; Report an error at tok_col: tok_err = A, skip the rest of the line.
fail: sta tok_err
lda #TOK_ERROR
sta tok_kind
lda #$00
sta tok_len
lda #$01
sta lx_skip
rts

scan_bad:
lda #LEXERR_INVALID_CHAR
jmp fail

; ------------------------------------------------------------ identifiers
scan_ident:
dey ; rescan from the first letter
lda #$00
sta tok_text_len
@loop: jsr getc
sta lx_ch
tax
lda cclass,x
cmp #CC_LETTER
beq @letter
cmp #CC_DIGIT
beq @keep
cmp #CC_UNDER
bne @done
beq @keep
@letter:
lda lx_ch
and #$7F ; $C1-$DA -> $41-$5A
sta lx_ch
@keep: ldx tok_text_len
cpx #LEX_IDENT_MAX
bcs @over
lda lx_ch
sta tok_text,x
@over: inc tok_text_len
iny
bne @loop
@done: lda tok_text_len
cmp #LEX_IDENT_MAX + 1
bcc @fits
lda #LEXERR_IDENT_TOO_LONG
jmp fail
@fits: jsr keyword_lookup ; A = KW_* or TOK_IDENT
jmp finish

; Compare tok_text with the keyword table. A = kind. Preserves Y.
keyword_lookup:
sty lx_save_y
ldx #$00 ; X = offset of the current entry
@entry: lda keywords,x
beq @ident ; end of table
cmp tok_text_len
bne @skip
ldy #$00
@cmp: lda keywords+1,x
cmp tok_text,y
bne @skip_from_y
inx
iny
cpy tok_text_len
bne @cmp
lda keywords+1,x ; kind follows the letters
ldy lx_save_y
rts
@skip_from_y: ; rewind X to the entry start
stx lx_tmp
tya
eor #$FF
sec
adc lx_tmp
tax
@skip: lda keywords,x ; next entry: 1 + length + 1 bytes
clc
adc #$02
stx lx_tmp
adc lx_tmp
tax
bne @entry
@ident: lda #TOK_IDENT
ldy lx_save_y
rts

; ------------------------------------------------------------- numbers
; Decimal: digits only; a letter or _ directly after them is BAD_NUMBER.
scan_dec:
dey
lda #$00
sta tok_value
sta tok_value+1
sta lx_ovf
@loop: jsr getc
tax
lda cclass,x
cmp #CC_DIGIT
bne @end
txa
sec
sbc #'0'
sta lx_digit
jsr times10
lda lx_digit
clc
adc tok_value
sta tok_value
bcc @next
inc tok_value+1
bne @next
inc lx_ovf
@next: iny
bne @loop
@end: jsr check_tail
bcs @bad
lda lx_ovf
beq @ok
lda #LEXERR_NUMBER_RANGE
jmp fail
@ok: lda #TOK_NUMBER
jmp finish
@bad: lda #LEXERR_BAD_NUMBER
jmp fail

; tok_value *= 10, setting lx_ovf on overflow past 16 bits.
times10:
lda tok_value ; v*2
asl a
sta lx_tmp
lda tok_value+1
rol a
sta lx_tmp+1
bcc :+
inc lx_ovf
: lda lx_tmp ; v*8 = (v*2)*4
sta tok_value
lda lx_tmp+1
sta tok_value+1
ldx #2
@shift: asl tok_value
rol tok_value+1
bcc :+
inc lx_ovf
: dex
bne @shift
lda tok_value ; v*8 + v*2
clc
adc lx_tmp
sta tok_value
lda tok_value+1
adc lx_tmp+1
sta tok_value+1
bcc :+
inc lx_ovf
: rts

; C=1 when the character at Y continues an identifier (letter, digit, _).
check_tail:
jsr getc
tax
lda cclass,x
cmp #CC_LETTER
beq @yes
cmp #CC_DIGIT
beq @yes
cmp #CC_UNDER
beq @yes
clc
rts
@yes: sec
rts

; Hexadecimal: $ then hex digits (A-F in either letter range).
scan_hex:
lda #$00
sta tok_value
sta tok_value+1
sta lx_ovf
sta lx_count
@loop: jsr getc
jsr hex_digit
bcc @end
sta lx_digit
lda tok_value+1 ; the top nibble must be clear to shift
and #$F0
beq :+
inc lx_ovf
: ldx #4
@shift: asl tok_value
rol tok_value+1
dex
bne @shift
lda tok_value
ora lx_digit
sta tok_value
inc lx_count
iny
bne @loop
@end: lda lx_count
beq scan_bad_number
jsr check_tail
bcs scan_bad_number
lda lx_ovf
bne scan_range
lda #TOK_NUMBER
jmp finish
scan_bad_number:
lda #LEXERR_BAD_NUMBER
jmp fail
scan_range:
lda #LEXERR_NUMBER_RANGE
jmp fail

; C=1 and A = 0..15 when A is a hex digit.
hex_digit:
cmp #'0'
bcc @no
cmp #'9' + 1
bcs @letter
sbc #'0' - 1 ; C=0 here: subtract '0'
sec
rts
@letter:
and #$7F ; fold $C1-$C6 to $41-$46
cmp #'A'
bcc @no
cmp #'F' + 1
bcs @no
sbc #'A' - 11 ; C=0: A - 'A' + 10
sec
rts
@no: clc
rts

; Binary: % then 0/1 digits.
scan_bin:
lda #$00
sta tok_value
sta tok_value+1
sta lx_ovf
sta lx_count
@loop: jsr getc
cmp #'0'
beq @bit
cmp #'1'
bne @end
@bit: lsr a ; C = the bit ('0'=$30, '1'=$31)
rol tok_value
rol tok_value+1
bcc :+
inc lx_ovf
: inc lx_count
iny
bne @loop
@end: lda lx_count
beq scan_bad_number
jsr check_tail
bcs scan_bad_number
lda lx_ovf
bne scan_range
lda #TOK_NUMBER
jmp finish

; --------------------------------------------------- strings and chars
scan_string:
lda #PET_QUOTE ; strings allow \" but not \'
sta lx_esc_quote
lda #$00
sta tok_text_len
@loop: jsr getc
cmp #$0D
beq @open
cmp #PET_QUOTE
beq @close
cmp #PET_BACKSLASH
bne @store
jsr escape ; A = decoded byte or C=1 bad escape
bcs @bad_escape
@store: ldx tok_text_len
sta tok_text,x
inc tok_text_len
iny
bne @loop
@close: iny
lda #TOK_STRING
jmp finish
@open: lda #LEXERR_UNTERMINATED_STRING
jmp fail
@bad_escape:
lda #LEXERR_BAD_ESCAPE
jmp fail

; Y at a backslash. Allowed after it: backslash, n, r, and lx_esc_quote
; (" in strings, ' in character literals). On success: C=0, A = byte, Y at
; the escape letter. On failure: C=1 and tok_col moved to the backslash.
escape: sty lx_tmp
iny
jsr getc
cmp #PET_BACKSLASH
beq @ok
cmp lx_esc_quote
beq @ok
and #$7F ; n/N and r/R in either letter range
cmp #'N'
beq @n
cmp #'R'
beq @r
ldx lx_tmp ; error column = the backslash
inx
stx tok_col
sec
rts
@n: lda #ESC_N_VALUE
clc
rts
@r: lda #ESC_R_VALUE
@ok: clc
rts

scan_char:
lda #PET_APOS ; character literals allow \' but not \"
sta lx_esc_quote
jsr getc
cmp #$0D
beq @bad
cmp #PET_APOS ; '' is empty
beq @bad
cmp #PET_BACKSLASH
bne @plain
jsr escape
bcs @bad_escape
@plain: sta tok_value
iny
jsr getc
cmp #PET_APOS
bne @bad
iny
lda #TOK_CHAR
jmp finish
@bad: lda #LEXERR_BAD_CHAR_LITERAL
jmp fail
@bad_escape:
lda #LEXERR_BAD_ESCAPE
jmp fail

; ---------------------------------------------------------- punctuation
scan_punct:
ldx #punct_count - 1
@find: cmp punct_chars,x
beq @found
dex
bpl @find
jmp scan_bad ; cannot happen: class table and list agree
@found: lda punct_kinds,x
cmp #TOK_SLASH
beq @slash
cmp #TOK_LT
beq @lt
cmp #TOK_GT
beq @gt
@one: jmp finish
@slash: pha
jsr getc
cmp #'/'
bne @single
pla ; comment: skip to end of line
lda #$01
sta lx_skip
jmp lex_next
@lt: pha
jsr getc
cmp #'>'
beq @ne
cmp #'='
beq @le
@single:
pla
jmp finish
@ne: pla
iny
lda #TOK_NE
jmp finish
@le: pla
iny
lda #TOK_LE
jmp finish
@gt: pha
jsr getc
cmp #'='
bne @single
pla
iny
lda #TOK_GE
jmp finish

.segment "RODATA"
punct_chars:
.byte "()[],.:=+-*/<>"
punct_count = * - punct_chars
punct_kinds:
.byte TOK_LPAREN, TOK_RPAREN, TOK_LBRACKET, TOK_RBRACKET, TOK_COMMA
.byte TOK_DOT, TOK_COLON, TOK_EQ, TOK_PLUS, TOK_MINUS, TOK_STAR
.byte TOK_SLASH, TOK_LT, TOK_GT
.assert * - punct_kinds = punct_count, error, "punctuation tables differ"

; Keyword table: length, normalized letters, kind; 0 ends the table.
.macro KW name, kind
.byte .strlen(name), name, kind
.endmacro
keywords:
KW "AND", KW_AND
KW "AS", KW_AS
KW "BOOLEAN", KW_BOOLEAN
KW "BYTE", KW_BYTE
KW "CHAR", KW_CHAR
KW "CLASS", KW_CLASS
KW "DELETE", KW_DELETE
KW "ELSE", KW_ELSE
KW "END", KW_END
KW "FALSE", KW_FALSE
KW "FOR", KW_FOR
KW "IF", KW_IF
KW "IMPLEMENTS", KW_IMPLEMENTS
KW "INTEGER", KW_INTEGER
KW "INTERFACE", KW_INTERFACE
KW "METHOD", KW_METHOD
KW "NEW", KW_NEW
KW "NOT", KW_NOT
KW "NULL", KW_NULL
KW "OR", KW_OR
KW "PRINT", KW_PRINT
KW "PRIVATE", KW_PRIVATE
KW "PROGRAM", KW_PROGRAM
KW "PUBLIC", KW_PUBLIC
KW "RETURN", KW_RETURN
KW "SELF", KW_SELF
KW "STEP", KW_STEP
KW "TO", KW_TO
KW "TRUE", KW_TRUE
KW "VAR", KW_VAR
KW "WHILE", KW_WHILE
KW "WORD", KW_WORD
.byte 0
.assert * - keywords < 256, error, "keyword table must stay below 256 bytes"

; Character classes for every PETSCII byte.
cclass:
.repeat 256, I
.if I = PET_SPACE .or I = PET_TAB
.byte CC_SPACE
.elseif (I >= $41 .and I <= $5A) .or (I >= $C1 .and I <= $DA)
.byte CC_LETTER
.elseif I >= '0' .and I <= '9'
.byte CC_DIGIT
.elseif I = PET_UNDERSCORE
.byte CC_UNDER
.elseif I = PET_QUOTE
.byte CC_QUOTE
.elseif I = PET_APOS
.byte CC_APOS
.elseif I = '$'
.byte CC_DOLLAR
.elseif I = '%'
.byte CC_PERCENT
.elseif I = '(' .or I = ')' .or I = '[' .or I = ']' .or I = ',' .or I = '.' .or I = ':' .or I = '=' .or I = '+' .or I = '-' .or I = '*' .or I = '/' .or I = '<' .or I = '>'
.byte CC_PUNCT
.else
.byte CC_BAD
.endif
.endrepeat

; ------------------------------------------------------------------- BSS
.segment "BSS"
tok_kind: .res 1
tok_line: .res 2
tok_col: .res 1
tok_len: .res 1
tok_value: .res 2
tok_err: .res 1
tok_text_len: .res 1
tok_text: .res LEX_TEXT_MAX
lx_have_line: .res 1
lx_skip: .res 1
lx_pos: .res 1
lx_start: .res 1
lx_ch: .res 1
lx_ovf: .res 1
lx_digit: .res 1
lx_count: .res 1
lx_save_y: .res 1
lx_esc_quote: .res 1
lx_tmp: .res 2

+ 21
- 0
newLang/tests/lexer/errors.n8 Ver fichero

@@ -0,0 +1,21 @@
// Lexer errors: one per line; each line after the error is skipped.
a = 1 & 2
abcdefghijklmnopq = 1
n = 65536
n = $10000
n = %11111111111111111
n = $
n = $g1
n = 12ab
n = %102
s = "open
s = "bad \q escape"
s = "no \' here"
c = ''
c = 'ab'
c = 'a
c = '\"'
c = '\q'
_x = 1
x = 1 ; y = 2
ok = $1F + %11 // lexing resumes on the next line

+ 132
- 0
newLang/tests/lexer/golden/arrays.tokens Ver fichero

@@ -0,0 +1,132 @@
1:1 KW_CLASS 5
1:7 IDENT 6 BUFFER
1:13 EOL
2:3 KW_PRIVATE 7
2:11 KW_BYTE 4
2:15 LBRACKET 1
2:16 NUMBER 1 4
2:17 RBRACKET 1
2:19 IDENT 4 DATA
2:23 EOL
3:1 EOL
4:3 KW_PUBLIC 6
4:10 KW_METHOD 6
4:17 IDENT 3 SET
4:20 LPAREN 1
4:21 IDENT 5 INDEX
4:27 KW_AS 2
4:30 KW_WORD 4
4:34 COMMA 1
4:36 IDENT 5 VALUE
4:42 KW_AS 2
4:45 KW_BYTE 4
4:49 RPAREN 1
4:50 EOL
5:5 KW_SELF 4
5:9 DOT 1
5:10 IDENT 4 DATA
5:14 LBRACKET 1
5:15 IDENT 5 INDEX
5:20 RBRACKET 1
5:22 EQ 1
5:24 IDENT 5 VALUE
5:29 EOL
6:3 KW_END 3
6:7 KW_METHOD 6
6:13 EOL
7:1 EOL
8:3 KW_PUBLIC 6
8:10 KW_METHOD 6
8:17 IDENT 3 GET
8:20 LPAREN 1
8:21 IDENT 5 INDEX
8:27 KW_AS 2
8:30 KW_WORD 4
8:34 RPAREN 1
8:36 KW_AS 2
8:39 KW_BYTE 4
8:43 EOL
9:5 KW_RETURN 6
9:12 KW_SELF 4
9:16 DOT 1
9:17 IDENT 4 DATA
9:21 LBRACKET 1
9:22 IDENT 5 INDEX
9:27 RBRACKET 1
9:28 EOL
10:3 KW_END 3
10:7 KW_METHOD 6
10:13 EOL
11:1 KW_END 3
11:5 KW_CLASS 5
11:10 EOL
12:1 EOL
13:1 KW_PROGRAM 7
13:9 IDENT 4 MAIN
13:13 EOL
14:3 KW_VAR 3
14:7 IDENT 6 BUFFER
14:14 KW_AS 2
14:17 IDENT 6 BUFFER
14:23 EOL
15:3 KW_VAR 3
15:7 IDENT 6 VALUES
15:14 KW_AS 2
15:17 KW_WORD 4
15:21 LBRACKET 1
15:22 NUMBER 1 4
15:23 RBRACKET 1
15:24 EOL
16:3 KW_VAR 3
16:7 IDENT 4 ITEM
16:12 KW_AS 2
16:15 KW_BYTE 4
16:19 EOL
17:3 IDENT 6 BUFFER
17:10 EQ 1
17:12 KW_NEW 3
17:16 IDENT 6 BUFFER
17:22 LPAREN 1
17:23 RPAREN 1
17:24 EOL
18:3 IDENT 6 VALUES
18:9 LBRACKET 1
18:10 NUMBER 1 0
18:11 RBRACKET 1
18:13 EQ 1
18:15 NUMBER 1 1
18:16 EOL
19:3 IDENT 6 BUFFER
19:9 DOT 1
19:10 IDENT 3 SET
19:13 LPAREN 1
19:14 IDENT 6 VALUES
19:20 LBRACKET 1
19:21 NUMBER 1 0
19:22 RBRACKET 1
19:23 COMMA 1
19:25 NUMBER 1 7
19:26 RPAREN 1
19:27 EOL
20:3 IDENT 4 ITEM
20:8 EQ 1
20:10 IDENT 6 BUFFER
20:16 DOT 1
20:17 IDENT 3 GET
20:20 LPAREN 1
20:21 IDENT 6 VALUES
20:27 LBRACKET 1
20:28 NUMBER 1 0
20:29 RBRACKET 1
20:30 RPAREN 1
20:31 EOL
21:3 KW_PRINT 5
21:9 IDENT 4 ITEM
21:13 EOL
22:3 KW_DELETE 6
22:10 IDENT 6 BUFFER
22:16 EOL
23:1 KW_END 3
23:5 KW_PROGRAM 7
23:12 EOL
23:0 EOF

+ 16
- 0
newLang/tests/lexer/golden/badchar.tokens Ver fichero

@@ -0,0 +1,16 @@
1:1 KW_PROGRAM 7
1:9 IDENT 4 MAIN
1:13 EOL
2:3 KW_VAR 3
2:7 IDENT 5 GLYPH
2:13 KW_AS 2
2:16 KW_CHAR 4
2:20 EOL
3:3 IDENT 5 GLYPH
3:9 EQ 1
3:11 ERROR BAD_CHAR_LITERAL
3:15 EOL
4:1 KW_END 3
4:5 KW_PROGRAM 7
4:12 EOL
4:0 EOF

+ 207
- 0
newLang/tests/lexer/golden/decls.tokens Ver fichero

@@ -0,0 +1,207 @@
1:1 KW_INTERFACE 9
1:11 IDENT 9 PRINTABLE
1:20 EOL
2:3 KW_METHOD 6
2:10 KW_PRINT 5
2:15 LPAREN 1
2:16 RPAREN 1
2:17 EOL
3:1 KW_END 3
3:5 KW_INTERFACE 9
3:14 EOL
4:1 EOL
5:1 KW_INTERFACE 9
5:11 IDENT 10 RESETTABLE
5:21 EOL
6:3 KW_METHOD 6
6:10 IDENT 5 RESET
6:15 LPAREN 1
6:16 RPAREN 1
6:17 EOL
7:3 KW_METHOD 6
7:10 IDENT 6 ACTIVE
7:16 LPAREN 1
7:17 RPAREN 1
7:19 KW_AS 2
7:22 KW_BOOLEAN 7
7:29 EOL
8:1 KW_END 3
8:5 KW_INTERFACE 9
8:14 EOL
9:1 EOL
10:1 KW_CLASS 5
10:7 IDENT 7 COUNTER
10:15 KW_IMPLEMENTS 10
10:26 IDENT 9 PRINTABLE
10:35 COMMA 1
10:37 IDENT 10 RESETTABLE
10:47 EOL
11:3 KW_PRIVATE 7
11:11 KW_WORD 4
11:16 IDENT 5 VALUE
11:21 EOL
12:3 KW_PRIVATE 7
12:11 KW_BYTE 4
12:15 LBRACKET 1
12:16 NUMBER 1 4
12:17 RBRACKET 1
12:19 IDENT 7 HISTORY
12:26 EOL
13:1 EOL
14:3 KW_PUBLIC 6
14:10 KW_METHOD 6
14:17 IDENT 4 INIT
14:21 LPAREN 1
14:22 IDENT 5 START
14:28 KW_AS 2
14:31 KW_WORD 4
14:35 RPAREN 1
14:36 EOL
15:5 KW_VAR 3
15:9 IDENT 4 COPY
15:14 KW_AS 2
15:17 KW_WORD 4
15:21 EOL
16:5 IDENT 4 COPY
16:10 EQ 1
16:12 IDENT 5 START
16:17 EOL
17:5 KW_SELF 4
17:9 DOT 1
17:10 IDENT 5 VALUE
17:16 EQ 1
17:18 IDENT 4 COPY
17:22 EOL
18:5 KW_SELF 4
18:9 DOT 1
18:10 IDENT 7 HISTORY
18:17 LBRACKET 1
18:18 NUMBER 1 0
18:19 RBRACKET 1
18:21 EQ 1
18:23 NUMBER 1 0
18:24 EOL
19:3 KW_END 3
19:7 KW_METHOD 6
19:13 EOL
20:1 EOL
21:3 KW_PUBLIC 6
21:10 KW_METHOD 6
21:17 IDENT 7 DESTROY
21:24 LPAREN 1
21:25 RPAREN 1
21:26 EOL
22:5 KW_SELF 4
22:9 DOT 1
22:10 IDENT 5 VALUE
22:16 EQ 1
22:18 NUMBER 1 0
22:19 EOL
23:5 KW_RETURN 6
23:11 EOL
24:3 KW_END 3
24:7 KW_METHOD 6
24:13 EOL
25:1 EOL
26:3 KW_PUBLIC 6
26:10 KW_METHOD 6
26:17 KW_PRINT 5
26:22 LPAREN 1
26:23 RPAREN 1
26:24 EOL
27:5 KW_PRINT 5
27:11 KW_SELF 4
27:15 DOT 1
27:16 IDENT 5 VALUE
27:21 EOL
28:3 KW_END 3
28:7 KW_METHOD 6
28:13 EOL
29:1 EOL
30:3 KW_PUBLIC 6
30:10 KW_METHOD 6
30:17 IDENT 5 RESET
30:22 LPAREN 1
30:23 RPAREN 1
30:24 EOL
31:5 KW_SELF 4
31:9 DOT 1
31:10 IDENT 5 VALUE
31:16 EQ 1
31:18 NUMBER 1 0
31:19 EOL
32:3 KW_END 3
32:7 KW_METHOD 6
32:13 EOL
33:1 EOL
34:3 KW_PUBLIC 6
34:10 KW_METHOD 6
34:17 IDENT 6 ACTIVE
34:23 LPAREN 1
34:24 RPAREN 1
34:26 KW_AS 2
34:29 KW_BOOLEAN 7
34:36 EOL
35:5 KW_IF 2
35:8 KW_SELF 4
35:12 DOT 1
35:13 IDENT 5 VALUE
35:19 NE 2
35:22 NUMBER 1 0
35:23 EOL
36:7 KW_RETURN 6
36:14 KW_TRUE 4
36:18 EOL
37:5 KW_ELSE 4
37:9 EOL
38:7 KW_RETURN 6
38:14 KW_FALSE 5
38:19 EOL
39:5 KW_END 3
39:9 KW_IF 2
39:11 EOL
40:3 KW_END 3
40:7 KW_METHOD 6
40:13 EOL
41:1 KW_END 3
41:5 KW_CLASS 5
41:10 EOL
42:1 EOL
43:1 KW_PROGRAM 7
43:9 IDENT 4 MAIN
43:13 EOL
44:3 KW_VAR 3
44:7 IDENT 5 COUNT
44:13 KW_AS 2
44:16 IDENT 7 COUNTER
44:23 EOL
45:3 KW_VAR 3
45:7 IDENT 6 OUTPUT
45:14 KW_AS 2
45:17 IDENT 9 PRINTABLE
45:26 EOL
46:3 IDENT 5 COUNT
46:9 EQ 1
46:11 KW_NEW 3
46:15 IDENT 7 COUNTER
46:22 LPAREN 1
46:23 NUMBER 2 10
46:25 RPAREN 1
46:26 EOL
47:3 IDENT 6 OUTPUT
47:10 EQ 1
47:12 IDENT 5 COUNT
47:17 EOL
48:3 IDENT 6 OUTPUT
48:9 DOT 1
48:10 KW_PRINT 5
48:15 LPAREN 1
48:16 RPAREN 1
48:17 EOL
49:3 KW_DELETE 6
49:10 IDENT 6 OUTPUT
49:16 EOL
50:1 KW_END 3
50:5 KW_PROGRAM 7
50:12 EOL
50:0 EOF

+ 82
- 0
newLang/tests/lexer/golden/errors.tokens Ver fichero

@@ -0,0 +1,82 @@
1:69 EOL
2:1 IDENT 1 A
2:3 EQ 1
2:5 NUMBER 1 1
2:7 ERROR INVALID_CHAR
2:10 EOL
3:1 ERROR IDENT_TOO_LONG
3:22 EOL
4:1 IDENT 1 N
4:3 EQ 1
4:5 ERROR NUMBER_RANGE
4:10 EOL
5:1 IDENT 1 N
5:3 EQ 1
5:5 ERROR NUMBER_RANGE
5:11 EOL
6:1 IDENT 1 N
6:3 EQ 1
6:5 ERROR NUMBER_RANGE
6:23 EOL
7:1 IDENT 1 N
7:3 EQ 1
7:5 ERROR BAD_NUMBER
7:6 EOL
8:1 IDENT 1 N
8:3 EQ 1
8:5 ERROR BAD_NUMBER
8:8 EOL
9:1 IDENT 1 N
9:3 EQ 1
9:5 ERROR BAD_NUMBER
9:9 EOL
10:1 IDENT 1 N
10:3 EQ 1
10:5 ERROR BAD_NUMBER
10:9 EOL
11:1 IDENT 1 S
11:3 EQ 1
11:5 ERROR UNTERMINATED_STRING
11:10 EOL
12:1 IDENT 1 S
12:3 EQ 1
12:10 ERROR BAD_ESCAPE
12:20 EOL
13:1 IDENT 1 S
13:3 EQ 1
13:9 ERROR BAD_ESCAPE
13:17 EOL
14:1 IDENT 1 C
14:3 EQ 1
14:5 ERROR BAD_CHAR_LITERAL
14:7 EOL
15:1 IDENT 1 C
15:3 EQ 1
15:5 ERROR BAD_CHAR_LITERAL
15:9 EOL
16:1 IDENT 1 C
16:3 EQ 1
16:5 ERROR BAD_CHAR_LITERAL
16:7 EOL
17:1 IDENT 1 C
17:3 EQ 1
17:6 ERROR BAD_ESCAPE
17:9 EOL
18:1 IDENT 1 C
18:3 EQ 1
18:6 ERROR BAD_ESCAPE
18:9 EOL
19:1 ERROR INVALID_CHAR
19:7 EOL
20:1 IDENT 1 X
20:3 EQ 1
20:5 NUMBER 1 1
20:7 ERROR INVALID_CHAR
20:14 EOL
21:1 IDENT 2 OK
21:4 EQ 1
21:6 NUMBER 3 31
21:10 PLUS 1
21:12 NUMBER 3 3
21:50 EOL
21:0 EOF

+ 180
- 0
newLang/tests/lexer/golden/exprs.tokens Ver fichero

@@ -0,0 +1,180 @@
1:1 KW_CLASS 5
1:7 IDENT 7 FORMULA
1:14 EOL
2:3 KW_PUBLIC 6
2:10 KW_METHOD 6
2:17 IDENT 4 EVAL
2:21 LPAREN 1
2:22 IDENT 1 A
2:24 KW_AS 2
2:27 KW_INTEGER 7
2:34 COMMA 1
2:36 IDENT 1 B
2:38 KW_AS 2
2:41 KW_INTEGER 7
2:48 RPAREN 1
2:50 KW_AS 2
2:53 KW_INTEGER 7
2:60 EOL
3:5 KW_VAR 3
3:9 IDENT 6 RESULT
3:16 KW_AS 2
3:19 KW_INTEGER 7
3:26 EOL
4:5 IDENT 6 RESULT
4:12 EQ 1
4:14 MINUS 1
4:15 LPAREN 1
4:16 IDENT 1 A
4:18 PLUS 1
4:20 IDENT 1 B
4:21 RPAREN 1
4:23 STAR 1
4:25 NUMBER 1 2
4:27 SLASH 1
4:29 NUMBER 1 1
4:30 EOL
5:5 KW_IF 2
5:8 KW_NOT 3
5:12 LPAREN 1
5:13 IDENT 1 A
5:15 EQ 1
5:17 IDENT 1 B
5:18 RPAREN 1
5:20 KW_AND 3
5:24 LPAREN 1
5:25 IDENT 1 A
5:27 NE 2
5:30 NUMBER 1 0
5:32 KW_OR 2
5:35 IDENT 1 B
5:37 GE 2
5:40 IDENT 1 A
5:41 RPAREN 1
5:42 EOL
6:7 IDENT 6 RESULT
6:14 EQ 1
6:16 IDENT 6 RESULT
6:23 PLUS 1
6:25 IDENT 1 A
6:27 MINUS 1
6:29 IDENT 1 B
6:30 EOL
7:5 KW_ELSE 4
7:9 EOL
8:7 IDENT 6 RESULT
8:14 EQ 1
8:16 NUMBER 1 0
8:17 EOL
9:5 KW_END 3
9:9 KW_IF 2
9:11 EOL
10:5 KW_IF 2
10:8 IDENT 1 A
10:10 LT 1
10:12 IDENT 1 B
10:13 EOL
11:7 IDENT 6 RESULT
11:14 EQ 1
11:16 IDENT 6 RESULT
11:23 PLUS 1
11:25 NUMBER 1 1
11:26 EOL
12:5 KW_END 3
12:9 KW_IF 2
12:11 EOL
13:5 KW_IF 2
13:8 IDENT 1 A
13:10 LE 2
13:13 IDENT 1 B
13:14 EOL
14:7 IDENT 6 RESULT
14:14 EQ 1
14:16 IDENT 6 RESULT
14:23 PLUS 1
14:25 NUMBER 1 1
14:26 EOL
15:5 KW_END 3
15:9 KW_IF 2
15:11 EOL
16:5 KW_IF 2
16:8 IDENT 1 A
16:10 GT 1
16:12 IDENT 1 B
16:13 EOL
17:7 IDENT 6 RESULT
17:14 EQ 1
17:16 IDENT 6 RESULT
17:23 MINUS 1
17:25 NUMBER 1 1
17:26 EOL
18:5 KW_END 3
18:9 KW_IF 2
18:11 EOL
19:5 KW_RETURN 6
19:12 IDENT 6 RESULT
19:18 EOL
20:3 KW_END 3
20:7 KW_METHOD 6
20:13 EOL
21:1 KW_END 3
21:5 KW_CLASS 5
21:10 EOL
22:1 EOL
23:1 KW_PROGRAM 7
23:9 IDENT 4 MAIN
23:13 EOL
24:3 KW_VAR 3
24:7 IDENT 7 FORMULA
24:15 KW_AS 2
24:18 IDENT 7 FORMULA
24:25 EOL
25:3 KW_VAR 3
25:7 IDENT 6 ANSWER
25:14 KW_AS 2
25:17 KW_INTEGER 7
25:24 EOL
26:3 KW_VAR 3
26:7 IDENT 5 GLYPH
26:13 KW_AS 2
26:16 KW_CHAR 4
26:20 EOL
27:3 IDENT 7 FORMULA
27:11 EQ 1
27:13 KW_NEW 3
27:17 IDENT 7 FORMULA
27:24 LPAREN 1
27:25 RPAREN 1
27:26 EOL
28:3 IDENT 6 ANSWER
28:10 EQ 1
28:12 IDENT 7 FORMULA
28:19 DOT 1
28:20 IDENT 4 EVAL
28:24 LPAREN 1
28:25 MINUS 1
28:26 NUMBER 1 2
28:27 COMMA 1
28:29 NUMBER 1 3
28:30 RPAREN 1
28:31 EOL
29:3 IDENT 5 GLYPH
29:9 EQ 1
29:11 CHAR 3 $C1
29:14 EOL
30:3 KW_PRINT 5
30:9 STRING 10 "answer{0D}"
30:19 EOL
31:3 KW_PRINT 5
31:9 IDENT 6 ANSWER
31:15 EOL
32:3 KW_PRINT 5
32:9 IDENT 5 GLYPH
32:14 EOL
33:3 KW_DELETE 6
33:10 IDENT 7 FORMULA
33:17 EOL
34:1 KW_END 3
34:5 KW_PROGRAM 7
34:12 EOL
34:0 EOF

+ 104
- 0
newLang/tests/lexer/golden/forward.tokens Ver fichero

@@ -0,0 +1,104 @@
1:1 KW_CLASS 5
1:7 IDENT 6 RUNNER
1:14 KW_IMPLEMENTS 10
1:25 IDENT 3 JOB
1:28 EOL
2:3 KW_PUBLIC 6
2:10 KW_METHOD 6
2:17 IDENT 7 EXECUTE
2:24 LPAREN 1
2:25 IDENT 4 CODE
2:30 KW_AS 2
2:33 KW_BYTE 4
2:37 RPAREN 1
2:39 KW_AS 2
2:42 KW_BOOLEAN 7
2:49 EOL
3:5 KW_IF 2
3:8 IDENT 4 CODE
3:13 EQ 1
3:15 NUMBER 1 0
3:16 EOL
4:7 KW_RETURN 6
4:14 KW_FALSE 5
4:19 EOL
5:5 KW_ELSE 4
5:9 EOL
6:7 KW_RETURN 6
6:14 KW_TRUE 4
6:18 EOL
7:5 KW_END 3
7:9 KW_IF 2
7:11 EOL
8:3 KW_END 3
8:7 KW_METHOD 6
8:13 EOL
9:1 KW_END 3
9:5 KW_CLASS 5
9:10 EOL
10:1 EOL
11:1 KW_INTERFACE 9
11:11 IDENT 3 JOB
11:14 EOL
12:3 KW_METHOD 6
12:10 IDENT 7 EXECUTE
12:17 LPAREN 1
12:18 IDENT 4 CODE
12:23 KW_AS 2
12:26 KW_BYTE 4
12:30 RPAREN 1
12:32 KW_AS 2
12:35 KW_BOOLEAN 7
12:42 EOL
13:1 KW_END 3
13:5 KW_INTERFACE 9
13:14 EOL
14:1 EOL
15:1 KW_PROGRAM 7
15:9 IDENT 4 MAIN
15:13 EOL
16:3 KW_VAR 3
16:7 IDENT 6 RUNNER
16:14 KW_AS 2
16:17 IDENT 6 RUNNER
16:23 EOL
17:3 KW_VAR 3
17:7 IDENT 3 JOB
17:11 KW_AS 2
17:14 IDENT 3 JOB
17:17 EOL
18:3 KW_VAR 3
18:7 IDENT 2 OK
18:10 KW_AS 2
18:13 KW_BOOLEAN 7
18:20 EOL
19:3 IDENT 6 RUNNER
19:10 EQ 1
19:12 KW_NEW 3
19:16 IDENT 6 RUNNER
19:22 LPAREN 1
19:23 RPAREN 1
19:24 EOL
20:3 IDENT 3 JOB
20:7 EQ 1
20:9 IDENT 6 RUNNER
20:15 EOL
21:3 IDENT 2 OK
21:6 EQ 1
21:8 IDENT 3 JOB
21:11 DOT 1
21:12 IDENT 7 EXECUTE
21:19 LPAREN 1
21:20 NUMBER 1 1
21:21 RPAREN 1
21:22 EOL
22:3 KW_PRINT 5
22:9 IDENT 2 OK
22:11 EOL
23:3 KW_DELETE 6
23:10 IDENT 3 JOB
23:13 EOL
24:1 KW_END 3
24:5 KW_PROGRAM 7
24:12 EOL
24:0 EOF

+ 8
- 0
newLang/tests/lexer/golden/minimal.tokens Ver fichero

@@ -0,0 +1,8 @@
1:48 EOL
2:1 KW_PROGRAM 7
2:9 IDENT 4 MAIN
2:13 EOL
3:1 KW_END 3
3:5 KW_PROGRAM 7
3:12 EOL
3:0 EOF

+ 63
- 0
newLang/tests/lexer/golden/nulls.tokens Ver fichero

@@ -0,0 +1,63 @@
1:1 KW_INTERFACE 9
1:11 IDENT 6 MARKER
1:17 EOL
2:3 KW_METHOD 6
2:10 IDENT 4 MARK
2:14 LPAREN 1
2:15 RPAREN 1
2:16 EOL
3:1 KW_END 3
3:5 KW_INTERFACE 9
3:14 EOL
4:1 EOL
5:1 KW_CLASS 5
5:7 IDENT 5 EMPTY
5:13 KW_IMPLEMENTS 10
5:24 IDENT 6 MARKER
5:30 EOL
6:3 KW_WORD 4
6:8 IDENT 5 VALUE
6:13 EOL
7:1 EOL
8:3 KW_METHOD 6
8:10 IDENT 4 MARK
8:14 LPAREN 1
8:15 RPAREN 1
8:16 EOL
9:5 KW_RETURN 6
9:11 EOL
10:3 KW_END 3
10:7 KW_METHOD 6
10:13 EOL
11:1 KW_END 3
11:5 KW_CLASS 5
11:10 EOL
12:1 EOL
13:1 KW_PROGRAM 7
13:9 IDENT 4 MAIN
13:13 EOL
14:3 KW_VAR 3
14:7 IDENT 6 OBJECT
14:14 KW_AS 2
14:17 IDENT 5 EMPTY
14:22 EOL
15:3 KW_VAR 3
15:7 IDENT 6 MARKER
15:14 KW_AS 2
15:17 IDENT 6 MARKER
15:23 EOL
16:3 IDENT 6 OBJECT
16:10 EQ 1
16:12 KW_NULL 4
16:16 EOL
17:3 IDENT 6 MARKER
17:10 EQ 1
17:12 KW_NULL 4
17:16 EOL
18:3 KW_DELETE 6
18:10 IDENT 6 MARKER
18:16 EOL
19:1 KW_END 3
19:5 KW_PROGRAM 7
19:12 EOL
19:0 EOF

+ 208
- 0
newLang/tests/lexer/golden/stmts.tokens Ver fichero

@@ -0,0 +1,208 @@
1:1 KW_CLASS 5
1:7 IDENT 6 WORKER
1:13 EOL
2:3 KW_PRIVATE 7
2:11 KW_WORD 4
2:16 IDENT 5 VALUE
2:21 EOL
3:1 EOL
4:3 KW_PUBLIC 6
4:10 KW_METHOD 6
4:17 IDENT 4 INIT
4:21 LPAREN 1
4:22 IDENT 5 START
4:28 KW_AS 2
4:31 KW_WORD 4
4:35 RPAREN 1
4:36 EOL
5:5 KW_SELF 4
5:9 DOT 1
5:10 IDENT 5 VALUE
5:16 EQ 1
5:18 IDENT 5 START
5:23 EOL
6:3 KW_END 3
6:7 KW_METHOD 6
6:13 EOL
7:1 EOL
8:3 KW_PUBLIC 6
8:10 KW_METHOD 6
8:17 IDENT 4 TICK
8:21 LPAREN 1
8:22 IDENT 6 AMOUNT
8:29 KW_AS 2
8:32 KW_WORD 4
8:36 RPAREN 1
8:37 EOL
9:5 KW_SELF 4
9:9 DOT 1
9:10 IDENT 5 VALUE
9:16 EQ 1
9:18 KW_SELF 4
9:22 DOT 1
9:23 IDENT 5 VALUE
9:29 PLUS 1
9:31 IDENT 6 AMOUNT
9:37 EOL
10:3 KW_END 3
10:7 KW_METHOD 6
10:13 EOL
11:1 EOL
12:3 KW_PUBLIC 6
12:10 KW_METHOD 6
12:17 IDENT 3 RUN
12:20 LPAREN 1
12:21 IDENT 5 LIMIT
12:27 KW_AS 2
12:30 KW_WORD 4
12:34 RPAREN 1
12:36 KW_AS 2
12:39 KW_WORD 4
12:43 EOL
13:5 KW_VAR 3
13:9 IDENT 1 I
13:11 KW_AS 2
13:14 KW_WORD 4
13:18 EOL
14:5 KW_VAR 3
14:9 IDENT 5 TOTAL
14:15 KW_AS 2
14:18 KW_WORD 4
14:22 EOL
15:5 IDENT 1 I
15:7 EQ 1
15:9 NUMBER 1 0
15:10 EOL
16:5 IDENT 5 TOTAL
16:11 EQ 1
16:13 NUMBER 1 0
16:14 EOL
17:5 KW_WHILE 5
17:11 IDENT 1 I
17:13 LT 1
17:15 IDENT 5 LIMIT
17:20 EOL
18:7 IDENT 5 TOTAL
18:13 EQ 1
18:15 IDENT 5 TOTAL
18:21 PLUS 1
18:23 IDENT 1 I
18:24 EOL
19:7 IDENT 1 I
19:9 EQ 1
19:11 IDENT 1 I
19:13 PLUS 1
19:15 NUMBER 1 1
19:16 EOL
20:5 KW_END 3
20:9 KW_WHILE 5
20:14 EOL
21:5 KW_IF 2
21:8 IDENT 5 TOTAL
21:14 GT 1
21:16 NUMBER 3 100
21:19 EOL
22:7 KW_PRINT 5
22:13 STRING 7 "large"
22:20 EOL
23:5 KW_ELSE 4
23:9 EOL
24:7 KW_PRINT 5
24:13 IDENT 5 TOTAL
24:18 EOL
25:5 KW_END 3
25:9 KW_IF 2
25:11 EOL
26:5 KW_FOR 3
26:9 IDENT 1 I
26:11 EQ 1
26:13 NUMBER 1 0
26:15 KW_TO 2
26:18 NUMBER 1 3
26:20 KW_STEP 4
26:25 NUMBER 1 1
26:26 EOL
27:7 KW_SELF 4
27:11 DOT 1
27:12 IDENT 4 TICK
27:16 LPAREN 1
27:17 IDENT 1 I
27:18 RPAREN 1
27:19 EOL
28:5 KW_END 3
28:9 KW_FOR 3
28:12 EOL
29:5 KW_FOR 3
29:9 IDENT 1 I
29:11 EQ 1
29:13 NUMBER 1 0
29:15 KW_TO 2
29:18 NUMBER 1 1
29:19 EOL
30:7 KW_SELF 4
30:11 DOT 1
30:12 IDENT 4 TICK
30:16 LPAREN 1
30:17 IDENT 1 I
30:18 RPAREN 1
30:19 EOL
31:5 KW_END 3
31:9 KW_FOR 3
31:12 EOL
32:5 KW_RETURN 6
32:12 IDENT 5 TOTAL
32:17 EOL
33:3 KW_END 3
33:7 KW_METHOD 6
33:13 EOL
34:1 KW_END 3
34:5 KW_CLASS 5
34:10 EOL
35:1 EOL
36:1 KW_PROGRAM 7
36:9 IDENT 4 MAIN
36:13 EOL
37:3 KW_VAR 3
37:7 IDENT 6 WORKER
37:14 KW_AS 2
37:17 IDENT 6 WORKER
37:23 EOL
38:3 KW_VAR 3
38:7 IDENT 5 TOTAL
38:13 KW_AS 2
38:16 KW_WORD 4
38:20 EOL
39:3 IDENT 6 WORKER
39:10 EQ 1
39:12 KW_NEW 3
39:16 IDENT 6 WORKER
39:22 LPAREN 1
39:23 NUMBER 1 1
39:24 RPAREN 1
39:25 EOL
40:3 IDENT 6 WORKER
40:9 DOT 1
40:10 IDENT 4 TICK
40:14 LPAREN 1
40:15 NUMBER 1 1
40:16 RPAREN 1
40:17 EOL
41:3 IDENT 5 TOTAL
41:9 EQ 1
41:11 IDENT 6 WORKER
41:17 DOT 1
41:18 IDENT 3 RUN
41:21 LPAREN 1
41:22 NUMBER 1 4
41:23 RPAREN 1
41:24 EOL
42:3 KW_PRINT 5
42:9 IDENT 5 TOTAL
42:14 EOL
43:3 KW_DELETE 6
43:10 IDENT 6 WORKER
43:16 EOL
44:1 KW_END 3
44:5 KW_PROGRAM 7
44:12 EOL
44:0 EOF

+ 119
- 0
newLang/tests/lexer/golden/tokens.tokens Ver fichero

@@ -0,0 +1,119 @@
1:75 EOL
2:1 KW_PROGRAM 7
2:9 IDENT 6 TOKENS
2:15 EOL
3:3 KW_VAR 3
3:7 IDENT 4 A_B1
3:12 KW_AS 2
3:15 KW_WORD 4
3:19 EOL
4:3 KW_VAR 3
4:7 IDENT 16 ABCDEFGHIJKLMNOP
4:24 KW_AS 2
4:27 KW_BYTE 4
4:31 EOL
5:3 IDENT 4 A_B1
5:8 EQ 1
5:10 NUMBER 5 65535
5:16 PLUS 1
5:18 NUMBER 5 10
5:24 PLUS 1
5:26 NUMBER 5 65535
5:32 PLUS 1
5:34 NUMBER 1 0
5:36 PLUS 1
5:38 NUMBER 2 0
5:41 PLUS 1
5:43 NUMBER 2 0
5:45 EOL
6:3 KW_END 3
6:7 KW_PROGRAM 7
6:15 KW_WHILE 5
6:21 KW_WHILE 5
6:26 EOL
7:3 IDENT 1 X
7:5 EQ 1
7:7 LPAREN 1
7:9 RPAREN 1
7:11 LBRACKET 1
7:13 RBRACKET 1
7:15 COMMA 1
7:17 DOT 1
7:19 COLON 1
7:21 EQ 1
7:23 PLUS 1
7:25 MINUS 1
7:27 STAR 1
7:29 SLASH 1
7:31 LT 1
7:33 GT 1
7:35 NE 2
7:38 LE 2
7:41 GE 2
7:43 EOL
8:3 IDENT 1 Y
8:4 EQ 1
8:5 IDENT 1 A
8:6 NE 2
8:8 IDENT 1 B
8:9 LE 2
8:11 IDENT 1 C
8:12 GE 2
8:14 IDENT 1 D
8:15 LT 1
8:16 IDENT 1 E
8:17 GT 1
8:18 IDENT 1 F
8:19 EOL
9:3 IDENT 1 S
9:5 EQ 1
9:7 STRING 11 "q{22}{5C}{0D}{0D}"
9:19 PLUS 1
9:21 STRING 2 ""
9:24 PLUS 1
9:26 STRING 7 "x ' y"
9:33 EOL
10:3 IDENT 1 C
10:5 EQ 1
10:7 CHAR 3 $58
10:11 PLUS 1
10:13 CHAR 4 $27
10:18 PLUS 1
10:20 CHAR 4 $5C
10:25 PLUS 1
10:27 CHAR 4 $0D
10:32 PLUS 1
10:34 CHAR 4 $0D
10:39 PLUS 1
10:41 CHAR 3 $22
10:44 EOL
11:1 EOL
12:2 IDENT 6 TABBED
12:9 EQ 1
12:11 NUMBER 1 1
12:29 EOL
13:3 IDENT 1 Z
13:5 EQ 1
13:7 NUMBER 1 7
13:9 SLASH 1
13:11 NUMBER 1 2
13:40 EOL
14:23 EOL
15:3 KW_SELF 4
15:7 DOT 1
15:8 IDENT 4 DATA
15:12 LBRACKET 1
15:13 IDENT 1 I
15:14 RBRACKET 1
15:16 EQ 1
15:18 KW_NOT 3
15:22 KW_TRUE 4
15:27 KW_AND 3
15:31 KW_FALSE 5
15:37 KW_OR 2
15:40 KW_NULL 4
15:44 EOL
16:1 KW_END 3
16:5 KW_PROGRAM 7
16:12 EOL
16:0 EOF

+ 286
- 0
newLang/tests/lexer/lexer-test.s Ver fichero

@@ -0,0 +1,286 @@
; E02-S02: lexer token-dump test in x128.
;
; For each fixture: src_load, then lex_next until TOK_EOF, appending every
; token to `dump`. The host decodes the dump to text and compares it with the
; golden files in tests/lexer/golden/.
;
; Dump format:
; fixture: $FF, fixture index, src_load result
; token: kind, line lo, line hi, col, then by kind:
; EOL, EOF nothing
; ERROR error code
; IDENT, STRING raw length, text length, text bytes
; NUMBER raw length, value lo, value hi
; CHAR raw length, value
; other raw length
; end: $FE
; Mailbox: PASS if every fixture loaded and the dump did not overflow.
; MB_DATA+0/1 dump address, +2/3 dump length, +4 fixture count.

.setcpu "6502"
.include "../../src/common/c128.inc"
.include "../../src/common/mailbox.inc"
.include "../../src/common/tokens.inc"

.import mmu_init, mmu_bank0, mmu_shutdown
.import src_load
.import lex_reset, lex_next
.import tok_kind, tok_line, tok_col, tok_len, tok_value, tok_text, tok_text_len, tok_err

DUMP_SIZE = $4000
MAX_TOKENS_PER_FIXTURE = 3000

.segment "LOADADDR"
.word $1C01

.segment "EXEHDR"
.word basic_end
.word 10
.byte $9E
.byte "7181"
.byte $00
basic_end:
.word $0000

.segment "MAILBOX"
.res MB_SIZE

.segment "ZEROPAGE"
dump_ptr: .res 2
fx_ptr: .res 2

.segment "STARTUP"
.export test_entry, dump
test_entry:
php
sei
ldx #MB_SIZE - 1
lda #$00
@clr: sta MB_BASE,x
dex
bpl @clr
lda #MB_MAGIC0_VALUE
sta MB_MAGIC0
lda #MB_MAGIC1_VALUE
sta MB_MAGIC1
lda #MB_PROD_PLATFORM
sta MB_PRODUCER
lda #MB_ST_RUNNING
sta MB_STATUS

jsr mmu_init
jsr mmu_bank0

lda #<dump
sta dump_ptr
lda #>dump
sta dump_ptr+1
lda #<fixtures
sta fx_ptr
lda #>fixtures
sta fx_ptr+1
lda #$00
sta failures
sta overflow
sta fx_index

@fixture:
lda #$FF
jsr emit
lda fx_index
jsr emit
ldy #0
lda (fx_ptr),y
sta name_lo
iny
lda (fx_ptr),y
tax
iny
lda (fx_ptr),y
tay
lda name_lo
jsr src_load
pha
jsr emit
pla
beq @lex
inc failures
jmp @advance
@lex: jsr lex_fixture
@advance:
clc
lda fx_ptr
adc #3
sta fx_ptr
bcc :+
inc fx_ptr+1
: inc fx_index
lda fx_index
cmp #FIXTURE_COUNT
bne @fixture

lda #$FE
jsr emit

; Report before mmu_shutdown: dump_ptr is in the private zero page,
; which the system mapping no longer shows.
lda #<dump
sta MB_DATA+0
lda #>dump
sta MB_DATA+1
sec
lda dump_ptr
sbc #<dump
sta MB_DATA+2
lda dump_ptr+1
sbc #>dump
sta MB_DATA+3
lda #FIXTURE_COUNT
sta MB_DATA+4
jsr mmu_shutdown
lda failures
ora overflow
sta MB_CODE
ldy #ok_msg - messages
lda MB_CODE
bne @fail
lda #MB_ST_PASS
bne @report
@fail: ldy #fail_msg - messages
lda #MB_ST_FAIL
@report:
sta MB_STATUS
plp
@print: lda messages,y
beq @end
sty print_y
jsr CHROUT
ldy print_y
iny
bne @print
@end: rts

.segment "CODE"

lex_fixture:
jsr lex_reset
lda #<MAX_TOKENS_PER_FIXTURE
sta budget
lda #>MAX_TOKENS_PER_FIXTURE
sta budget+1
@token: jsr lex_next
jsr emit_token
lda tok_kind
cmp #TOK_EOF
beq @done
lda budget ; runaway guard
bne :+
dec budget+1
: dec budget
lda budget
ora budget+1
bne @token
inc failures
@done: rts

emit_token:
lda tok_kind
jsr emit
lda tok_line
jsr emit
lda tok_line+1
jsr emit
lda tok_col
jsr emit
lda tok_kind
cmp #TOK_EOF
beq @none
cmp #TOK_EOL
beq @none
cmp #TOK_ERROR
bne @len
lda tok_err
jmp emit
@none: rts
@len: lda tok_len
jsr emit
lda tok_kind
cmp #TOK_IDENT
beq @text
cmp #TOK_STRING
beq @text
cmp #TOK_NUMBER
beq @number
cmp #TOK_CHAR
bne @none
lda tok_value
jmp emit
@number:
lda tok_value
jsr emit
lda tok_value+1
jmp emit
@text: lda tok_text_len
jsr emit
ldx #$00
@byte: cpx tok_text_len
beq @none
lda tok_text,x
stx emit_x
jsr emit
ldx emit_x
inx
bne @byte

; Append A to the dump; stop writing (and flag) once it is full.
emit: ldy dump_ptr+1
cpy #>(dump + DUMP_SIZE)
bcc @room
lda #$01
sta overflow
rts
@room: ldy #$00
sta (dump_ptr),y
inc dump_ptr
bne :+
inc dump_ptr+1
: rts

.segment "RODATA"
; name address, name length
fixtures:
.macro FIXTURE name
.local text, done
.word text
.byte done - text
.pushseg
.segment "DATA" ; names live apart from the table
text: .byte name
done:
.popseg
.endmacro
FIXTURE "MINIMAL.N8"
FIXTURE "DECLS.N8"
FIXTURE "ARRAYS.N8"
FIXTURE "EXPRS.N8"
FIXTURE "FORWARD.N8"
FIXTURE "NULLS.N8"
FIXTURE "STMTS.N8"
FIXTURE "TOKENS.N8"
FIXTURE "ERRORS.N8"
FIXTURE "BADCHAR.N8"
FIXTURE_COUNT = 10

messages:
ok_msg: .byte "NOVA128 LEXER TEST OK", $0D, $00
fail_msg: .byte "NOVA128 LEXER TEST FAIL", $0D, $00

.segment "BSS"
failures: .res 1
overflow: .res 1
fx_index: .res 1
name_lo: .res 1
budget: .res 2
emit_x: .res 1
print_y: .res 1
dump: .res DUMP_SIZE

+ 219
- 0
newLang/tests/lexer/reference-lexer.ps1 Ver fichero

@@ -0,0 +1,219 @@
# Host reference lexer for Nova128 (E02-S02). Dot-source this file.
#
# An independent implementation of docs/language-spec.md 2.2 plus the token
# rules documented at the top of src/compiler/lexer.s. Golden files must be
# reproduced by BOTH this lexer and the C128 lexer.
#
# Token text format, one per line:
# L:C KIND [raw-length] [detail]
# IDENT detail = normalized name (PETSCII $41-$5A shown as A-Z)
# NUMBER detail = decimal value
# STRING detail = "text" (see Format-PetsciiText)
# CHAR detail = $hh
# ERROR detail = LEXERR name (no raw length)
# EOL/EOF no length

$script:Keywords = @{
"AND" = "KW_AND"; "AS" = "KW_AS"; "BOOLEAN" = "KW_BOOLEAN"; "BYTE" = "KW_BYTE"
"CHAR" = "KW_CHAR"; "CLASS" = "KW_CLASS"; "DELETE" = "KW_DELETE"; "ELSE" = "KW_ELSE"
"END" = "KW_END"; "FALSE" = "KW_FALSE"; "FOR" = "KW_FOR"; "IF" = "KW_IF"
"IMPLEMENTS" = "KW_IMPLEMENTS"; "INTEGER" = "KW_INTEGER"; "INTERFACE" = "KW_INTERFACE"
"METHOD" = "KW_METHOD"; "NEW" = "KW_NEW"; "NOT" = "KW_NOT"; "NULL" = "KW_NULL"
"OR" = "KW_OR"; "PRINT" = "KW_PRINT"; "PRIVATE" = "KW_PRIVATE"; "PROGRAM" = "KW_PROGRAM"
"PUBLIC" = "KW_PUBLIC"; "RETURN" = "KW_RETURN"; "SELF" = "KW_SELF"; "STEP" = "KW_STEP"
"TO" = "KW_TO"; "TRUE" = "KW_TRUE"; "VAR" = "KW_VAR"; "WHILE" = "KW_WHILE"; "WORD" = "KW_WORD"
}

# Token kind numbers from src/common/tokens.inc (used to decode C128 dumps).
$script:KindNames = @{
0x00 = "EOF"; 0x01 = "EOL"; 0x02 = "IDENT"; 0x03 = "NUMBER"; 0x04 = "STRING"
0x05 = "CHAR"; 0x06 = "ERROR"
0x10 = "LPAREN"; 0x11 = "RPAREN"; 0x12 = "LBRACKET"; 0x13 = "RBRACKET"; 0x14 = "COMMA"
0x15 = "DOT"; 0x16 = "COLON"; 0x17 = "EQ"; 0x18 = "PLUS"; 0x19 = "MINUS"; 0x1A = "STAR"
0x1B = "SLASH"; 0x1C = "LT"; 0x1D = "GT"; 0x1E = "NE"; 0x1F = "LE"; 0x20 = "GE"
}
$i = 0x40
foreach ($kw in ($script:Keywords.Keys | Sort-Object)) {
$script:KindNames[$i] = $script:Keywords[$kw]; $i++
}
$script:ErrorNames = @{
1 = "INVALID_CHAR"; 2 = "IDENT_TOO_LONG"; 3 = "NUMBER_RANGE"; 4 = "BAD_NUMBER"
5 = "UNTERMINATED_STRING"; 6 = "BAD_ESCAPE"; 7 = "BAD_CHAR_LITERAL"
}
$script:Punct = @{
0x28 = "LPAREN"; 0x29 = "RPAREN"; 0x5B = "LBRACKET"; 0x5D = "RBRACKET"; 0x2C = "COMMA"
0x2E = "DOT"; 0x3A = "COLON"; 0x3D = "EQ"; 0x2B = "PLUS"; 0x2D = "MINUS"; 0x2A = "STAR"
0x2F = "SLASH"; 0x3C = "LT"; 0x3E = "GT"
}

function Format-PetsciiText([byte[]]$bytes) {
$sb = New-Object System.Text.StringBuilder
foreach ($b in $bytes) {
if ($b -ge 0x41 -and $b -le 0x5A) { [void]$sb.Append([char]($b + 0x20)) } # a-z
elseif ($b -ge 0xC1 -and $b -le 0xDA) { [void]$sb.Append([char]($b - 0x80)) } # A-Z
elseif ($b -ge 0x20 -and $b -le 0x3F -and $b -ne 0x22) { [void]$sb.Append([char]$b) }
else { [void]$sb.Append(("{{{0:X2}}}" -f $b)) }
}
return '"' + $sb.ToString() + '"'
}

function Format-Name([byte[]]$bytes) {
return -join ($bytes | ForEach-Object { [char]$_ }) # $41-$5A, digits, $5F
}

function Test-Letter([int]$b) { return ($b -ge 0x41 -and $b -le 0x5A) -or ($b -ge 0xC1 -and $b -le 0xDA) }
function Test-Digit([int]$b) { return $b -ge 0x30 -and $b -le 0x39 }
function Test-IdentPart([int]$b) { return (Test-Letter $b) -or (Test-Digit $b) -or $b -eq 0x5F }

# Returns the token lines for one PETSCII source.
function Get-ReferenceTokens([byte[]]$Source) {
$out = New-Object System.Collections.Generic.List[string]
$lines = New-Object System.Collections.Generic.List[byte[]]
$start = 0
for ($k = 0; $k -lt $Source.Length; $k++) {
if ($Source[$k] -eq 13) {
$lines.Add([byte[]]$(if ($k -gt $start) { $Source[$start..($k - 1)] } else { @() }))
$start = $k + 1
}
}
if ($start -lt $Source.Length) { $lines.Add([byte[]]$Source[$start..($Source.Length - 1)]) }

for ($ln = 0; $ln -lt $lines.Count; $ln++) {
$line = $lines[$ln]; $n = $line.Length; $lineNo = $ln + 1
$at = { param($p) if ($p -lt $n) { [int]$line[$p] } else { 13 } }
$p = 0
while ($true) {
while ((& $at $p) -eq 0x20 -or (& $at $p) -eq 0x09) { $p++ }
$c = & $at $p
if ($c -eq 13) { break }
$col = $p + 1; $s = $p
$emit = { param($kind, $detail) $out.Add(("{0}:{1} {2} {3}{4}" -f $lineNo, $col, $kind, ($p - $s), $detail)) }
$err = $null; $errCol = $col

if (Test-Letter $c) {
$name = New-Object System.Collections.Generic.List[byte]
while (Test-IdentPart (& $at $p)) {
$b = & $at $p
if (Test-Letter $b) { $b = $b -band 0x7F }
$name.Add([byte]$b); $p++
}
if ($name.Count -gt 16) { $err = "IDENT_TOO_LONG" }
else {
$text = Format-Name $name.ToArray()
if ($script:Keywords.ContainsKey($text)) { & $emit $script:Keywords[$text] "" }
else { & $emit "IDENT" (" " + $text) }
}
} elseif ((Test-Digit $c) -or $c -eq 0x24 -or $c -eq 0x25) {
$value = [long]0; $count = 0
if ($c -eq 0x24) {
$p++
while ($true) {
$b = & $at $p; $d = -1
if (Test-Digit $b) { $d = $b - 0x30 }
elseif ((Test-Letter $b) -and ($b -band 0x7F) -ge 0x41 -and ($b -band 0x7F) -le 0x46) {
$d = ($b -band 0x7F) - 0x41 + 10
}
if ($d -lt 0) { break }
$value = $value * 16 + $d; $count++; $p++
}
} elseif ($c -eq 0x25) {
$p++
while ((& $at $p) -eq 0x30 -or (& $at $p) -eq 0x31) {
$value = $value * 2 + ((& $at $p) - 0x30); $count++; $p++
}
} else {
while (Test-Digit (& $at $p)) { $value = $value * 10 + ((& $at $p) - 0x30); $count++; $p++ }
}
if ($count -eq 0 -or (Test-IdentPart (& $at $p))) { $err = "BAD_NUMBER" }
elseif ($value -gt 65535) { $err = "NUMBER_RANGE" }
else { & $emit "NUMBER" (" " + $value) }
} elseif ($c -eq 0x22) {
$p++; $text = New-Object System.Collections.Generic.List[byte]
while ($null -eq $err) {
$b = & $at $p
if ($b -eq 13) { $err = "UNTERMINATED_STRING"; break }
if ($b -eq 0x22) { $p++; break }
if ($b -eq 0x5C) {
$e = & $at ($p + 1)
if ($e -eq 0x5C -or $e -eq 0x22) { $b = $e }
elseif (($e -band 0x7F) -eq 0x4E -or ($e -band 0x7F) -eq 0x52) { $b = 0x0D }
else { $err = "BAD_ESCAPE"; $errCol = $p + 1; break }
$p++
}
$text.Add([byte]$b); $p++
}
if ($null -eq $err) { & $emit "STRING" (" " + (Format-PetsciiText $text.ToArray())) }
} elseif ($c -eq 0x27) {
$p++; $b = & $at $p
if ($b -eq 13 -or $b -eq 0x27) { $err = "BAD_CHAR_LITERAL" }
else {
if ($b -eq 0x5C) {
$e = & $at ($p + 1)
if ($e -eq 0x5C -or $e -eq 0x27) { $b = $e }
elseif (($e -band 0x7F) -eq 0x4E -or ($e -band 0x7F) -eq 0x52) { $b = 0x0D }
else { $err = "BAD_ESCAPE"; $errCol = $p + 1 }
$p++
}
if ($null -eq $err) {
$p++
if ((& $at $p) -ne 0x27) { $err = "BAD_CHAR_LITERAL" }
else { $p++; & $emit "CHAR" (" `${0:X2}" -f $b) }
}
}
} elseif ($script:Punct.ContainsKey($c)) {
$next = & $at ($p + 1)
if ($c -eq 0x2F -and $next -eq 0x2F) { break } # comment
$p++
if ($c -eq 0x3C -and $next -eq 0x3E) { $p++; & $emit "NE" "" }
elseif ($c -eq 0x3C -and $next -eq 0x3D) { $p++; & $emit "LE" "" }
elseif ($c -eq 0x3E -and $next -eq 0x3D) { $p++; & $emit "GE" "" }
else { & $emit $script:Punct[$c] "" }
} else {
$err = "INVALID_CHAR"
}

if ($null -ne $err) {
$out.Add(("{0}:{1} ERROR {2}" -f $lineNo, $errCol, $err))
break # skip the line
}
}
$out.Add(("{0}:{1} EOL" -f $lineNo, ($n + 1)))
}
$out.Add(("{0}:0 EOF" -f $lines.Count))
return ,$out
}

# Decode the C128 dump (tests/lexer/lexer-test.s) into per-fixture token lines.
# Returns a list (in dump order) of @{ Index; Load; Tokens }.
function ConvertFrom-TokenDump([byte[]]$Dump) {
$result = New-Object System.Collections.Generic.List[object]
$p = 0; $current = $null
while ($p -lt $Dump.Length) {
$kind = $Dump[$p]
if ($kind -eq 0xFE) { break }
if ($kind -eq 0xFF) {
$current = [pscustomobject]@{ Index = [int]$Dump[$p + 1]; Load = $Dump[$p + 2]
Tokens = New-Object System.Collections.Generic.List[string] }
$result.Add($current)
$p += 3; continue
}
$line = $Dump[$p + 1] + 256 * $Dump[$p + 2]; $col = $Dump[$p + 3]; $p += 4
$name = $script:KindNames[[int]$kind]
if ($null -eq $name) { throw ("unknown token kind `${0:X2} in dump" -f $kind) }
switch ($name) {
"EOF" { $current.Tokens.Add("${line}:$col EOF") }
"EOL" { $current.Tokens.Add("${line}:$col EOL") }
"ERROR" { $current.Tokens.Add("${line}:$col ERROR " + $script:ErrorNames[[int]$Dump[$p]]); $p++ }
{ $_ -in "IDENT", "STRING" } {
$len = $Dump[$p]; $tl = $Dump[$p + 1]
$text = if ($tl -gt 0) { [byte[]]$Dump[($p + 2)..($p + 1 + $tl)] } else { [byte[]]@() }
$detail = if ($name -eq "IDENT") { Format-Name $text } else { Format-PetsciiText $text }
$current.Tokens.Add("${line}:$col $name $len $detail"); $p += 2 + $tl
}
"NUMBER" { $current.Tokens.Add("${line}:$col NUMBER $($Dump[$p]) $($Dump[$p + 1] + 256 * $Dump[$p + 2])"); $p += 3 }
"CHAR" { $current.Tokens.Add(("{0}:{1} CHAR {2} `${3:X2}" -f $line, $col, $Dump[$p], $Dump[$p + 1])); $p += 2 }
default { $current.Tokens.Add("${line}:$col $name $($Dump[$p])"); $p++ }
}
}
return ,$result
}

+ 16
- 0
newLang/tests/lexer/tokens.n8 Ver fichero

@@ -0,0 +1,16 @@
// Lexical edge cases for E02-S02. Every line here must lex without error.
program Tokens
var a_b1 as word
var ABCDEFGHIJKLMNOP as byte
a_b1 = $ffFF + %1010 + 65535 + 0 + $0 + %0
END Program WHILE wHiLe
x = ( ) [ ] , . : = + - * / < > <> <= >=
y=a<>b<=c>=d<e>f
s = "q\"\\\n\r" + "" + "x ' y"
c = 'x' + '\'' + '\\' + '\n' + '\R' + '"'

tabbed = 1 // tab separated
z = 7 / 2 // division, then a comment
// comment-only line
self.data[i] = not true and false or null
end program

Escribir Vista previa
Cargando…
Cancelar
Guardar

Powered by TurnKey Linux.