Просмотр исходного кода
newLang: add the PETSCII lexer (E02-S02)
lexer.s turns cached source lines into tokens with stable kinds and error codes (tokens.inc): 32 keywords, identifiers folded across both PETSCII letter ranges, decimal/hex/binary numbers to 65535, strings and character literals with escapes, all operators, and comments. An error reports its exact column, then the rest of the line is skipped. An independent host reference lexer and the C128 lexer in x128 both reproduce the reviewed golden token files: 10 fixtures, 1,119 tokens, including one fixture per error kind. A case-folding mutation is caught. Spec 2.2 now lists <= and >=, matching section 8. D119 (proposed): the \n and \r escapes both produce $0D. The monitor helper now reads RAM bank 0, because BASIC ROM hides $4000+ after a program returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>main
22 измененных файлов: 2604 добавлений и 5 удалений
-
+3 -1newLang/docs/decisions.md
-
+5 -2newLang/docs/language-spec.md
-
+85 -0newLang/planning/claims/E02-S02.md
-
+63 -0newLang/scripts/build-lexer-test.ps1
-
+6 -2newLang/scripts/lib/Nova128Test.psm1
-
+55 -0newLang/scripts/verify-lexer-test.ps1
-
+97 -0newLang/src/common/tokens.inc
-
+629 -0newLang/src/compiler/lexer.s
-
+21 -0newLang/tests/lexer/errors.n8
-
+132 -0newLang/tests/lexer/golden/arrays.tokens
-
+16 -0newLang/tests/lexer/golden/badchar.tokens
-
+207 -0newLang/tests/lexer/golden/decls.tokens
-
+82 -0newLang/tests/lexer/golden/errors.tokens
-
+180 -0newLang/tests/lexer/golden/exprs.tokens
-
+104 -0newLang/tests/lexer/golden/forward.tokens
-
+8 -0newLang/tests/lexer/golden/minimal.tokens
-
+63 -0newLang/tests/lexer/golden/nulls.tokens
-
+208 -0newLang/tests/lexer/golden/stmts.tokens
-
+119 -0newLang/tests/lexer/golden/tokens.tokens
-
+286 -0newLang/tests/lexer/lexer-test.s
-
+219 -0newLang/tests/lexer/reference-lexer.ps1
-
+16 -0newLang/tests/lexer/tokens.n8
+ 3
- 1
newLang/docs/decisions.md
Просмотреть файл
| @@ -43,7 +43,9 @@ documented default. `Proposed` means a later change still needs approval. | |||||
| ## Proposed | ## Proposed | ||||
| No proposed product changes are currently pending. | |||||
| | ID | Proposal | Consequence | | |||||
| |---|---|---| | |||||
| | D119 | The string and character escapes `\n` and `\r` both produce PETSCII `$0D`. | Commodore output has one line-end character, so both escapes start a new line. Implemented as `ESC_N_VALUE`/`ESC_R_VALUE` in `src/common/tokens.inc`; changing them later affects only those constants and the lexer golden files. | | |||||
| ## Deferred | ## Deferred | ||||
+ 5
- 2
newLang/docs/language-spec.md
Просмотреть файл
| @@ -60,8 +60,11 @@ this specification defines its syntax, semantics, diagnostics, and tests. | |||||
| - Character literals use single quotes and contain exactly one PETSCII | - Character literals use single quotes and contain exactly one PETSCII | ||||
| character or one supported escape: `\'`, `\\`, `\n`, or `\r`. | character or one supported escape: `\'`, `\\`, `\n`, or `\r`. | ||||
| - Reserved punctuation is `(`, `)`, `[`, `]`, `,`, `.`, `:`, `=`, `+`, `-`, | - Reserved punctuation is `(`, `)`, `[`, `]`, `,`, `.`, `:`, `=`, `+`, `-`, | ||||
| `*`, `/`, `<`, `>`, `<>`, and the single quote that delimits character | |||||
| literals. | |||||
| `*`, `/`, `<`, `>`, `<>`, `<=`, `>=`, and the single quote that delimits | |||||
| character literals. (`<=` and `>=` were missing from this list although | |||||
| section 8 defines them; corrected in E02-S02.) | |||||
| - `\n` and `\r` both produce PETSCII `$0D`, the Commodore line end (D119, | |||||
| proposed). | |||||
| - Boolean operators are the words `and`, `or`, and `not`; `&&` and `||` are | - Boolean operators are the words `and`, `or`, and `not`; `&&` and `||` are | ||||
| not aliases. | not aliases. | ||||
+ 85
- 0
newLang/planning/claims/E02-S02.md
Просмотреть файл
| @@ -0,0 +1,85 @@ | |||||
| --- | |||||
| story: E02-S02 | |||||
| owner: e02_lexer | |||||
| status: done | |||||
| started: 2026-11-07 | |||||
| depends-on: [E02-S01, E00-S02] | |||||
| --- | |||||
| # E02-S02 Claim | |||||
| ## Intended change | |||||
| Add the PETSCII lexer (`src/compiler/lexer.s`) with stable token kinds and | |||||
| error codes (`src/common/tokens.inc`), and prove it against golden token files | |||||
| that an independent host reference lexer also reproduces. | |||||
| ## File area | |||||
| - `src/compiler/lexer.s`, `src/common/tokens.inc` | |||||
| - `tests/lexer/` (fixtures, golden files, reference lexer, x128 test) | |||||
| - `scripts/build-lexer-test.ps1`, `scripts/verify-lexer-test.ps1`, | |||||
| `scripts/lib/Nova128Test.psm1` (monitor bank fix) | |||||
| - `docs/language-spec.md` 2.2, `docs/decisions.md` (D119) | |||||
| - `planning/claims/E02-S02.md` | |||||
| - generated files under `build/` | |||||
| ## Acceptance evidence | |||||
| - [x] Both supported PETSCII letter ranges compare case-insensitively: | |||||
| `END Program WHILE wHiLe` yields the same keywords; `$ffFF` (both ranges) | |||||
| is 65535; mixed-case identifiers normalize to `$41-$5A`. A mutation that | |||||
| disables folding fails every fixture with a capitalized identifier. | |||||
| - [x] Keywords, identifiers, numbers, strings, operators, comments, EOL, and | |||||
| EOF match the specification: `tests/lexer/tokens.n8` covers all 32 | |||||
| keywords used by the grammar, every operator including `<>`/`<=`/`>=`, | |||||
| decimal/hex/binary limits, all string and character escapes, tabs, blank | |||||
| and comment-only lines. | |||||
| - [x] Invalid bytes and overlong identifiers report exact positions: | |||||
| `tests/lexer/errors.n8` has one error per line for all seven `LEXERR_*` | |||||
| codes (invalid `&`, `;`, leading `_`, 17-character identifier, ranges, | |||||
| malformed numbers, unterminated string, bad escapes, bad character | |||||
| literals), each at its documented column, with recovery on the next line. | |||||
| The grammar fixture `bad_char.n8` gives `BAD_CHAR_LITERAL` at 3:11, | |||||
| matching its pre-existing `.expect` sidecar. | |||||
| - [x] Token dump of a Vim-format fixture matches a golden expected file: the | |||||
| C128 lexer's dump for all 10 fixtures (1,119 tokens) matches | |||||
| `tests/lexer/golden/*.tokens`, which the host reference lexer also | |||||
| reproduces. Fixtures use Vim's exact PETSCII save format; a live Vim edit | |||||
| is E07-S03. | |||||
| ## Coordination notes | |||||
| - This claim file was created after implementation had started, contrary to | |||||
| `planning/agent-workflow.md`. No other agent was working in this area. | |||||
| - Spec 2.2 omitted `<=`/`>=` from the punctuation list although section 8 and | |||||
| the grammar fixtures use them; 2.2 was corrected to match. | |||||
| - The spec defines `\n`/`\r` escapes but not their bytes. Both produce `$0D` | |||||
| for now; recorded as D119 (Proposed) for user approval. | |||||
| - `init` and `destroy` are identifiers, not keywords (spec 6.3 uses them as | |||||
| method names). | |||||
| ## Work log | |||||
| - Golden files were generated by the reference lexer and then reviewed line | |||||
| by line against the fixtures (error columns, escape bytes, EOL columns) | |||||
| before the C128 run. | |||||
| - Fixed in the test harness: `Read-Nova128Memory` now reads RAM bank 0 | |||||
| (`bank ram`); the CPU view after return shows BASIC ROM at `$4000`, which | |||||
| had corrupted the dump read-back. The lexer test computed its mailbox | |||||
| fields from a private-zero-page pointer after `mmu_shutdown`; reordered. | |||||
| - Found that the Bash tool collapses `\\` and can turn `\n` into a newline, | |||||
| corrupting fixtures and docs; rewrote affected files with the Write tool | |||||
| and byte-checked them. | |||||
| ## Completion | |||||
| - Tests run: `scripts/verify-lexer-test.ps1` (pass plus case-folding | |||||
| mutation), reference-lexer golden check in `build-lexer-test.ps1`. | |||||
| - Results: 10/10 fixtures, 1,119 tokens match; mutation fails 9 fixtures as | |||||
| expected; source restored. | |||||
| - Files changed: see file area. | |||||
| - Remaining risks: D119 awaits approval; the lexer reports at most one error | |||||
| per line by design; `$A0` (shifted space) and other PETSCII control bytes | |||||
| are invalid outside strings, which a Vim user could type by accident; | |||||
| lexer error codes still need user-facing `Exxx` IDs (E02-S04). | |||||
+ 63
- 0
newLang/scripts/build-lexer-test.ps1
Просмотреть файл
| @@ -0,0 +1,63 @@ | |||||
| # E02-S02: build the lexer test PRG and a fresh D81 holding the fixtures. | |||||
| # The host reference lexer must reproduce tests/lexer/golden/*.tokens exactly; | |||||
| # -UpdateGolden rewrites them instead (review the diff before committing). | |||||
| [CmdletBinding()] | |||||
| param([switch]$UpdateGolden) | |||||
| $ErrorActionPreference = "Stop" | |||||
| $project = Split-Path $PSScriptRoot -Parent | |||||
| Import-Module (Join-Path $PSScriptRoot "lib\Nova128Test.psm1") -Force | |||||
| . (Join-Path $project "tests\lexer\reference-lexer.ps1") | |||||
| $out = Join-Path $project "build\lexer-test" | |||||
| $golden = Join-Path $project "tests\lexer\golden" | |||||
| New-Item -ItemType Directory -Force -Path $out, $golden | Out-Null | |||||
| # Order must match the FIXTURE list in tests/lexer/lexer-test.s. | |||||
| $grammar = Join-Path $project "tests\grammar" | |||||
| $fixtures = [ordered]@{ | |||||
| "minimal.n8" = Join-Path $grammar "accepted\minimal.n8" | |||||
| "decls.n8" = Join-Path $grammar "accepted\decls.n8" | |||||
| "arrays.n8" = Join-Path $grammar "accepted\arrays.n8" | |||||
| "exprs.n8" = Join-Path $grammar "accepted\exprs.n8" | |||||
| "forward.n8" = Join-Path $grammar "accepted\forward.n8" | |||||
| "nulls.n8" = Join-Path $grammar "accepted\nulls.n8" | |||||
| "stmts.n8" = Join-Path $grammar "accepted\stmts.n8" | |||||
| "tokens.n8" = Join-Path $project "tests\lexer\tokens.n8" | |||||
| "errors.n8" = Join-Path $project "tests\lexer\errors.n8" | |||||
| "badchar.n8" = Join-Path $grammar "rejected\bad_char.n8" | |||||
| } | |||||
| $link = Invoke-Nova128Link -Name "lexer-test" -BuildDir $out ` | |||||
| -Config (Join-Path $project "cfg\compiler.cfg") -Sources @( | |||||
| (Join-Path $project "src\compiler\mmu.s"), | |||||
| (Join-Path $project "src\compiler\diskio.s"), | |||||
| (Join-Path $project "src\compiler\source.s"), | |||||
| (Join-Path $project "src\compiler\lexer.s"), | |||||
| (Join-Path $project "tests\lexer\lexer-test.s")) | |||||
| if ((Get-Nova128Label $link.Labels "test_entry") -ne 0x1C0D) { throw "test_entry is not at `$1C0D" } | |||||
| $files = @(@{ Name = "lexer-test"; Path = $link.Program; Type = "prg" }) | |||||
| $mismatch = @() | |||||
| foreach ($name in $fixtures.Keys) { | |||||
| $bytes = ConvertTo-Nova128Petscii (Get-Content -Raw -LiteralPath $fixtures[$name]) | |||||
| $pet = Join-Path $out ($name -replace '\.n8$', '.pet') | |||||
| [IO.File]::WriteAllBytes($pet, $bytes) | |||||
| $files += @{ Name = $name; Path = $pet; Type = "seq" } | |||||
| $tokens = (Get-ReferenceTokens $bytes) -join "`n" | |||||
| $goldenPath = Join-Path $golden ($name -replace '\.n8$', '.tokens') | |||||
| if ($UpdateGolden) { | |||||
| [IO.File]::WriteAllText($goldenPath, $tokens + "`n") | |||||
| } elseif (-not (Test-Path -LiteralPath $goldenPath)) { | |||||
| $mismatch += "$name (no golden file; run with -UpdateGolden and review)" | |||||
| } elseif (([IO.File]::ReadAllText($goldenPath).TrimEnd("`n")) -ne $tokens) { | |||||
| $mismatch += "$name (reference lexer differs from golden)" | |||||
| } | |||||
| } | |||||
| if ($mismatch) { throw ("golden check failed:`n " + ($mismatch -join "`n ")) } | |||||
| $disk = Join-Path $out "lexer-test.d81" | |||||
| New-Nova128Disk -Path $disk -Files $files | |||||
| Write-Host ("built {0}" -f $link.Program) | |||||
| Write-Host ("disk {0} ({1} fixtures); reference lexer matches golden files" -f $disk, $fixtures.Count) | |||||
+ 6
- 2
newLang/scripts/lib/Nova128Test.psm1
Просмотреть файл
| @@ -171,9 +171,13 @@ function Invoke-Nova128Monitor { | |||||
| } | } | ||||
| # Read memory through the monitor. Returns byte[] (length = Count). | # Read memory through the monitor. Returns byte[] (length = Count). | ||||
| # Reads RAM bank 0 by default, not the CPU's current view: after a program | |||||
| # returns to BASIC, $4000-$BFFF shows BASIC ROM in the CPU view. | |||||
| function Read-Nova128Memory { | function Read-Nova128Memory { | ||||
| param([Parameter(Mandatory)]$Session, [int]$Address, [int]$Count) | |||||
| $text = Invoke-Nova128Monitor $Session ("m `${0:X4} `${1:X4}" -f $Address, ($Address + $Count - 1)) | |||||
| param([Parameter(Mandatory)]$Session, [int]$Address, [int]$Count, [string]$Bank = "ram") | |||||
| Invoke-Nova128Monitor $Session "bank $Bank" | Out-Null | |||||
| $timeout = 5000 + 2 * $Count # large dumps print slowly | |||||
| $text = Invoke-Nova128Monitor $Session ("m `${0:X4} `${1:X4}" -f $Address, ($Address + $Count - 1)) -TimeoutMs $timeout | |||||
| $memory = @{} | $memory = @{} | ||||
| foreach ($row in [regex]::Matches($text, "(?i)C:([0-9a-f]{4})((?:[ ]{1,2}[0-9a-f]{2}){1,16})")) { | foreach ($row in [regex]::Matches($text, "(?i)C:([0-9a-f]{4})((?:[ ]{1,2}[0-9a-f]{2}){1,16})")) { | ||||
| $a = [Convert]::ToInt32($row.Groups[1].Value, 16) | $a = [Convert]::ToInt32($row.Groups[1].Value, 16) | ||||
+ 55
- 0
newLang/scripts/verify-lexer-test.ps1
Просмотреть файл
| @@ -0,0 +1,55 @@ | |||||
| # E02-S02: run the C128 lexer over every fixture in x128 and compare its token | |||||
| # dump with tests/lexer/golden/*.tokens (which the host reference lexer must | |||||
| # also reproduce; build-lexer-test.ps1 checks that first). | |||||
| [CmdletBinding()] | |||||
| param([int]$Port = 6532, [int]$TimeoutSeconds = 180) | |||||
| $ErrorActionPreference = "Stop" | |||||
| $project = Split-Path $PSScriptRoot -Parent | |||||
| Import-Module (Join-Path $PSScriptRoot "lib\Nova128Test.psm1") -Force | |||||
| . (Join-Path $project "tests\lexer\reference-lexer.ps1") | |||||
| & (Join-Path $PSScriptRoot "build-lexer-test.ps1") | |||||
| $out = Join-Path $project "build\lexer-test" | |||||
| $disk = Join-Path $out "lexer-test.d81" | |||||
| $screenshot = Join-Path $out "lexer-test-vicii.png" | |||||
| $names = @("minimal", "decls", "arrays", "exprs", "forward", "nulls", "stmts", | |||||
| "tokens", "errors", "badchar") | |||||
| $session = Start-Nova128X128 -Autostart $disk -Drive8Image $disk -Port $Port -Screenshot $screenshot | |||||
| try { | |||||
| $mb = Wait-Nova128Mailbox $session -TimeoutSeconds $TimeoutSeconds | |||||
| $address = $mb[8] + 256 * $mb[9] | |||||
| $length = $mb[10] + 256 * $mb[11] | |||||
| $dump = Read-Nova128Memory $session $address $length | |||||
| } finally { | |||||
| Stop-Nova128X128 $session | |||||
| } | |||||
| [IO.File]::WriteAllBytes((Join-Path $out "lexer-dump.bin"), $dump) | |||||
| $fixtures = ConvertFrom-TokenDump $dump | |||||
| $failures = @() | |||||
| if ($mb[3] -ne 0xA5) { $failures += "C128 reported FAIL (code `$" + $mb[4].ToString("X2") + ")" } | |||||
| if ($fixtures.Count -ne $names.Count) { $failures += "dump holds $($fixtures.Count) fixtures, expected $($names.Count)" } | |||||
| for ($i = 0; $i -lt $names.Count; $i++) { | |||||
| $name = $names[$i] | |||||
| $golden = (Get-Content -LiteralPath (Join-Path $project "tests\lexer\golden\$name.tokens")) | | |||||
| Where-Object { $_ -ne "" } | |||||
| $entry = $fixtures[$i] | |||||
| if ($null -eq $entry) { $failures += "${name}: missing from dump"; continue } | |||||
| if ($entry.Load -ne 0) { $failures += "${name}: src_load returned $($entry.Load)"; continue } | |||||
| $got = @($entry.Tokens) | |||||
| $diff = Compare-Object -ReferenceObject $golden -DifferenceObject $got -SyncWindow 0 | |||||
| if ($diff) { | |||||
| $first = 0 | |||||
| while ($first -lt [Math]::Min($golden.Count, $got.Count) -and $golden[$first] -eq $got[$first]) { $first++ } | |||||
| $failures += ("{0}: token {1} differs: golden '{2}' vs C128 '{3}' ({4} vs {5} tokens)" -f | |||||
| $name, ($first + 1), $golden[$first], $got[$first], $golden.Count, $got.Count) | |||||
| Write-Host (" {0,-8} {1,4} tokens FAIL" -f $name, $got.Count) | |||||
| } else { | |||||
| Write-Host (" {0,-8} {1,4} tokens match" -f $name, $got.Count) | |||||
| } | |||||
| } | |||||
| if ($failures) { throw ("lexer verification failed:`n " + ($failures -join "`n ")) } | |||||
| Write-Host ("verified C128 lexer: {0} fixtures, {1} dump bytes, all tokens match golden files" -f $names.Count, $length) | |||||
| Write-Host "screenshot: $screenshot" | |||||
+ 97
- 0
newLang/src/common/tokens.inc
Просмотреть файл
| @@ -0,0 +1,97 @@ | |||||
| ; tokens.inc - Nova128 token kinds and lexer error codes (E02-S02). | |||||
| ; | |||||
| ; Values are stable: parsers, dumps, and golden files depend on them. Add new | |||||
| ; kinds at the end of their group; never renumber. | |||||
| .ifndef TOKENS_INC | |||||
| TOKENS_INC = 1 | |||||
| ; ---------------------------------------------------------- basic kinds | |||||
| TOK_EOF = $00 | |||||
| TOK_EOL = $01 | |||||
| TOK_IDENT = $02 ; tok_text = normalized name ($41-$5A, digits, $5F) | |||||
| TOK_NUMBER = $03 ; tok_value = 0..65535 | |||||
| TOK_STRING = $04 ; tok_text = decoded bytes | |||||
| TOK_CHAR = $05 ; tok_value = one PETSCII byte | |||||
| TOK_ERROR = $06 ; tok_err = LEXERR_*; the rest of the line is skipped | |||||
| ; ---------------------------------------------------------- punctuation | |||||
| TOK_LPAREN = $10 | |||||
| TOK_RPAREN = $11 | |||||
| TOK_LBRACKET = $12 | |||||
| TOK_RBRACKET = $13 | |||||
| TOK_COMMA = $14 | |||||
| TOK_DOT = $15 | |||||
| TOK_COLON = $16 | |||||
| TOK_EQ = $17 ; = (assignment and equality) | |||||
| TOK_PLUS = $18 | |||||
| TOK_MINUS = $19 | |||||
| TOK_STAR = $1A | |||||
| TOK_SLASH = $1B | |||||
| TOK_LT = $1C | |||||
| TOK_GT = $1D | |||||
| TOK_NE = $1E ; <> | |||||
| TOK_LE = $1F ; <= | |||||
| TOK_GE = $20 ; >= | |||||
| ; ------------------------------------------------------------- keywords | |||||
| ; Ordered as in the lexer's keyword table (alphabetical). | |||||
| TOK_KW_FIRST = $40 | |||||
| KW_AND = $40 | |||||
| KW_AS = $41 | |||||
| KW_BOOLEAN = $42 | |||||
| KW_BYTE = $43 | |||||
| KW_CHAR = $44 | |||||
| KW_CLASS = $45 | |||||
| KW_DELETE = $46 | |||||
| KW_ELSE = $47 | |||||
| KW_END = $48 | |||||
| KW_FALSE = $49 | |||||
| KW_FOR = $4A | |||||
| KW_IF = $4B | |||||
| KW_IMPLEMENTS = $4C | |||||
| KW_INTEGER = $4D | |||||
| KW_INTERFACE = $4E | |||||
| KW_METHOD = $4F | |||||
| KW_NEW = $50 | |||||
| KW_NOT = $51 | |||||
| KW_NULL = $52 | |||||
| KW_OR = $53 | |||||
| KW_PRINT = $54 | |||||
| KW_PRIVATE = $55 | |||||
| KW_PROGRAM = $56 | |||||
| KW_PUBLIC = $57 | |||||
| KW_RETURN = $58 | |||||
| KW_SELF = $59 | |||||
| KW_STEP = $5A | |||||
| KW_TO = $5B | |||||
| KW_TRUE = $5C | |||||
| KW_VAR = $5D | |||||
| KW_WHILE = $5E | |||||
| KW_WORD = $5F | |||||
| TOK_KW_LAST = $5F | |||||
| ; ------------------------------------------------------ lexer error codes | |||||
| LEXERR_INVALID_CHAR = 1 ; byte cannot start any token | |||||
| LEXERR_IDENT_TOO_LONG = 2 ; more than 16 identifier characters | |||||
| LEXERR_NUMBER_RANGE = 3 ; value above 65535 | |||||
| LEXERR_BAD_NUMBER = 4 ; $/% without digits, or digits run into letters | |||||
| LEXERR_UNTERMINATED_STRING = 5 ; end of line inside "..." | |||||
| LEXERR_BAD_ESCAPE = 6 ; \ followed by an unsupported character | |||||
| LEXERR_BAD_CHAR_LITERAL = 7 ; '' empty, several characters, or unterminated | |||||
| ; ------------------------------------------------------------- limits | |||||
| LEX_IDENT_MAX = 16 ; significant identifier characters | |||||
| LEX_TEXT_MAX = 80 ; tok_text capacity (one source line) | |||||
| ; ------------------------------------------------------- PETSCII bytes | |||||
| PET_SPACE = $20 | |||||
| PET_TAB = $09 | |||||
| PET_QUOTE = $22 | |||||
| PET_APOS = $27 | |||||
| PET_BACKSLASH = $5C ; ASCII '\' (shown as a pound sign on the C128) | |||||
| PET_UNDERSCORE = $5F ; ASCII '_' (shown as a left arrow) | |||||
| ESC_N_VALUE = $0D ; "\n" (D119, proposed) | |||||
| ESC_R_VALUE = $0D ; "\r" (D119, proposed) | |||||
| .endif | |||||
+ 629
- 0
newLang/src/compiler/lexer.s
Просмотреть файл
| @@ -0,0 +1,629 @@ | |||||
| ; lexer.s - Nova128 PETSCII lexer (E02-S02). | |||||
| ; | |||||
| ; Pulls lines from source.s and returns one token per lex_next call: | |||||
| ; tok_kind TOK_* / KW_* (src/common/tokens.inc) | |||||
| ; tok_line 1-based line (2 bytes) tok_col 1-based start column | |||||
| ; tok_len raw source length of the token (0 for EOL/EOF/ERROR) | |||||
| ; tok_value NUMBER value, or CHAR byte | |||||
| ; tok_text / tok_text_len IDENT: normalized name; STRING: decoded bytes | |||||
| ; tok_err LEXERR_* when tok_kind = TOK_ERROR | |||||
| ; | |||||
| ; Rules (docs/language-spec.md 2.2): | |||||
| ; - Letters $41-$5A and $C1-$DA are folded to $41-$5A in identifiers and | |||||
| ; keywords. Original spelling is src_line[tok_col-1 .. +tok_len]. | |||||
| ; - Every line, including the last one and comment-only or blank lines, | |||||
| ; ends with one TOK_EOL whose column is line length + 1. | |||||
| ; - After a TOK_ERROR the rest of that line is skipped; TOK_EOL follows. | |||||
| ; - Error columns: the token's first character, except BAD_ESCAPE, which | |||||
| ; points at the backslash. | |||||
| .setcpu "6502" | |||||
| .include "../common/tokens.inc" | |||||
| .import src_rewind, src_next_line, src_line, src_line_len, src_line_no | |||||
| .export lex_reset, lex_next | |||||
| .export tok_kind, tok_line, tok_col, tok_len, tok_value | |||||
| .export tok_text, tok_text_len, tok_err | |||||
| CC_BAD = 0 | |||||
| CC_SPACE = 1 | |||||
| CC_LETTER = 2 | |||||
| CC_DIGIT = 3 | |||||
| CC_UNDER = 4 | |||||
| CC_PUNCT = 5 | |||||
| CC_QUOTE = 6 | |||||
| CC_APOS = 7 | |||||
| CC_DOLLAR = 8 | |||||
| CC_PERCENT = 9 | |||||
| ; ------------------------------------------------------------------ CODE | |||||
| .segment "CODE" | |||||
| ; Start lexing from the first source line. | |||||
| lex_reset: | |||||
| jsr src_rewind | |||||
| lda #$00 | |||||
| sta lx_have_line | |||||
| sta lx_skip | |||||
| rts | |||||
| lex_next: | |||||
| lda #$00 | |||||
| sta tok_len | |||||
| sta tok_text_len | |||||
| sta tok_value | |||||
| sta tok_value+1 | |||||
| sta tok_err | |||||
| lda lx_have_line | |||||
| bne @have | |||||
| jsr src_next_line | |||||
| bcc @fresh | |||||
| lda #TOK_EOF ; line stays at the last line delivered | |||||
| sta tok_kind | |||||
| lda src_line_no | |||||
| sta tok_line | |||||
| lda src_line_no+1 | |||||
| sta tok_line+1 | |||||
| lda #$00 | |||||
| sta tok_col | |||||
| rts | |||||
| @fresh: lda #$01 | |||||
| sta lx_have_line | |||||
| lda #$00 | |||||
| sta lx_pos | |||||
| sta lx_skip | |||||
| @have: lda src_line_no | |||||
| sta tok_line | |||||
| lda src_line_no+1 | |||||
| sta tok_line+1 | |||||
| lda lx_skip | |||||
| bne @eol | |||||
| ldy lx_pos | |||||
| @ws: jsr getc | |||||
| tax | |||||
| lda cclass,x | |||||
| cmp #CC_SPACE | |||||
| bne @start | |||||
| iny | |||||
| bne @ws | |||||
| @start: cpx #$0D ; getc sentinel: end of line | |||||
| beq @eol | |||||
| sty lx_start | |||||
| iny | |||||
| sty tok_col ; 1-based start column | |||||
| stx lx_ch | |||||
| asl a ; dispatch on character class | |||||
| tax | |||||
| lda scan_table+1,x | |||||
| pha | |||||
| lda scan_table,x | |||||
| pha | |||||
| lda lx_ch | |||||
| rts ; jumps to scan_* with Y = start + 1 | |||||
| @eol: lda #TOK_EOL | |||||
| sta tok_kind | |||||
| ldx src_line_len | |||||
| inx | |||||
| stx tok_col | |||||
| lda #$00 | |||||
| sta lx_have_line | |||||
| rts | |||||
| scan_table: | |||||
| .word scan_bad - 1, scan_bad - 1, scan_ident - 1, scan_dec - 1 | |||||
| .word scan_bad - 1, scan_punct - 1, scan_string - 1, scan_char - 1 | |||||
| .word scan_hex - 1, scan_bin - 1 | |||||
| ; A = character at Y, or $0D past the end of the line. Preserves X/Y. | |||||
| getc: cpy src_line_len | |||||
| bcs @end | |||||
| lda src_line,y | |||||
| rts | |||||
| @end: lda #$0D | |||||
| rts | |||||
| ; Finish a good token: tok_kind = A, lx_pos = Y, tok_len = Y - start. | |||||
| finish: sta tok_kind | |||||
| sty lx_pos | |||||
| tya | |||||
| sec | |||||
| sbc lx_start | |||||
| sta tok_len | |||||
| rts | |||||
| ; Report an error at tok_col: tok_err = A, skip the rest of the line. | |||||
| fail: sta tok_err | |||||
| lda #TOK_ERROR | |||||
| sta tok_kind | |||||
| lda #$00 | |||||
| sta tok_len | |||||
| lda #$01 | |||||
| sta lx_skip | |||||
| rts | |||||
| scan_bad: | |||||
| lda #LEXERR_INVALID_CHAR | |||||
| jmp fail | |||||
| ; ------------------------------------------------------------ identifiers | |||||
| scan_ident: | |||||
| dey ; rescan from the first letter | |||||
| lda #$00 | |||||
| sta tok_text_len | |||||
| @loop: jsr getc | |||||
| sta lx_ch | |||||
| tax | |||||
| lda cclass,x | |||||
| cmp #CC_LETTER | |||||
| beq @letter | |||||
| cmp #CC_DIGIT | |||||
| beq @keep | |||||
| cmp #CC_UNDER | |||||
| bne @done | |||||
| beq @keep | |||||
| @letter: | |||||
| lda lx_ch | |||||
| and #$7F ; $C1-$DA -> $41-$5A | |||||
| sta lx_ch | |||||
| @keep: ldx tok_text_len | |||||
| cpx #LEX_IDENT_MAX | |||||
| bcs @over | |||||
| lda lx_ch | |||||
| sta tok_text,x | |||||
| @over: inc tok_text_len | |||||
| iny | |||||
| bne @loop | |||||
| @done: lda tok_text_len | |||||
| cmp #LEX_IDENT_MAX + 1 | |||||
| bcc @fits | |||||
| lda #LEXERR_IDENT_TOO_LONG | |||||
| jmp fail | |||||
| @fits: jsr keyword_lookup ; A = KW_* or TOK_IDENT | |||||
| jmp finish | |||||
| ; Compare tok_text with the keyword table. A = kind. Preserves Y. | |||||
| keyword_lookup: | |||||
| sty lx_save_y | |||||
| ldx #$00 ; X = offset of the current entry | |||||
| @entry: lda keywords,x | |||||
| beq @ident ; end of table | |||||
| cmp tok_text_len | |||||
| bne @skip | |||||
| ldy #$00 | |||||
| @cmp: lda keywords+1,x | |||||
| cmp tok_text,y | |||||
| bne @skip_from_y | |||||
| inx | |||||
| iny | |||||
| cpy tok_text_len | |||||
| bne @cmp | |||||
| lda keywords+1,x ; kind follows the letters | |||||
| ldy lx_save_y | |||||
| rts | |||||
| @skip_from_y: ; rewind X to the entry start | |||||
| stx lx_tmp | |||||
| tya | |||||
| eor #$FF | |||||
| sec | |||||
| adc lx_tmp | |||||
| tax | |||||
| @skip: lda keywords,x ; next entry: 1 + length + 1 bytes | |||||
| clc | |||||
| adc #$02 | |||||
| stx lx_tmp | |||||
| adc lx_tmp | |||||
| tax | |||||
| bne @entry | |||||
| @ident: lda #TOK_IDENT | |||||
| ldy lx_save_y | |||||
| rts | |||||
| ; ------------------------------------------------------------- numbers | |||||
| ; Decimal: digits only; a letter or _ directly after them is BAD_NUMBER. | |||||
| scan_dec: | |||||
| dey | |||||
| lda #$00 | |||||
| sta tok_value | |||||
| sta tok_value+1 | |||||
| sta lx_ovf | |||||
| @loop: jsr getc | |||||
| tax | |||||
| lda cclass,x | |||||
| cmp #CC_DIGIT | |||||
| bne @end | |||||
| txa | |||||
| sec | |||||
| sbc #'0' | |||||
| sta lx_digit | |||||
| jsr times10 | |||||
| lda lx_digit | |||||
| clc | |||||
| adc tok_value | |||||
| sta tok_value | |||||
| bcc @next | |||||
| inc tok_value+1 | |||||
| bne @next | |||||
| inc lx_ovf | |||||
| @next: iny | |||||
| bne @loop | |||||
| @end: jsr check_tail | |||||
| bcs @bad | |||||
| lda lx_ovf | |||||
| beq @ok | |||||
| lda #LEXERR_NUMBER_RANGE | |||||
| jmp fail | |||||
| @ok: lda #TOK_NUMBER | |||||
| jmp finish | |||||
| @bad: lda #LEXERR_BAD_NUMBER | |||||
| jmp fail | |||||
| ; tok_value *= 10, setting lx_ovf on overflow past 16 bits. | |||||
| times10: | |||||
| lda tok_value ; v*2 | |||||
| asl a | |||||
| sta lx_tmp | |||||
| lda tok_value+1 | |||||
| rol a | |||||
| sta lx_tmp+1 | |||||
| bcc :+ | |||||
| inc lx_ovf | |||||
| : lda lx_tmp ; v*8 = (v*2)*4 | |||||
| sta tok_value | |||||
| lda lx_tmp+1 | |||||
| sta tok_value+1 | |||||
| ldx #2 | |||||
| @shift: asl tok_value | |||||
| rol tok_value+1 | |||||
| bcc :+ | |||||
| inc lx_ovf | |||||
| : dex | |||||
| bne @shift | |||||
| lda tok_value ; v*8 + v*2 | |||||
| clc | |||||
| adc lx_tmp | |||||
| sta tok_value | |||||
| lda tok_value+1 | |||||
| adc lx_tmp+1 | |||||
| sta tok_value+1 | |||||
| bcc :+ | |||||
| inc lx_ovf | |||||
| : rts | |||||
| ; C=1 when the character at Y continues an identifier (letter, digit, _). | |||||
| check_tail: | |||||
| jsr getc | |||||
| tax | |||||
| lda cclass,x | |||||
| cmp #CC_LETTER | |||||
| beq @yes | |||||
| cmp #CC_DIGIT | |||||
| beq @yes | |||||
| cmp #CC_UNDER | |||||
| beq @yes | |||||
| clc | |||||
| rts | |||||
| @yes: sec | |||||
| rts | |||||
| ; Hexadecimal: $ then hex digits (A-F in either letter range). | |||||
| scan_hex: | |||||
| lda #$00 | |||||
| sta tok_value | |||||
| sta tok_value+1 | |||||
| sta lx_ovf | |||||
| sta lx_count | |||||
| @loop: jsr getc | |||||
| jsr hex_digit | |||||
| bcc @end | |||||
| sta lx_digit | |||||
| lda tok_value+1 ; the top nibble must be clear to shift | |||||
| and #$F0 | |||||
| beq :+ | |||||
| inc lx_ovf | |||||
| : ldx #4 | |||||
| @shift: asl tok_value | |||||
| rol tok_value+1 | |||||
| dex | |||||
| bne @shift | |||||
| lda tok_value | |||||
| ora lx_digit | |||||
| sta tok_value | |||||
| inc lx_count | |||||
| iny | |||||
| bne @loop | |||||
| @end: lda lx_count | |||||
| beq scan_bad_number | |||||
| jsr check_tail | |||||
| bcs scan_bad_number | |||||
| lda lx_ovf | |||||
| bne scan_range | |||||
| lda #TOK_NUMBER | |||||
| jmp finish | |||||
| scan_bad_number: | |||||
| lda #LEXERR_BAD_NUMBER | |||||
| jmp fail | |||||
| scan_range: | |||||
| lda #LEXERR_NUMBER_RANGE | |||||
| jmp fail | |||||
| ; C=1 and A = 0..15 when A is a hex digit. | |||||
| hex_digit: | |||||
| cmp #'0' | |||||
| bcc @no | |||||
| cmp #'9' + 1 | |||||
| bcs @letter | |||||
| sbc #'0' - 1 ; C=0 here: subtract '0' | |||||
| sec | |||||
| rts | |||||
| @letter: | |||||
| and #$7F ; fold $C1-$C6 to $41-$46 | |||||
| cmp #'A' | |||||
| bcc @no | |||||
| cmp #'F' + 1 | |||||
| bcs @no | |||||
| sbc #'A' - 11 ; C=0: A - 'A' + 10 | |||||
| sec | |||||
| rts | |||||
| @no: clc | |||||
| rts | |||||
| ; Binary: % then 0/1 digits. | |||||
| scan_bin: | |||||
| lda #$00 | |||||
| sta tok_value | |||||
| sta tok_value+1 | |||||
| sta lx_ovf | |||||
| sta lx_count | |||||
| @loop: jsr getc | |||||
| cmp #'0' | |||||
| beq @bit | |||||
| cmp #'1' | |||||
| bne @end | |||||
| @bit: lsr a ; C = the bit ('0'=$30, '1'=$31) | |||||
| rol tok_value | |||||
| rol tok_value+1 | |||||
| bcc :+ | |||||
| inc lx_ovf | |||||
| : inc lx_count | |||||
| iny | |||||
| bne @loop | |||||
| @end: lda lx_count | |||||
| beq scan_bad_number | |||||
| jsr check_tail | |||||
| bcs scan_bad_number | |||||
| lda lx_ovf | |||||
| bne scan_range | |||||
| lda #TOK_NUMBER | |||||
| jmp finish | |||||
| ; --------------------------------------------------- strings and chars | |||||
| scan_string: | |||||
| lda #PET_QUOTE ; strings allow \" but not \' | |||||
| sta lx_esc_quote | |||||
| lda #$00 | |||||
| sta tok_text_len | |||||
| @loop: jsr getc | |||||
| cmp #$0D | |||||
| beq @open | |||||
| cmp #PET_QUOTE | |||||
| beq @close | |||||
| cmp #PET_BACKSLASH | |||||
| bne @store | |||||
| jsr escape ; A = decoded byte or C=1 bad escape | |||||
| bcs @bad_escape | |||||
| @store: ldx tok_text_len | |||||
| sta tok_text,x | |||||
| inc tok_text_len | |||||
| iny | |||||
| bne @loop | |||||
| @close: iny | |||||
| lda #TOK_STRING | |||||
| jmp finish | |||||
| @open: lda #LEXERR_UNTERMINATED_STRING | |||||
| jmp fail | |||||
| @bad_escape: | |||||
| lda #LEXERR_BAD_ESCAPE | |||||
| jmp fail | |||||
| ; Y at a backslash. Allowed after it: backslash, n, r, and lx_esc_quote | |||||
| ; (" in strings, ' in character literals). On success: C=0, A = byte, Y at | |||||
| ; the escape letter. On failure: C=1 and tok_col moved to the backslash. | |||||
| escape: sty lx_tmp | |||||
| iny | |||||
| jsr getc | |||||
| cmp #PET_BACKSLASH | |||||
| beq @ok | |||||
| cmp lx_esc_quote | |||||
| beq @ok | |||||
| and #$7F ; n/N and r/R in either letter range | |||||
| cmp #'N' | |||||
| beq @n | |||||
| cmp #'R' | |||||
| beq @r | |||||
| ldx lx_tmp ; error column = the backslash | |||||
| inx | |||||
| stx tok_col | |||||
| sec | |||||
| rts | |||||
| @n: lda #ESC_N_VALUE | |||||
| clc | |||||
| rts | |||||
| @r: lda #ESC_R_VALUE | |||||
| @ok: clc | |||||
| rts | |||||
| scan_char: | |||||
| lda #PET_APOS ; character literals allow \' but not \" | |||||
| sta lx_esc_quote | |||||
| jsr getc | |||||
| cmp #$0D | |||||
| beq @bad | |||||
| cmp #PET_APOS ; '' is empty | |||||
| beq @bad | |||||
| cmp #PET_BACKSLASH | |||||
| bne @plain | |||||
| jsr escape | |||||
| bcs @bad_escape | |||||
| @plain: sta tok_value | |||||
| iny | |||||
| jsr getc | |||||
| cmp #PET_APOS | |||||
| bne @bad | |||||
| iny | |||||
| lda #TOK_CHAR | |||||
| jmp finish | |||||
| @bad: lda #LEXERR_BAD_CHAR_LITERAL | |||||
| jmp fail | |||||
| @bad_escape: | |||||
| lda #LEXERR_BAD_ESCAPE | |||||
| jmp fail | |||||
| ; ---------------------------------------------------------- punctuation | |||||
| scan_punct: | |||||
| ldx #punct_count - 1 | |||||
| @find: cmp punct_chars,x | |||||
| beq @found | |||||
| dex | |||||
| bpl @find | |||||
| jmp scan_bad ; cannot happen: class table and list agree | |||||
| @found: lda punct_kinds,x | |||||
| cmp #TOK_SLASH | |||||
| beq @slash | |||||
| cmp #TOK_LT | |||||
| beq @lt | |||||
| cmp #TOK_GT | |||||
| beq @gt | |||||
| @one: jmp finish | |||||
| @slash: pha | |||||
| jsr getc | |||||
| cmp #'/' | |||||
| bne @single | |||||
| pla ; comment: skip to end of line | |||||
| lda #$01 | |||||
| sta lx_skip | |||||
| jmp lex_next | |||||
| @lt: pha | |||||
| jsr getc | |||||
| cmp #'>' | |||||
| beq @ne | |||||
| cmp #'=' | |||||
| beq @le | |||||
| @single: | |||||
| pla | |||||
| jmp finish | |||||
| @ne: pla | |||||
| iny | |||||
| lda #TOK_NE | |||||
| jmp finish | |||||
| @le: pla | |||||
| iny | |||||
| lda #TOK_LE | |||||
| jmp finish | |||||
| @gt: pha | |||||
| jsr getc | |||||
| cmp #'=' | |||||
| bne @single | |||||
| pla | |||||
| iny | |||||
| lda #TOK_GE | |||||
| jmp finish | |||||
| .segment "RODATA" | |||||
| punct_chars: | |||||
| .byte "()[],.:=+-*/<>" | |||||
| punct_count = * - punct_chars | |||||
| punct_kinds: | |||||
| .byte TOK_LPAREN, TOK_RPAREN, TOK_LBRACKET, TOK_RBRACKET, TOK_COMMA | |||||
| .byte TOK_DOT, TOK_COLON, TOK_EQ, TOK_PLUS, TOK_MINUS, TOK_STAR | |||||
| .byte TOK_SLASH, TOK_LT, TOK_GT | |||||
| .assert * - punct_kinds = punct_count, error, "punctuation tables differ" | |||||
| ; Keyword table: length, normalized letters, kind; 0 ends the table. | |||||
| .macro KW name, kind | |||||
| .byte .strlen(name), name, kind | |||||
| .endmacro | |||||
| keywords: | |||||
| KW "AND", KW_AND | |||||
| KW "AS", KW_AS | |||||
| KW "BOOLEAN", KW_BOOLEAN | |||||
| KW "BYTE", KW_BYTE | |||||
| KW "CHAR", KW_CHAR | |||||
| KW "CLASS", KW_CLASS | |||||
| KW "DELETE", KW_DELETE | |||||
| KW "ELSE", KW_ELSE | |||||
| KW "END", KW_END | |||||
| KW "FALSE", KW_FALSE | |||||
| KW "FOR", KW_FOR | |||||
| KW "IF", KW_IF | |||||
| KW "IMPLEMENTS", KW_IMPLEMENTS | |||||
| KW "INTEGER", KW_INTEGER | |||||
| KW "INTERFACE", KW_INTERFACE | |||||
| KW "METHOD", KW_METHOD | |||||
| KW "NEW", KW_NEW | |||||
| KW "NOT", KW_NOT | |||||
| KW "NULL", KW_NULL | |||||
| KW "OR", KW_OR | |||||
| KW "PRINT", KW_PRINT | |||||
| KW "PRIVATE", KW_PRIVATE | |||||
| KW "PROGRAM", KW_PROGRAM | |||||
| KW "PUBLIC", KW_PUBLIC | |||||
| KW "RETURN", KW_RETURN | |||||
| KW "SELF", KW_SELF | |||||
| KW "STEP", KW_STEP | |||||
| KW "TO", KW_TO | |||||
| KW "TRUE", KW_TRUE | |||||
| KW "VAR", KW_VAR | |||||
| KW "WHILE", KW_WHILE | |||||
| KW "WORD", KW_WORD | |||||
| .byte 0 | |||||
| .assert * - keywords < 256, error, "keyword table must stay below 256 bytes" | |||||
| ; Character classes for every PETSCII byte. | |||||
| cclass: | |||||
| .repeat 256, I | |||||
| .if I = PET_SPACE .or I = PET_TAB | |||||
| .byte CC_SPACE | |||||
| .elseif (I >= $41 .and I <= $5A) .or (I >= $C1 .and I <= $DA) | |||||
| .byte CC_LETTER | |||||
| .elseif I >= '0' .and I <= '9' | |||||
| .byte CC_DIGIT | |||||
| .elseif I = PET_UNDERSCORE | |||||
| .byte CC_UNDER | |||||
| .elseif I = PET_QUOTE | |||||
| .byte CC_QUOTE | |||||
| .elseif I = PET_APOS | |||||
| .byte CC_APOS | |||||
| .elseif I = '$' | |||||
| .byte CC_DOLLAR | |||||
| .elseif I = '%' | |||||
| .byte CC_PERCENT | |||||
| .elseif I = '(' .or I = ')' .or I = '[' .or I = ']' .or I = ',' .or I = '.' .or I = ':' .or I = '=' .or I = '+' .or I = '-' .or I = '*' .or I = '/' .or I = '<' .or I = '>' | |||||
| .byte CC_PUNCT | |||||
| .else | |||||
| .byte CC_BAD | |||||
| .endif | |||||
| .endrepeat | |||||
| ; ------------------------------------------------------------------- BSS | |||||
| .segment "BSS" | |||||
| tok_kind: .res 1 | |||||
| tok_line: .res 2 | |||||
| tok_col: .res 1 | |||||
| tok_len: .res 1 | |||||
| tok_value: .res 2 | |||||
| tok_err: .res 1 | |||||
| tok_text_len: .res 1 | |||||
| tok_text: .res LEX_TEXT_MAX | |||||
| lx_have_line: .res 1 | |||||
| lx_skip: .res 1 | |||||
| lx_pos: .res 1 | |||||
| lx_start: .res 1 | |||||
| lx_ch: .res 1 | |||||
| lx_ovf: .res 1 | |||||
| lx_digit: .res 1 | |||||
| lx_count: .res 1 | |||||
| lx_save_y: .res 1 | |||||
| lx_esc_quote: .res 1 | |||||
| lx_tmp: .res 2 | |||||
+ 21
- 0
newLang/tests/lexer/errors.n8
Просмотреть файл
| @@ -0,0 +1,21 @@ | |||||
| // Lexer errors: one per line; each line after the error is skipped. | |||||
| a = 1 & 2 | |||||
| abcdefghijklmnopq = 1 | |||||
| n = 65536 | |||||
| n = $10000 | |||||
| n = %11111111111111111 | |||||
| n = $ | |||||
| n = $g1 | |||||
| n = 12ab | |||||
| n = %102 | |||||
| s = "open | |||||
| s = "bad \q escape" | |||||
| s = "no \' here" | |||||
| c = '' | |||||
| c = 'ab' | |||||
| c = 'a | |||||
| c = '\"' | |||||
| c = '\q' | |||||
| _x = 1 | |||||
| x = 1 ; y = 2 | |||||
| ok = $1F + %11 // lexing resumes on the next line | |||||
+ 132
- 0
newLang/tests/lexer/golden/arrays.tokens
Просмотреть файл
| @@ -0,0 +1,132 @@ | |||||
| 1:1 KW_CLASS 5 | |||||
| 1:7 IDENT 6 BUFFER | |||||
| 1:13 EOL | |||||
| 2:3 KW_PRIVATE 7 | |||||
| 2:11 KW_BYTE 4 | |||||
| 2:15 LBRACKET 1 | |||||
| 2:16 NUMBER 1 4 | |||||
| 2:17 RBRACKET 1 | |||||
| 2:19 IDENT 4 DATA | |||||
| 2:23 EOL | |||||
| 3:1 EOL | |||||
| 4:3 KW_PUBLIC 6 | |||||
| 4:10 KW_METHOD 6 | |||||
| 4:17 IDENT 3 SET | |||||
| 4:20 LPAREN 1 | |||||
| 4:21 IDENT 5 INDEX | |||||
| 4:27 KW_AS 2 | |||||
| 4:30 KW_WORD 4 | |||||
| 4:34 COMMA 1 | |||||
| 4:36 IDENT 5 VALUE | |||||
| 4:42 KW_AS 2 | |||||
| 4:45 KW_BYTE 4 | |||||
| 4:49 RPAREN 1 | |||||
| 4:50 EOL | |||||
| 5:5 KW_SELF 4 | |||||
| 5:9 DOT 1 | |||||
| 5:10 IDENT 4 DATA | |||||
| 5:14 LBRACKET 1 | |||||
| 5:15 IDENT 5 INDEX | |||||
| 5:20 RBRACKET 1 | |||||
| 5:22 EQ 1 | |||||
| 5:24 IDENT 5 VALUE | |||||
| 5:29 EOL | |||||
| 6:3 KW_END 3 | |||||
| 6:7 KW_METHOD 6 | |||||
| 6:13 EOL | |||||
| 7:1 EOL | |||||
| 8:3 KW_PUBLIC 6 | |||||
| 8:10 KW_METHOD 6 | |||||
| 8:17 IDENT 3 GET | |||||
| 8:20 LPAREN 1 | |||||
| 8:21 IDENT 5 INDEX | |||||
| 8:27 KW_AS 2 | |||||
| 8:30 KW_WORD 4 | |||||
| 8:34 RPAREN 1 | |||||
| 8:36 KW_AS 2 | |||||
| 8:39 KW_BYTE 4 | |||||
| 8:43 EOL | |||||
| 9:5 KW_RETURN 6 | |||||
| 9:12 KW_SELF 4 | |||||
| 9:16 DOT 1 | |||||
| 9:17 IDENT 4 DATA | |||||
| 9:21 LBRACKET 1 | |||||
| 9:22 IDENT 5 INDEX | |||||
| 9:27 RBRACKET 1 | |||||
| 9:28 EOL | |||||
| 10:3 KW_END 3 | |||||
| 10:7 KW_METHOD 6 | |||||
| 10:13 EOL | |||||
| 11:1 KW_END 3 | |||||
| 11:5 KW_CLASS 5 | |||||
| 11:10 EOL | |||||
| 12:1 EOL | |||||
| 13:1 KW_PROGRAM 7 | |||||
| 13:9 IDENT 4 MAIN | |||||
| 13:13 EOL | |||||
| 14:3 KW_VAR 3 | |||||
| 14:7 IDENT 6 BUFFER | |||||
| 14:14 KW_AS 2 | |||||
| 14:17 IDENT 6 BUFFER | |||||
| 14:23 EOL | |||||
| 15:3 KW_VAR 3 | |||||
| 15:7 IDENT 6 VALUES | |||||
| 15:14 KW_AS 2 | |||||
| 15:17 KW_WORD 4 | |||||
| 15:21 LBRACKET 1 | |||||
| 15:22 NUMBER 1 4 | |||||
| 15:23 RBRACKET 1 | |||||
| 15:24 EOL | |||||
| 16:3 KW_VAR 3 | |||||
| 16:7 IDENT 4 ITEM | |||||
| 16:12 KW_AS 2 | |||||
| 16:15 KW_BYTE 4 | |||||
| 16:19 EOL | |||||
| 17:3 IDENT 6 BUFFER | |||||
| 17:10 EQ 1 | |||||
| 17:12 KW_NEW 3 | |||||
| 17:16 IDENT 6 BUFFER | |||||
| 17:22 LPAREN 1 | |||||
| 17:23 RPAREN 1 | |||||
| 17:24 EOL | |||||
| 18:3 IDENT 6 VALUES | |||||
| 18:9 LBRACKET 1 | |||||
| 18:10 NUMBER 1 0 | |||||
| 18:11 RBRACKET 1 | |||||
| 18:13 EQ 1 | |||||
| 18:15 NUMBER 1 1 | |||||
| 18:16 EOL | |||||
| 19:3 IDENT 6 BUFFER | |||||
| 19:9 DOT 1 | |||||
| 19:10 IDENT 3 SET | |||||
| 19:13 LPAREN 1 | |||||
| 19:14 IDENT 6 VALUES | |||||
| 19:20 LBRACKET 1 | |||||
| 19:21 NUMBER 1 0 | |||||
| 19:22 RBRACKET 1 | |||||
| 19:23 COMMA 1 | |||||
| 19:25 NUMBER 1 7 | |||||
| 19:26 RPAREN 1 | |||||
| 19:27 EOL | |||||
| 20:3 IDENT 4 ITEM | |||||
| 20:8 EQ 1 | |||||
| 20:10 IDENT 6 BUFFER | |||||
| 20:16 DOT 1 | |||||
| 20:17 IDENT 3 GET | |||||
| 20:20 LPAREN 1 | |||||
| 20:21 IDENT 6 VALUES | |||||
| 20:27 LBRACKET 1 | |||||
| 20:28 NUMBER 1 0 | |||||
| 20:29 RBRACKET 1 | |||||
| 20:30 RPAREN 1 | |||||
| 20:31 EOL | |||||
| 21:3 KW_PRINT 5 | |||||
| 21:9 IDENT 4 ITEM | |||||
| 21:13 EOL | |||||
| 22:3 KW_DELETE 6 | |||||
| 22:10 IDENT 6 BUFFER | |||||
| 22:16 EOL | |||||
| 23:1 KW_END 3 | |||||
| 23:5 KW_PROGRAM 7 | |||||
| 23:12 EOL | |||||
| 23:0 EOF | |||||
+ 16
- 0
newLang/tests/lexer/golden/badchar.tokens
Просмотреть файл
| @@ -0,0 +1,16 @@ | |||||
| 1:1 KW_PROGRAM 7 | |||||
| 1:9 IDENT 4 MAIN | |||||
| 1:13 EOL | |||||
| 2:3 KW_VAR 3 | |||||
| 2:7 IDENT 5 GLYPH | |||||
| 2:13 KW_AS 2 | |||||
| 2:16 KW_CHAR 4 | |||||
| 2:20 EOL | |||||
| 3:3 IDENT 5 GLYPH | |||||
| 3:9 EQ 1 | |||||
| 3:11 ERROR BAD_CHAR_LITERAL | |||||
| 3:15 EOL | |||||
| 4:1 KW_END 3 | |||||
| 4:5 KW_PROGRAM 7 | |||||
| 4:12 EOL | |||||
| 4:0 EOF | |||||
+ 207
- 0
newLang/tests/lexer/golden/decls.tokens
Просмотреть файл
| @@ -0,0 +1,207 @@ | |||||
| 1:1 KW_INTERFACE 9 | |||||
| 1:11 IDENT 9 PRINTABLE | |||||
| 1:20 EOL | |||||
| 2:3 KW_METHOD 6 | |||||
| 2:10 KW_PRINT 5 | |||||
| 2:15 LPAREN 1 | |||||
| 2:16 RPAREN 1 | |||||
| 2:17 EOL | |||||
| 3:1 KW_END 3 | |||||
| 3:5 KW_INTERFACE 9 | |||||
| 3:14 EOL | |||||
| 4:1 EOL | |||||
| 5:1 KW_INTERFACE 9 | |||||
| 5:11 IDENT 10 RESETTABLE | |||||
| 5:21 EOL | |||||
| 6:3 KW_METHOD 6 | |||||
| 6:10 IDENT 5 RESET | |||||
| 6:15 LPAREN 1 | |||||
| 6:16 RPAREN 1 | |||||
| 6:17 EOL | |||||
| 7:3 KW_METHOD 6 | |||||
| 7:10 IDENT 6 ACTIVE | |||||
| 7:16 LPAREN 1 | |||||
| 7:17 RPAREN 1 | |||||
| 7:19 KW_AS 2 | |||||
| 7:22 KW_BOOLEAN 7 | |||||
| 7:29 EOL | |||||
| 8:1 KW_END 3 | |||||
| 8:5 KW_INTERFACE 9 | |||||
| 8:14 EOL | |||||
| 9:1 EOL | |||||
| 10:1 KW_CLASS 5 | |||||
| 10:7 IDENT 7 COUNTER | |||||
| 10:15 KW_IMPLEMENTS 10 | |||||
| 10:26 IDENT 9 PRINTABLE | |||||
| 10:35 COMMA 1 | |||||
| 10:37 IDENT 10 RESETTABLE | |||||
| 10:47 EOL | |||||
| 11:3 KW_PRIVATE 7 | |||||
| 11:11 KW_WORD 4 | |||||
| 11:16 IDENT 5 VALUE | |||||
| 11:21 EOL | |||||
| 12:3 KW_PRIVATE 7 | |||||
| 12:11 KW_BYTE 4 | |||||
| 12:15 LBRACKET 1 | |||||
| 12:16 NUMBER 1 4 | |||||
| 12:17 RBRACKET 1 | |||||
| 12:19 IDENT 7 HISTORY | |||||
| 12:26 EOL | |||||
| 13:1 EOL | |||||
| 14:3 KW_PUBLIC 6 | |||||
| 14:10 KW_METHOD 6 | |||||
| 14:17 IDENT 4 INIT | |||||
| 14:21 LPAREN 1 | |||||
| 14:22 IDENT 5 START | |||||
| 14:28 KW_AS 2 | |||||
| 14:31 KW_WORD 4 | |||||
| 14:35 RPAREN 1 | |||||
| 14:36 EOL | |||||
| 15:5 KW_VAR 3 | |||||
| 15:9 IDENT 4 COPY | |||||
| 15:14 KW_AS 2 | |||||
| 15:17 KW_WORD 4 | |||||
| 15:21 EOL | |||||
| 16:5 IDENT 4 COPY | |||||
| 16:10 EQ 1 | |||||
| 16:12 IDENT 5 START | |||||
| 16:17 EOL | |||||
| 17:5 KW_SELF 4 | |||||
| 17:9 DOT 1 | |||||
| 17:10 IDENT 5 VALUE | |||||
| 17:16 EQ 1 | |||||
| 17:18 IDENT 4 COPY | |||||
| 17:22 EOL | |||||
| 18:5 KW_SELF 4 | |||||
| 18:9 DOT 1 | |||||
| 18:10 IDENT 7 HISTORY | |||||
| 18:17 LBRACKET 1 | |||||
| 18:18 NUMBER 1 0 | |||||
| 18:19 RBRACKET 1 | |||||
| 18:21 EQ 1 | |||||
| 18:23 NUMBER 1 0 | |||||
| 18:24 EOL | |||||
| 19:3 KW_END 3 | |||||
| 19:7 KW_METHOD 6 | |||||
| 19:13 EOL | |||||
| 20:1 EOL | |||||
| 21:3 KW_PUBLIC 6 | |||||
| 21:10 KW_METHOD 6 | |||||
| 21:17 IDENT 7 DESTROY | |||||
| 21:24 LPAREN 1 | |||||
| 21:25 RPAREN 1 | |||||
| 21:26 EOL | |||||
| 22:5 KW_SELF 4 | |||||
| 22:9 DOT 1 | |||||
| 22:10 IDENT 5 VALUE | |||||
| 22:16 EQ 1 | |||||
| 22:18 NUMBER 1 0 | |||||
| 22:19 EOL | |||||
| 23:5 KW_RETURN 6 | |||||
| 23:11 EOL | |||||
| 24:3 KW_END 3 | |||||
| 24:7 KW_METHOD 6 | |||||
| 24:13 EOL | |||||
| 25:1 EOL | |||||
| 26:3 KW_PUBLIC 6 | |||||
| 26:10 KW_METHOD 6 | |||||
| 26:17 KW_PRINT 5 | |||||
| 26:22 LPAREN 1 | |||||
| 26:23 RPAREN 1 | |||||
| 26:24 EOL | |||||
| 27:5 KW_PRINT 5 | |||||
| 27:11 KW_SELF 4 | |||||
| 27:15 DOT 1 | |||||
| 27:16 IDENT 5 VALUE | |||||
| 27:21 EOL | |||||
| 28:3 KW_END 3 | |||||
| 28:7 KW_METHOD 6 | |||||
| 28:13 EOL | |||||
| 29:1 EOL | |||||
| 30:3 KW_PUBLIC 6 | |||||
| 30:10 KW_METHOD 6 | |||||
| 30:17 IDENT 5 RESET | |||||
| 30:22 LPAREN 1 | |||||
| 30:23 RPAREN 1 | |||||
| 30:24 EOL | |||||
| 31:5 KW_SELF 4 | |||||
| 31:9 DOT 1 | |||||
| 31:10 IDENT 5 VALUE | |||||
| 31:16 EQ 1 | |||||
| 31:18 NUMBER 1 0 | |||||
| 31:19 EOL | |||||
| 32:3 KW_END 3 | |||||
| 32:7 KW_METHOD 6 | |||||
| 32:13 EOL | |||||
| 33:1 EOL | |||||
| 34:3 KW_PUBLIC 6 | |||||
| 34:10 KW_METHOD 6 | |||||
| 34:17 IDENT 6 ACTIVE | |||||
| 34:23 LPAREN 1 | |||||
| 34:24 RPAREN 1 | |||||
| 34:26 KW_AS 2 | |||||
| 34:29 KW_BOOLEAN 7 | |||||
| 34:36 EOL | |||||
| 35:5 KW_IF 2 | |||||
| 35:8 KW_SELF 4 | |||||
| 35:12 DOT 1 | |||||
| 35:13 IDENT 5 VALUE | |||||
| 35:19 NE 2 | |||||
| 35:22 NUMBER 1 0 | |||||
| 35:23 EOL | |||||
| 36:7 KW_RETURN 6 | |||||
| 36:14 KW_TRUE 4 | |||||
| 36:18 EOL | |||||
| 37:5 KW_ELSE 4 | |||||
| 37:9 EOL | |||||
| 38:7 KW_RETURN 6 | |||||
| 38:14 KW_FALSE 5 | |||||
| 38:19 EOL | |||||
| 39:5 KW_END 3 | |||||
| 39:9 KW_IF 2 | |||||
| 39:11 EOL | |||||
| 40:3 KW_END 3 | |||||
| 40:7 KW_METHOD 6 | |||||
| 40:13 EOL | |||||
| 41:1 KW_END 3 | |||||
| 41:5 KW_CLASS 5 | |||||
| 41:10 EOL | |||||
| 42:1 EOL | |||||
| 43:1 KW_PROGRAM 7 | |||||
| 43:9 IDENT 4 MAIN | |||||
| 43:13 EOL | |||||
| 44:3 KW_VAR 3 | |||||
| 44:7 IDENT 5 COUNT | |||||
| 44:13 KW_AS 2 | |||||
| 44:16 IDENT 7 COUNTER | |||||
| 44:23 EOL | |||||
| 45:3 KW_VAR 3 | |||||
| 45:7 IDENT 6 OUTPUT | |||||
| 45:14 KW_AS 2 | |||||
| 45:17 IDENT 9 PRINTABLE | |||||
| 45:26 EOL | |||||
| 46:3 IDENT 5 COUNT | |||||
| 46:9 EQ 1 | |||||
| 46:11 KW_NEW 3 | |||||
| 46:15 IDENT 7 COUNTER | |||||
| 46:22 LPAREN 1 | |||||
| 46:23 NUMBER 2 10 | |||||
| 46:25 RPAREN 1 | |||||
| 46:26 EOL | |||||
| 47:3 IDENT 6 OUTPUT | |||||
| 47:10 EQ 1 | |||||
| 47:12 IDENT 5 COUNT | |||||
| 47:17 EOL | |||||
| 48:3 IDENT 6 OUTPUT | |||||
| 48:9 DOT 1 | |||||
| 48:10 KW_PRINT 5 | |||||
| 48:15 LPAREN 1 | |||||
| 48:16 RPAREN 1 | |||||
| 48:17 EOL | |||||
| 49:3 KW_DELETE 6 | |||||
| 49:10 IDENT 6 OUTPUT | |||||
| 49:16 EOL | |||||
| 50:1 KW_END 3 | |||||
| 50:5 KW_PROGRAM 7 | |||||
| 50:12 EOL | |||||
| 50:0 EOF | |||||
+ 82
- 0
newLang/tests/lexer/golden/errors.tokens
Просмотреть файл
| @@ -0,0 +1,82 @@ | |||||
| 1:69 EOL | |||||
| 2:1 IDENT 1 A | |||||
| 2:3 EQ 1 | |||||
| 2:5 NUMBER 1 1 | |||||
| 2:7 ERROR INVALID_CHAR | |||||
| 2:10 EOL | |||||
| 3:1 ERROR IDENT_TOO_LONG | |||||
| 3:22 EOL | |||||
| 4:1 IDENT 1 N | |||||
| 4:3 EQ 1 | |||||
| 4:5 ERROR NUMBER_RANGE | |||||
| 4:10 EOL | |||||
| 5:1 IDENT 1 N | |||||
| 5:3 EQ 1 | |||||
| 5:5 ERROR NUMBER_RANGE | |||||
| 5:11 EOL | |||||
| 6:1 IDENT 1 N | |||||
| 6:3 EQ 1 | |||||
| 6:5 ERROR NUMBER_RANGE | |||||
| 6:23 EOL | |||||
| 7:1 IDENT 1 N | |||||
| 7:3 EQ 1 | |||||
| 7:5 ERROR BAD_NUMBER | |||||
| 7:6 EOL | |||||
| 8:1 IDENT 1 N | |||||
| 8:3 EQ 1 | |||||
| 8:5 ERROR BAD_NUMBER | |||||
| 8:8 EOL | |||||
| 9:1 IDENT 1 N | |||||
| 9:3 EQ 1 | |||||
| 9:5 ERROR BAD_NUMBER | |||||
| 9:9 EOL | |||||
| 10:1 IDENT 1 N | |||||
| 10:3 EQ 1 | |||||
| 10:5 ERROR BAD_NUMBER | |||||
| 10:9 EOL | |||||
| 11:1 IDENT 1 S | |||||
| 11:3 EQ 1 | |||||
| 11:5 ERROR UNTERMINATED_STRING | |||||
| 11:10 EOL | |||||
| 12:1 IDENT 1 S | |||||
| 12:3 EQ 1 | |||||
| 12:10 ERROR BAD_ESCAPE | |||||
| 12:20 EOL | |||||
| 13:1 IDENT 1 S | |||||
| 13:3 EQ 1 | |||||
| 13:9 ERROR BAD_ESCAPE | |||||
| 13:17 EOL | |||||
| 14:1 IDENT 1 C | |||||
| 14:3 EQ 1 | |||||
| 14:5 ERROR BAD_CHAR_LITERAL | |||||
| 14:7 EOL | |||||
| 15:1 IDENT 1 C | |||||
| 15:3 EQ 1 | |||||
| 15:5 ERROR BAD_CHAR_LITERAL | |||||
| 15:9 EOL | |||||
| 16:1 IDENT 1 C | |||||
| 16:3 EQ 1 | |||||
| 16:5 ERROR BAD_CHAR_LITERAL | |||||
| 16:7 EOL | |||||
| 17:1 IDENT 1 C | |||||
| 17:3 EQ 1 | |||||
| 17:6 ERROR BAD_ESCAPE | |||||
| 17:9 EOL | |||||
| 18:1 IDENT 1 C | |||||
| 18:3 EQ 1 | |||||
| 18:6 ERROR BAD_ESCAPE | |||||
| 18:9 EOL | |||||
| 19:1 ERROR INVALID_CHAR | |||||
| 19:7 EOL | |||||
| 20:1 IDENT 1 X | |||||
| 20:3 EQ 1 | |||||
| 20:5 NUMBER 1 1 | |||||
| 20:7 ERROR INVALID_CHAR | |||||
| 20:14 EOL | |||||
| 21:1 IDENT 2 OK | |||||
| 21:4 EQ 1 | |||||
| 21:6 NUMBER 3 31 | |||||
| 21:10 PLUS 1 | |||||
| 21:12 NUMBER 3 3 | |||||
| 21:50 EOL | |||||
| 21:0 EOF | |||||
+ 180
- 0
newLang/tests/lexer/golden/exprs.tokens
Просмотреть файл
| @@ -0,0 +1,180 @@ | |||||
| 1:1 KW_CLASS 5 | |||||
| 1:7 IDENT 7 FORMULA | |||||
| 1:14 EOL | |||||
| 2:3 KW_PUBLIC 6 | |||||
| 2:10 KW_METHOD 6 | |||||
| 2:17 IDENT 4 EVAL | |||||
| 2:21 LPAREN 1 | |||||
| 2:22 IDENT 1 A | |||||
| 2:24 KW_AS 2 | |||||
| 2:27 KW_INTEGER 7 | |||||
| 2:34 COMMA 1 | |||||
| 2:36 IDENT 1 B | |||||
| 2:38 KW_AS 2 | |||||
| 2:41 KW_INTEGER 7 | |||||
| 2:48 RPAREN 1 | |||||
| 2:50 KW_AS 2 | |||||
| 2:53 KW_INTEGER 7 | |||||
| 2:60 EOL | |||||
| 3:5 KW_VAR 3 | |||||
| 3:9 IDENT 6 RESULT | |||||
| 3:16 KW_AS 2 | |||||
| 3:19 KW_INTEGER 7 | |||||
| 3:26 EOL | |||||
| 4:5 IDENT 6 RESULT | |||||
| 4:12 EQ 1 | |||||
| 4:14 MINUS 1 | |||||
| 4:15 LPAREN 1 | |||||
| 4:16 IDENT 1 A | |||||
| 4:18 PLUS 1 | |||||
| 4:20 IDENT 1 B | |||||
| 4:21 RPAREN 1 | |||||
| 4:23 STAR 1 | |||||
| 4:25 NUMBER 1 2 | |||||
| 4:27 SLASH 1 | |||||
| 4:29 NUMBER 1 1 | |||||
| 4:30 EOL | |||||
| 5:5 KW_IF 2 | |||||
| 5:8 KW_NOT 3 | |||||
| 5:12 LPAREN 1 | |||||
| 5:13 IDENT 1 A | |||||
| 5:15 EQ 1 | |||||
| 5:17 IDENT 1 B | |||||
| 5:18 RPAREN 1 | |||||
| 5:20 KW_AND 3 | |||||
| 5:24 LPAREN 1 | |||||
| 5:25 IDENT 1 A | |||||
| 5:27 NE 2 | |||||
| 5:30 NUMBER 1 0 | |||||
| 5:32 KW_OR 2 | |||||
| 5:35 IDENT 1 B | |||||
| 5:37 GE 2 | |||||
| 5:40 IDENT 1 A | |||||
| 5:41 RPAREN 1 | |||||
| 5:42 EOL | |||||
| 6:7 IDENT 6 RESULT | |||||
| 6:14 EQ 1 | |||||
| 6:16 IDENT 6 RESULT | |||||
| 6:23 PLUS 1 | |||||
| 6:25 IDENT 1 A | |||||
| 6:27 MINUS 1 | |||||
| 6:29 IDENT 1 B | |||||
| 6:30 EOL | |||||
| 7:5 KW_ELSE 4 | |||||
| 7:9 EOL | |||||
| 8:7 IDENT 6 RESULT | |||||
| 8:14 EQ 1 | |||||
| 8:16 NUMBER 1 0 | |||||
| 8:17 EOL | |||||
| 9:5 KW_END 3 | |||||
| 9:9 KW_IF 2 | |||||
| 9:11 EOL | |||||
| 10:5 KW_IF 2 | |||||
| 10:8 IDENT 1 A | |||||
| 10:10 LT 1 | |||||
| 10:12 IDENT 1 B | |||||
| 10:13 EOL | |||||
| 11:7 IDENT 6 RESULT | |||||
| 11:14 EQ 1 | |||||
| 11:16 IDENT 6 RESULT | |||||
| 11:23 PLUS 1 | |||||
| 11:25 NUMBER 1 1 | |||||
| 11:26 EOL | |||||
| 12:5 KW_END 3 | |||||
| 12:9 KW_IF 2 | |||||
| 12:11 EOL | |||||
| 13:5 KW_IF 2 | |||||
| 13:8 IDENT 1 A | |||||
| 13:10 LE 2 | |||||
| 13:13 IDENT 1 B | |||||
| 13:14 EOL | |||||
| 14:7 IDENT 6 RESULT | |||||
| 14:14 EQ 1 | |||||
| 14:16 IDENT 6 RESULT | |||||
| 14:23 PLUS 1 | |||||
| 14:25 NUMBER 1 1 | |||||
| 14:26 EOL | |||||
| 15:5 KW_END 3 | |||||
| 15:9 KW_IF 2 | |||||
| 15:11 EOL | |||||
| 16:5 KW_IF 2 | |||||
| 16:8 IDENT 1 A | |||||
| 16:10 GT 1 | |||||
| 16:12 IDENT 1 B | |||||
| 16:13 EOL | |||||
| 17:7 IDENT 6 RESULT | |||||
| 17:14 EQ 1 | |||||
| 17:16 IDENT 6 RESULT | |||||
| 17:23 MINUS 1 | |||||
| 17:25 NUMBER 1 1 | |||||
| 17:26 EOL | |||||
| 18:5 KW_END 3 | |||||
| 18:9 KW_IF 2 | |||||
| 18:11 EOL | |||||
| 19:5 KW_RETURN 6 | |||||
| 19:12 IDENT 6 RESULT | |||||
| 19:18 EOL | |||||
| 20:3 KW_END 3 | |||||
| 20:7 KW_METHOD 6 | |||||
| 20:13 EOL | |||||
| 21:1 KW_END 3 | |||||
| 21:5 KW_CLASS 5 | |||||
| 21:10 EOL | |||||
| 22:1 EOL | |||||
| 23:1 KW_PROGRAM 7 | |||||
| 23:9 IDENT 4 MAIN | |||||
| 23:13 EOL | |||||
| 24:3 KW_VAR 3 | |||||
| 24:7 IDENT 7 FORMULA | |||||
| 24:15 KW_AS 2 | |||||
| 24:18 IDENT 7 FORMULA | |||||
| 24:25 EOL | |||||
| 25:3 KW_VAR 3 | |||||
| 25:7 IDENT 6 ANSWER | |||||
| 25:14 KW_AS 2 | |||||
| 25:17 KW_INTEGER 7 | |||||
| 25:24 EOL | |||||
| 26:3 KW_VAR 3 | |||||
| 26:7 IDENT 5 GLYPH | |||||
| 26:13 KW_AS 2 | |||||
| 26:16 KW_CHAR 4 | |||||
| 26:20 EOL | |||||
| 27:3 IDENT 7 FORMULA | |||||
| 27:11 EQ 1 | |||||
| 27:13 KW_NEW 3 | |||||
| 27:17 IDENT 7 FORMULA | |||||
| 27:24 LPAREN 1 | |||||
| 27:25 RPAREN 1 | |||||
| 27:26 EOL | |||||
| 28:3 IDENT 6 ANSWER | |||||
| 28:10 EQ 1 | |||||
| 28:12 IDENT 7 FORMULA | |||||
| 28:19 DOT 1 | |||||
| 28:20 IDENT 4 EVAL | |||||
| 28:24 LPAREN 1 | |||||
| 28:25 MINUS 1 | |||||
| 28:26 NUMBER 1 2 | |||||
| 28:27 COMMA 1 | |||||
| 28:29 NUMBER 1 3 | |||||
| 28:30 RPAREN 1 | |||||
| 28:31 EOL | |||||
| 29:3 IDENT 5 GLYPH | |||||
| 29:9 EQ 1 | |||||
| 29:11 CHAR 3 $C1 | |||||
| 29:14 EOL | |||||
| 30:3 KW_PRINT 5 | |||||
| 30:9 STRING 10 "answer{0D}" | |||||
| 30:19 EOL | |||||
| 31:3 KW_PRINT 5 | |||||
| 31:9 IDENT 6 ANSWER | |||||
| 31:15 EOL | |||||
| 32:3 KW_PRINT 5 | |||||
| 32:9 IDENT 5 GLYPH | |||||
| 32:14 EOL | |||||
| 33:3 KW_DELETE 6 | |||||
| 33:10 IDENT 7 FORMULA | |||||
| 33:17 EOL | |||||
| 34:1 KW_END 3 | |||||
| 34:5 KW_PROGRAM 7 | |||||
| 34:12 EOL | |||||
| 34:0 EOF | |||||
+ 104
- 0
newLang/tests/lexer/golden/forward.tokens
Просмотреть файл
| @@ -0,0 +1,104 @@ | |||||
| 1:1 KW_CLASS 5 | |||||
| 1:7 IDENT 6 RUNNER | |||||
| 1:14 KW_IMPLEMENTS 10 | |||||
| 1:25 IDENT 3 JOB | |||||
| 1:28 EOL | |||||
| 2:3 KW_PUBLIC 6 | |||||
| 2:10 KW_METHOD 6 | |||||
| 2:17 IDENT 7 EXECUTE | |||||
| 2:24 LPAREN 1 | |||||
| 2:25 IDENT 4 CODE | |||||
| 2:30 KW_AS 2 | |||||
| 2:33 KW_BYTE 4 | |||||
| 2:37 RPAREN 1 | |||||
| 2:39 KW_AS 2 | |||||
| 2:42 KW_BOOLEAN 7 | |||||
| 2:49 EOL | |||||
| 3:5 KW_IF 2 | |||||
| 3:8 IDENT 4 CODE | |||||
| 3:13 EQ 1 | |||||
| 3:15 NUMBER 1 0 | |||||
| 3:16 EOL | |||||
| 4:7 KW_RETURN 6 | |||||
| 4:14 KW_FALSE 5 | |||||
| 4:19 EOL | |||||
| 5:5 KW_ELSE 4 | |||||
| 5:9 EOL | |||||
| 6:7 KW_RETURN 6 | |||||
| 6:14 KW_TRUE 4 | |||||
| 6:18 EOL | |||||
| 7:5 KW_END 3 | |||||
| 7:9 KW_IF 2 | |||||
| 7:11 EOL | |||||
| 8:3 KW_END 3 | |||||
| 8:7 KW_METHOD 6 | |||||
| 8:13 EOL | |||||
| 9:1 KW_END 3 | |||||
| 9:5 KW_CLASS 5 | |||||
| 9:10 EOL | |||||
| 10:1 EOL | |||||
| 11:1 KW_INTERFACE 9 | |||||
| 11:11 IDENT 3 JOB | |||||
| 11:14 EOL | |||||
| 12:3 KW_METHOD 6 | |||||
| 12:10 IDENT 7 EXECUTE | |||||
| 12:17 LPAREN 1 | |||||
| 12:18 IDENT 4 CODE | |||||
| 12:23 KW_AS 2 | |||||
| 12:26 KW_BYTE 4 | |||||
| 12:30 RPAREN 1 | |||||
| 12:32 KW_AS 2 | |||||
| 12:35 KW_BOOLEAN 7 | |||||
| 12:42 EOL | |||||
| 13:1 KW_END 3 | |||||
| 13:5 KW_INTERFACE 9 | |||||
| 13:14 EOL | |||||
| 14:1 EOL | |||||
| 15:1 KW_PROGRAM 7 | |||||
| 15:9 IDENT 4 MAIN | |||||
| 15:13 EOL | |||||
| 16:3 KW_VAR 3 | |||||
| 16:7 IDENT 6 RUNNER | |||||
| 16:14 KW_AS 2 | |||||
| 16:17 IDENT 6 RUNNER | |||||
| 16:23 EOL | |||||
| 17:3 KW_VAR 3 | |||||
| 17:7 IDENT 3 JOB | |||||
| 17:11 KW_AS 2 | |||||
| 17:14 IDENT 3 JOB | |||||
| 17:17 EOL | |||||
| 18:3 KW_VAR 3 | |||||
| 18:7 IDENT 2 OK | |||||
| 18:10 KW_AS 2 | |||||
| 18:13 KW_BOOLEAN 7 | |||||
| 18:20 EOL | |||||
| 19:3 IDENT 6 RUNNER | |||||
| 19:10 EQ 1 | |||||
| 19:12 KW_NEW 3 | |||||
| 19:16 IDENT 6 RUNNER | |||||
| 19:22 LPAREN 1 | |||||
| 19:23 RPAREN 1 | |||||
| 19:24 EOL | |||||
| 20:3 IDENT 3 JOB | |||||
| 20:7 EQ 1 | |||||
| 20:9 IDENT 6 RUNNER | |||||
| 20:15 EOL | |||||
| 21:3 IDENT 2 OK | |||||
| 21:6 EQ 1 | |||||
| 21:8 IDENT 3 JOB | |||||
| 21:11 DOT 1 | |||||
| 21:12 IDENT 7 EXECUTE | |||||
| 21:19 LPAREN 1 | |||||
| 21:20 NUMBER 1 1 | |||||
| 21:21 RPAREN 1 | |||||
| 21:22 EOL | |||||
| 22:3 KW_PRINT 5 | |||||
| 22:9 IDENT 2 OK | |||||
| 22:11 EOL | |||||
| 23:3 KW_DELETE 6 | |||||
| 23:10 IDENT 3 JOB | |||||
| 23:13 EOL | |||||
| 24:1 KW_END 3 | |||||
| 24:5 KW_PROGRAM 7 | |||||
| 24:12 EOL | |||||
| 24:0 EOF | |||||
+ 8
- 0
newLang/tests/lexer/golden/minimal.tokens
Просмотреть файл
| @@ -0,0 +1,8 @@ | |||||
| 1:48 EOL | |||||
| 2:1 KW_PROGRAM 7 | |||||
| 2:9 IDENT 4 MAIN | |||||
| 2:13 EOL | |||||
| 3:1 KW_END 3 | |||||
| 3:5 KW_PROGRAM 7 | |||||
| 3:12 EOL | |||||
| 3:0 EOF | |||||
+ 63
- 0
newLang/tests/lexer/golden/nulls.tokens
Просмотреть файл
| @@ -0,0 +1,63 @@ | |||||
| 1:1 KW_INTERFACE 9 | |||||
| 1:11 IDENT 6 MARKER | |||||
| 1:17 EOL | |||||
| 2:3 KW_METHOD 6 | |||||
| 2:10 IDENT 4 MARK | |||||
| 2:14 LPAREN 1 | |||||
| 2:15 RPAREN 1 | |||||
| 2:16 EOL | |||||
| 3:1 KW_END 3 | |||||
| 3:5 KW_INTERFACE 9 | |||||
| 3:14 EOL | |||||
| 4:1 EOL | |||||
| 5:1 KW_CLASS 5 | |||||
| 5:7 IDENT 5 EMPTY | |||||
| 5:13 KW_IMPLEMENTS 10 | |||||
| 5:24 IDENT 6 MARKER | |||||
| 5:30 EOL | |||||
| 6:3 KW_WORD 4 | |||||
| 6:8 IDENT 5 VALUE | |||||
| 6:13 EOL | |||||
| 7:1 EOL | |||||
| 8:3 KW_METHOD 6 | |||||
| 8:10 IDENT 4 MARK | |||||
| 8:14 LPAREN 1 | |||||
| 8:15 RPAREN 1 | |||||
| 8:16 EOL | |||||
| 9:5 KW_RETURN 6 | |||||
| 9:11 EOL | |||||
| 10:3 KW_END 3 | |||||
| 10:7 KW_METHOD 6 | |||||
| 10:13 EOL | |||||
| 11:1 KW_END 3 | |||||
| 11:5 KW_CLASS 5 | |||||
| 11:10 EOL | |||||
| 12:1 EOL | |||||
| 13:1 KW_PROGRAM 7 | |||||
| 13:9 IDENT 4 MAIN | |||||
| 13:13 EOL | |||||
| 14:3 KW_VAR 3 | |||||
| 14:7 IDENT 6 OBJECT | |||||
| 14:14 KW_AS 2 | |||||
| 14:17 IDENT 5 EMPTY | |||||
| 14:22 EOL | |||||
| 15:3 KW_VAR 3 | |||||
| 15:7 IDENT 6 MARKER | |||||
| 15:14 KW_AS 2 | |||||
| 15:17 IDENT 6 MARKER | |||||
| 15:23 EOL | |||||
| 16:3 IDENT 6 OBJECT | |||||
| 16:10 EQ 1 | |||||
| 16:12 KW_NULL 4 | |||||
| 16:16 EOL | |||||
| 17:3 IDENT 6 MARKER | |||||
| 17:10 EQ 1 | |||||
| 17:12 KW_NULL 4 | |||||
| 17:16 EOL | |||||
| 18:3 KW_DELETE 6 | |||||
| 18:10 IDENT 6 MARKER | |||||
| 18:16 EOL | |||||
| 19:1 KW_END 3 | |||||
| 19:5 KW_PROGRAM 7 | |||||
| 19:12 EOL | |||||
| 19:0 EOF | |||||
+ 208
- 0
newLang/tests/lexer/golden/stmts.tokens
Просмотреть файл
| @@ -0,0 +1,208 @@ | |||||
| 1:1 KW_CLASS 5 | |||||
| 1:7 IDENT 6 WORKER | |||||
| 1:13 EOL | |||||
| 2:3 KW_PRIVATE 7 | |||||
| 2:11 KW_WORD 4 | |||||
| 2:16 IDENT 5 VALUE | |||||
| 2:21 EOL | |||||
| 3:1 EOL | |||||
| 4:3 KW_PUBLIC 6 | |||||
| 4:10 KW_METHOD 6 | |||||
| 4:17 IDENT 4 INIT | |||||
| 4:21 LPAREN 1 | |||||
| 4:22 IDENT 5 START | |||||
| 4:28 KW_AS 2 | |||||
| 4:31 KW_WORD 4 | |||||
| 4:35 RPAREN 1 | |||||
| 4:36 EOL | |||||
| 5:5 KW_SELF 4 | |||||
| 5:9 DOT 1 | |||||
| 5:10 IDENT 5 VALUE | |||||
| 5:16 EQ 1 | |||||
| 5:18 IDENT 5 START | |||||
| 5:23 EOL | |||||
| 6:3 KW_END 3 | |||||
| 6:7 KW_METHOD 6 | |||||
| 6:13 EOL | |||||
| 7:1 EOL | |||||
| 8:3 KW_PUBLIC 6 | |||||
| 8:10 KW_METHOD 6 | |||||
| 8:17 IDENT 4 TICK | |||||
| 8:21 LPAREN 1 | |||||
| 8:22 IDENT 6 AMOUNT | |||||
| 8:29 KW_AS 2 | |||||
| 8:32 KW_WORD 4 | |||||
| 8:36 RPAREN 1 | |||||
| 8:37 EOL | |||||
| 9:5 KW_SELF 4 | |||||
| 9:9 DOT 1 | |||||
| 9:10 IDENT 5 VALUE | |||||
| 9:16 EQ 1 | |||||
| 9:18 KW_SELF 4 | |||||
| 9:22 DOT 1 | |||||
| 9:23 IDENT 5 VALUE | |||||
| 9:29 PLUS 1 | |||||
| 9:31 IDENT 6 AMOUNT | |||||
| 9:37 EOL | |||||
| 10:3 KW_END 3 | |||||
| 10:7 KW_METHOD 6 | |||||
| 10:13 EOL | |||||
| 11:1 EOL | |||||
| 12:3 KW_PUBLIC 6 | |||||
| 12:10 KW_METHOD 6 | |||||
| 12:17 IDENT 3 RUN | |||||
| 12:20 LPAREN 1 | |||||
| 12:21 IDENT 5 LIMIT | |||||
| 12:27 KW_AS 2 | |||||
| 12:30 KW_WORD 4 | |||||
| 12:34 RPAREN 1 | |||||
| 12:36 KW_AS 2 | |||||
| 12:39 KW_WORD 4 | |||||
| 12:43 EOL | |||||
| 13:5 KW_VAR 3 | |||||
| 13:9 IDENT 1 I | |||||
| 13:11 KW_AS 2 | |||||
| 13:14 KW_WORD 4 | |||||
| 13:18 EOL | |||||
| 14:5 KW_VAR 3 | |||||
| 14:9 IDENT 5 TOTAL | |||||
| 14:15 KW_AS 2 | |||||
| 14:18 KW_WORD 4 | |||||
| 14:22 EOL | |||||
| 15:5 IDENT 1 I | |||||
| 15:7 EQ 1 | |||||
| 15:9 NUMBER 1 0 | |||||
| 15:10 EOL | |||||
| 16:5 IDENT 5 TOTAL | |||||
| 16:11 EQ 1 | |||||
| 16:13 NUMBER 1 0 | |||||
| 16:14 EOL | |||||
| 17:5 KW_WHILE 5 | |||||
| 17:11 IDENT 1 I | |||||
| 17:13 LT 1 | |||||
| 17:15 IDENT 5 LIMIT | |||||
| 17:20 EOL | |||||
| 18:7 IDENT 5 TOTAL | |||||
| 18:13 EQ 1 | |||||
| 18:15 IDENT 5 TOTAL | |||||
| 18:21 PLUS 1 | |||||
| 18:23 IDENT 1 I | |||||
| 18:24 EOL | |||||
| 19:7 IDENT 1 I | |||||
| 19:9 EQ 1 | |||||
| 19:11 IDENT 1 I | |||||
| 19:13 PLUS 1 | |||||
| 19:15 NUMBER 1 1 | |||||
| 19:16 EOL | |||||
| 20:5 KW_END 3 | |||||
| 20:9 KW_WHILE 5 | |||||
| 20:14 EOL | |||||
| 21:5 KW_IF 2 | |||||
| 21:8 IDENT 5 TOTAL | |||||
| 21:14 GT 1 | |||||
| 21:16 NUMBER 3 100 | |||||
| 21:19 EOL | |||||
| 22:7 KW_PRINT 5 | |||||
| 22:13 STRING 7 "large" | |||||
| 22:20 EOL | |||||
| 23:5 KW_ELSE 4 | |||||
| 23:9 EOL | |||||
| 24:7 KW_PRINT 5 | |||||
| 24:13 IDENT 5 TOTAL | |||||
| 24:18 EOL | |||||
| 25:5 KW_END 3 | |||||
| 25:9 KW_IF 2 | |||||
| 25:11 EOL | |||||
| 26:5 KW_FOR 3 | |||||
| 26:9 IDENT 1 I | |||||
| 26:11 EQ 1 | |||||
| 26:13 NUMBER 1 0 | |||||
| 26:15 KW_TO 2 | |||||
| 26:18 NUMBER 1 3 | |||||
| 26:20 KW_STEP 4 | |||||
| 26:25 NUMBER 1 1 | |||||
| 26:26 EOL | |||||
| 27:7 KW_SELF 4 | |||||
| 27:11 DOT 1 | |||||
| 27:12 IDENT 4 TICK | |||||
| 27:16 LPAREN 1 | |||||
| 27:17 IDENT 1 I | |||||
| 27:18 RPAREN 1 | |||||
| 27:19 EOL | |||||
| 28:5 KW_END 3 | |||||
| 28:9 KW_FOR 3 | |||||
| 28:12 EOL | |||||
| 29:5 KW_FOR 3 | |||||
| 29:9 IDENT 1 I | |||||
| 29:11 EQ 1 | |||||
| 29:13 NUMBER 1 0 | |||||
| 29:15 KW_TO 2 | |||||
| 29:18 NUMBER 1 1 | |||||
| 29:19 EOL | |||||
| 30:7 KW_SELF 4 | |||||
| 30:11 DOT 1 | |||||
| 30:12 IDENT 4 TICK | |||||
| 30:16 LPAREN 1 | |||||
| 30:17 IDENT 1 I | |||||
| 30:18 RPAREN 1 | |||||
| 30:19 EOL | |||||
| 31:5 KW_END 3 | |||||
| 31:9 KW_FOR 3 | |||||
| 31:12 EOL | |||||
| 32:5 KW_RETURN 6 | |||||
| 32:12 IDENT 5 TOTAL | |||||
| 32:17 EOL | |||||
| 33:3 KW_END 3 | |||||
| 33:7 KW_METHOD 6 | |||||
| 33:13 EOL | |||||
| 34:1 KW_END 3 | |||||
| 34:5 KW_CLASS 5 | |||||
| 34:10 EOL | |||||
| 35:1 EOL | |||||
| 36:1 KW_PROGRAM 7 | |||||
| 36:9 IDENT 4 MAIN | |||||
| 36:13 EOL | |||||
| 37:3 KW_VAR 3 | |||||
| 37:7 IDENT 6 WORKER | |||||
| 37:14 KW_AS 2 | |||||
| 37:17 IDENT 6 WORKER | |||||
| 37:23 EOL | |||||
| 38:3 KW_VAR 3 | |||||
| 38:7 IDENT 5 TOTAL | |||||
| 38:13 KW_AS 2 | |||||
| 38:16 KW_WORD 4 | |||||
| 38:20 EOL | |||||
| 39:3 IDENT 6 WORKER | |||||
| 39:10 EQ 1 | |||||
| 39:12 KW_NEW 3 | |||||
| 39:16 IDENT 6 WORKER | |||||
| 39:22 LPAREN 1 | |||||
| 39:23 NUMBER 1 1 | |||||
| 39:24 RPAREN 1 | |||||
| 39:25 EOL | |||||
| 40:3 IDENT 6 WORKER | |||||
| 40:9 DOT 1 | |||||
| 40:10 IDENT 4 TICK | |||||
| 40:14 LPAREN 1 | |||||
| 40:15 NUMBER 1 1 | |||||
| 40:16 RPAREN 1 | |||||
| 40:17 EOL | |||||
| 41:3 IDENT 5 TOTAL | |||||
| 41:9 EQ 1 | |||||
| 41:11 IDENT 6 WORKER | |||||
| 41:17 DOT 1 | |||||
| 41:18 IDENT 3 RUN | |||||
| 41:21 LPAREN 1 | |||||
| 41:22 NUMBER 1 4 | |||||
| 41:23 RPAREN 1 | |||||
| 41:24 EOL | |||||
| 42:3 KW_PRINT 5 | |||||
| 42:9 IDENT 5 TOTAL | |||||
| 42:14 EOL | |||||
| 43:3 KW_DELETE 6 | |||||
| 43:10 IDENT 6 WORKER | |||||
| 43:16 EOL | |||||
| 44:1 KW_END 3 | |||||
| 44:5 KW_PROGRAM 7 | |||||
| 44:12 EOL | |||||
| 44:0 EOF | |||||
+ 119
- 0
newLang/tests/lexer/golden/tokens.tokens
Просмотреть файл
| @@ -0,0 +1,119 @@ | |||||
| 1:75 EOL | |||||
| 2:1 KW_PROGRAM 7 | |||||
| 2:9 IDENT 6 TOKENS | |||||
| 2:15 EOL | |||||
| 3:3 KW_VAR 3 | |||||
| 3:7 IDENT 4 A_B1 | |||||
| 3:12 KW_AS 2 | |||||
| 3:15 KW_WORD 4 | |||||
| 3:19 EOL | |||||
| 4:3 KW_VAR 3 | |||||
| 4:7 IDENT 16 ABCDEFGHIJKLMNOP | |||||
| 4:24 KW_AS 2 | |||||
| 4:27 KW_BYTE 4 | |||||
| 4:31 EOL | |||||
| 5:3 IDENT 4 A_B1 | |||||
| 5:8 EQ 1 | |||||
| 5:10 NUMBER 5 65535 | |||||
| 5:16 PLUS 1 | |||||
| 5:18 NUMBER 5 10 | |||||
| 5:24 PLUS 1 | |||||
| 5:26 NUMBER 5 65535 | |||||
| 5:32 PLUS 1 | |||||
| 5:34 NUMBER 1 0 | |||||
| 5:36 PLUS 1 | |||||
| 5:38 NUMBER 2 0 | |||||
| 5:41 PLUS 1 | |||||
| 5:43 NUMBER 2 0 | |||||
| 5:45 EOL | |||||
| 6:3 KW_END 3 | |||||
| 6:7 KW_PROGRAM 7 | |||||
| 6:15 KW_WHILE 5 | |||||
| 6:21 KW_WHILE 5 | |||||
| 6:26 EOL | |||||
| 7:3 IDENT 1 X | |||||
| 7:5 EQ 1 | |||||
| 7:7 LPAREN 1 | |||||
| 7:9 RPAREN 1 | |||||
| 7:11 LBRACKET 1 | |||||
| 7:13 RBRACKET 1 | |||||
| 7:15 COMMA 1 | |||||
| 7:17 DOT 1 | |||||
| 7:19 COLON 1 | |||||
| 7:21 EQ 1 | |||||
| 7:23 PLUS 1 | |||||
| 7:25 MINUS 1 | |||||
| 7:27 STAR 1 | |||||
| 7:29 SLASH 1 | |||||
| 7:31 LT 1 | |||||
| 7:33 GT 1 | |||||
| 7:35 NE 2 | |||||
| 7:38 LE 2 | |||||
| 7:41 GE 2 | |||||
| 7:43 EOL | |||||
| 8:3 IDENT 1 Y | |||||
| 8:4 EQ 1 | |||||
| 8:5 IDENT 1 A | |||||
| 8:6 NE 2 | |||||
| 8:8 IDENT 1 B | |||||
| 8:9 LE 2 | |||||
| 8:11 IDENT 1 C | |||||
| 8:12 GE 2 | |||||
| 8:14 IDENT 1 D | |||||
| 8:15 LT 1 | |||||
| 8:16 IDENT 1 E | |||||
| 8:17 GT 1 | |||||
| 8:18 IDENT 1 F | |||||
| 8:19 EOL | |||||
| 9:3 IDENT 1 S | |||||
| 9:5 EQ 1 | |||||
| 9:7 STRING 11 "q{22}{5C}{0D}{0D}" | |||||
| 9:19 PLUS 1 | |||||
| 9:21 STRING 2 "" | |||||
| 9:24 PLUS 1 | |||||
| 9:26 STRING 7 "x ' y" | |||||
| 9:33 EOL | |||||
| 10:3 IDENT 1 C | |||||
| 10:5 EQ 1 | |||||
| 10:7 CHAR 3 $58 | |||||
| 10:11 PLUS 1 | |||||
| 10:13 CHAR 4 $27 | |||||
| 10:18 PLUS 1 | |||||
| 10:20 CHAR 4 $5C | |||||
| 10:25 PLUS 1 | |||||
| 10:27 CHAR 4 $0D | |||||
| 10:32 PLUS 1 | |||||
| 10:34 CHAR 4 $0D | |||||
| 10:39 PLUS 1 | |||||
| 10:41 CHAR 3 $22 | |||||
| 10:44 EOL | |||||
| 11:1 EOL | |||||
| 12:2 IDENT 6 TABBED | |||||
| 12:9 EQ 1 | |||||
| 12:11 NUMBER 1 1 | |||||
| 12:29 EOL | |||||
| 13:3 IDENT 1 Z | |||||
| 13:5 EQ 1 | |||||
| 13:7 NUMBER 1 7 | |||||
| 13:9 SLASH 1 | |||||
| 13:11 NUMBER 1 2 | |||||
| 13:40 EOL | |||||
| 14:23 EOL | |||||
| 15:3 KW_SELF 4 | |||||
| 15:7 DOT 1 | |||||
| 15:8 IDENT 4 DATA | |||||
| 15:12 LBRACKET 1 | |||||
| 15:13 IDENT 1 I | |||||
| 15:14 RBRACKET 1 | |||||
| 15:16 EQ 1 | |||||
| 15:18 KW_NOT 3 | |||||
| 15:22 KW_TRUE 4 | |||||
| 15:27 KW_AND 3 | |||||
| 15:31 KW_FALSE 5 | |||||
| 15:37 KW_OR 2 | |||||
| 15:40 KW_NULL 4 | |||||
| 15:44 EOL | |||||
| 16:1 KW_END 3 | |||||
| 16:5 KW_PROGRAM 7 | |||||
| 16:12 EOL | |||||
| 16:0 EOF | |||||
+ 286
- 0
newLang/tests/lexer/lexer-test.s
Просмотреть файл
| @@ -0,0 +1,286 @@ | |||||
| ; E02-S02: lexer token-dump test in x128. | |||||
| ; | |||||
| ; For each fixture: src_load, then lex_next until TOK_EOF, appending every | |||||
| ; token to `dump`. The host decodes the dump to text and compares it with the | |||||
| ; golden files in tests/lexer/golden/. | |||||
| ; | |||||
| ; Dump format: | |||||
| ; fixture: $FF, fixture index, src_load result | |||||
| ; token: kind, line lo, line hi, col, then by kind: | |||||
| ; EOL, EOF nothing | |||||
| ; ERROR error code | |||||
| ; IDENT, STRING raw length, text length, text bytes | |||||
| ; NUMBER raw length, value lo, value hi | |||||
| ; CHAR raw length, value | |||||
| ; other raw length | |||||
| ; end: $FE | |||||
| ; Mailbox: PASS if every fixture loaded and the dump did not overflow. | |||||
| ; MB_DATA+0/1 dump address, +2/3 dump length, +4 fixture count. | |||||
| .setcpu "6502" | |||||
| .include "../../src/common/c128.inc" | |||||
| .include "../../src/common/mailbox.inc" | |||||
| .include "../../src/common/tokens.inc" | |||||
| .import mmu_init, mmu_bank0, mmu_shutdown | |||||
| .import src_load | |||||
| .import lex_reset, lex_next | |||||
| .import tok_kind, tok_line, tok_col, tok_len, tok_value, tok_text, tok_text_len, tok_err | |||||
| DUMP_SIZE = $4000 | |||||
| MAX_TOKENS_PER_FIXTURE = 3000 | |||||
| .segment "LOADADDR" | |||||
| .word $1C01 | |||||
| .segment "EXEHDR" | |||||
| .word basic_end | |||||
| .word 10 | |||||
| .byte $9E | |||||
| .byte "7181" | |||||
| .byte $00 | |||||
| basic_end: | |||||
| .word $0000 | |||||
| .segment "MAILBOX" | |||||
| .res MB_SIZE | |||||
| .segment "ZEROPAGE" | |||||
| dump_ptr: .res 2 | |||||
| fx_ptr: .res 2 | |||||
| .segment "STARTUP" | |||||
| .export test_entry, dump | |||||
| test_entry: | |||||
| php | |||||
| sei | |||||
| ldx #MB_SIZE - 1 | |||||
| lda #$00 | |||||
| @clr: sta MB_BASE,x | |||||
| dex | |||||
| bpl @clr | |||||
| lda #MB_MAGIC0_VALUE | |||||
| sta MB_MAGIC0 | |||||
| lda #MB_MAGIC1_VALUE | |||||
| sta MB_MAGIC1 | |||||
| lda #MB_PROD_PLATFORM | |||||
| sta MB_PRODUCER | |||||
| lda #MB_ST_RUNNING | |||||
| sta MB_STATUS | |||||
| jsr mmu_init | |||||
| jsr mmu_bank0 | |||||
| lda #<dump | |||||
| sta dump_ptr | |||||
| lda #>dump | |||||
| sta dump_ptr+1 | |||||
| lda #<fixtures | |||||
| sta fx_ptr | |||||
| lda #>fixtures | |||||
| sta fx_ptr+1 | |||||
| lda #$00 | |||||
| sta failures | |||||
| sta overflow | |||||
| sta fx_index | |||||
| @fixture: | |||||
| lda #$FF | |||||
| jsr emit | |||||
| lda fx_index | |||||
| jsr emit | |||||
| ldy #0 | |||||
| lda (fx_ptr),y | |||||
| sta name_lo | |||||
| iny | |||||
| lda (fx_ptr),y | |||||
| tax | |||||
| iny | |||||
| lda (fx_ptr),y | |||||
| tay | |||||
| lda name_lo | |||||
| jsr src_load | |||||
| pha | |||||
| jsr emit | |||||
| pla | |||||
| beq @lex | |||||
| inc failures | |||||
| jmp @advance | |||||
| @lex: jsr lex_fixture | |||||
| @advance: | |||||
| clc | |||||
| lda fx_ptr | |||||
| adc #3 | |||||
| sta fx_ptr | |||||
| bcc :+ | |||||
| inc fx_ptr+1 | |||||
| : inc fx_index | |||||
| lda fx_index | |||||
| cmp #FIXTURE_COUNT | |||||
| bne @fixture | |||||
| lda #$FE | |||||
| jsr emit | |||||
| ; Report before mmu_shutdown: dump_ptr is in the private zero page, | |||||
| ; which the system mapping no longer shows. | |||||
| lda #<dump | |||||
| sta MB_DATA+0 | |||||
| lda #>dump | |||||
| sta MB_DATA+1 | |||||
| sec | |||||
| lda dump_ptr | |||||
| sbc #<dump | |||||
| sta MB_DATA+2 | |||||
| lda dump_ptr+1 | |||||
| sbc #>dump | |||||
| sta MB_DATA+3 | |||||
| lda #FIXTURE_COUNT | |||||
| sta MB_DATA+4 | |||||
| jsr mmu_shutdown | |||||
| lda failures | |||||
| ora overflow | |||||
| sta MB_CODE | |||||
| ldy #ok_msg - messages | |||||
| lda MB_CODE | |||||
| bne @fail | |||||
| lda #MB_ST_PASS | |||||
| bne @report | |||||
| @fail: ldy #fail_msg - messages | |||||
| lda #MB_ST_FAIL | |||||
| @report: | |||||
| sta MB_STATUS | |||||
| plp | |||||
| @print: lda messages,y | |||||
| beq @end | |||||
| sty print_y | |||||
| jsr CHROUT | |||||
| ldy print_y | |||||
| iny | |||||
| bne @print | |||||
| @end: rts | |||||
| .segment "CODE" | |||||
| lex_fixture: | |||||
| jsr lex_reset | |||||
| lda #<MAX_TOKENS_PER_FIXTURE | |||||
| sta budget | |||||
| lda #>MAX_TOKENS_PER_FIXTURE | |||||
| sta budget+1 | |||||
| @token: jsr lex_next | |||||
| jsr emit_token | |||||
| lda tok_kind | |||||
| cmp #TOK_EOF | |||||
| beq @done | |||||
| lda budget ; runaway guard | |||||
| bne :+ | |||||
| dec budget+1 | |||||
| : dec budget | |||||
| lda budget | |||||
| ora budget+1 | |||||
| bne @token | |||||
| inc failures | |||||
| @done: rts | |||||
| emit_token: | |||||
| lda tok_kind | |||||
| jsr emit | |||||
| lda tok_line | |||||
| jsr emit | |||||
| lda tok_line+1 | |||||
| jsr emit | |||||
| lda tok_col | |||||
| jsr emit | |||||
| lda tok_kind | |||||
| cmp #TOK_EOF | |||||
| beq @none | |||||
| cmp #TOK_EOL | |||||
| beq @none | |||||
| cmp #TOK_ERROR | |||||
| bne @len | |||||
| lda tok_err | |||||
| jmp emit | |||||
| @none: rts | |||||
| @len: lda tok_len | |||||
| jsr emit | |||||
| lda tok_kind | |||||
| cmp #TOK_IDENT | |||||
| beq @text | |||||
| cmp #TOK_STRING | |||||
| beq @text | |||||
| cmp #TOK_NUMBER | |||||
| beq @number | |||||
| cmp #TOK_CHAR | |||||
| bne @none | |||||
| lda tok_value | |||||
| jmp emit | |||||
| @number: | |||||
| lda tok_value | |||||
| jsr emit | |||||
| lda tok_value+1 | |||||
| jmp emit | |||||
| @text: lda tok_text_len | |||||
| jsr emit | |||||
| ldx #$00 | |||||
| @byte: cpx tok_text_len | |||||
| beq @none | |||||
| lda tok_text,x | |||||
| stx emit_x | |||||
| jsr emit | |||||
| ldx emit_x | |||||
| inx | |||||
| bne @byte | |||||
| ; Append A to the dump; stop writing (and flag) once it is full. | |||||
| emit: ldy dump_ptr+1 | |||||
| cpy #>(dump + DUMP_SIZE) | |||||
| bcc @room | |||||
| lda #$01 | |||||
| sta overflow | |||||
| rts | |||||
| @room: ldy #$00 | |||||
| sta (dump_ptr),y | |||||
| inc dump_ptr | |||||
| bne :+ | |||||
| inc dump_ptr+1 | |||||
| : rts | |||||
| .segment "RODATA" | |||||
| ; name address, name length | |||||
| fixtures: | |||||
| .macro FIXTURE name | |||||
| .local text, done | |||||
| .word text | |||||
| .byte done - text | |||||
| .pushseg | |||||
| .segment "DATA" ; names live apart from the table | |||||
| text: .byte name | |||||
| done: | |||||
| .popseg | |||||
| .endmacro | |||||
| FIXTURE "MINIMAL.N8" | |||||
| FIXTURE "DECLS.N8" | |||||
| FIXTURE "ARRAYS.N8" | |||||
| FIXTURE "EXPRS.N8" | |||||
| FIXTURE "FORWARD.N8" | |||||
| FIXTURE "NULLS.N8" | |||||
| FIXTURE "STMTS.N8" | |||||
| FIXTURE "TOKENS.N8" | |||||
| FIXTURE "ERRORS.N8" | |||||
| FIXTURE "BADCHAR.N8" | |||||
| FIXTURE_COUNT = 10 | |||||
| messages: | |||||
| ok_msg: .byte "NOVA128 LEXER TEST OK", $0D, $00 | |||||
| fail_msg: .byte "NOVA128 LEXER TEST FAIL", $0D, $00 | |||||
| .segment "BSS" | |||||
| failures: .res 1 | |||||
| overflow: .res 1 | |||||
| fx_index: .res 1 | |||||
| name_lo: .res 1 | |||||
| budget: .res 2 | |||||
| emit_x: .res 1 | |||||
| print_y: .res 1 | |||||
| dump: .res DUMP_SIZE | |||||
+ 219
- 0
newLang/tests/lexer/reference-lexer.ps1
Просмотреть файл
| @@ -0,0 +1,219 @@ | |||||
| # Host reference lexer for Nova128 (E02-S02). Dot-source this file. | |||||
| # | |||||
| # An independent implementation of docs/language-spec.md 2.2 plus the token | |||||
| # rules documented at the top of src/compiler/lexer.s. Golden files must be | |||||
| # reproduced by BOTH this lexer and the C128 lexer. | |||||
| # | |||||
| # Token text format, one per line: | |||||
| # L:C KIND [raw-length] [detail] | |||||
| # IDENT detail = normalized name (PETSCII $41-$5A shown as A-Z) | |||||
| # NUMBER detail = decimal value | |||||
| # STRING detail = "text" (see Format-PetsciiText) | |||||
| # CHAR detail = $hh | |||||
| # ERROR detail = LEXERR name (no raw length) | |||||
| # EOL/EOF no length | |||||
| $script:Keywords = @{ | |||||
| "AND" = "KW_AND"; "AS" = "KW_AS"; "BOOLEAN" = "KW_BOOLEAN"; "BYTE" = "KW_BYTE" | |||||
| "CHAR" = "KW_CHAR"; "CLASS" = "KW_CLASS"; "DELETE" = "KW_DELETE"; "ELSE" = "KW_ELSE" | |||||
| "END" = "KW_END"; "FALSE" = "KW_FALSE"; "FOR" = "KW_FOR"; "IF" = "KW_IF" | |||||
| "IMPLEMENTS" = "KW_IMPLEMENTS"; "INTEGER" = "KW_INTEGER"; "INTERFACE" = "KW_INTERFACE" | |||||
| "METHOD" = "KW_METHOD"; "NEW" = "KW_NEW"; "NOT" = "KW_NOT"; "NULL" = "KW_NULL" | |||||
| "OR" = "KW_OR"; "PRINT" = "KW_PRINT"; "PRIVATE" = "KW_PRIVATE"; "PROGRAM" = "KW_PROGRAM" | |||||
| "PUBLIC" = "KW_PUBLIC"; "RETURN" = "KW_RETURN"; "SELF" = "KW_SELF"; "STEP" = "KW_STEP" | |||||
| "TO" = "KW_TO"; "TRUE" = "KW_TRUE"; "VAR" = "KW_VAR"; "WHILE" = "KW_WHILE"; "WORD" = "KW_WORD" | |||||
| } | |||||
| # Token kind numbers from src/common/tokens.inc (used to decode C128 dumps). | |||||
| $script:KindNames = @{ | |||||
| 0x00 = "EOF"; 0x01 = "EOL"; 0x02 = "IDENT"; 0x03 = "NUMBER"; 0x04 = "STRING" | |||||
| 0x05 = "CHAR"; 0x06 = "ERROR" | |||||
| 0x10 = "LPAREN"; 0x11 = "RPAREN"; 0x12 = "LBRACKET"; 0x13 = "RBRACKET"; 0x14 = "COMMA" | |||||
| 0x15 = "DOT"; 0x16 = "COLON"; 0x17 = "EQ"; 0x18 = "PLUS"; 0x19 = "MINUS"; 0x1A = "STAR" | |||||
| 0x1B = "SLASH"; 0x1C = "LT"; 0x1D = "GT"; 0x1E = "NE"; 0x1F = "LE"; 0x20 = "GE" | |||||
| } | |||||
| $i = 0x40 | |||||
| foreach ($kw in ($script:Keywords.Keys | Sort-Object)) { | |||||
| $script:KindNames[$i] = $script:Keywords[$kw]; $i++ | |||||
| } | |||||
| $script:ErrorNames = @{ | |||||
| 1 = "INVALID_CHAR"; 2 = "IDENT_TOO_LONG"; 3 = "NUMBER_RANGE"; 4 = "BAD_NUMBER" | |||||
| 5 = "UNTERMINATED_STRING"; 6 = "BAD_ESCAPE"; 7 = "BAD_CHAR_LITERAL" | |||||
| } | |||||
| $script:Punct = @{ | |||||
| 0x28 = "LPAREN"; 0x29 = "RPAREN"; 0x5B = "LBRACKET"; 0x5D = "RBRACKET"; 0x2C = "COMMA" | |||||
| 0x2E = "DOT"; 0x3A = "COLON"; 0x3D = "EQ"; 0x2B = "PLUS"; 0x2D = "MINUS"; 0x2A = "STAR" | |||||
| 0x2F = "SLASH"; 0x3C = "LT"; 0x3E = "GT" | |||||
| } | |||||
| function Format-PetsciiText([byte[]]$bytes) { | |||||
| $sb = New-Object System.Text.StringBuilder | |||||
| foreach ($b in $bytes) { | |||||
| if ($b -ge 0x41 -and $b -le 0x5A) { [void]$sb.Append([char]($b + 0x20)) } # a-z | |||||
| elseif ($b -ge 0xC1 -and $b -le 0xDA) { [void]$sb.Append([char]($b - 0x80)) } # A-Z | |||||
| elseif ($b -ge 0x20 -and $b -le 0x3F -and $b -ne 0x22) { [void]$sb.Append([char]$b) } | |||||
| else { [void]$sb.Append(("{{{0:X2}}}" -f $b)) } | |||||
| } | |||||
| return '"' + $sb.ToString() + '"' | |||||
| } | |||||
| function Format-Name([byte[]]$bytes) { | |||||
| return -join ($bytes | ForEach-Object { [char]$_ }) # $41-$5A, digits, $5F | |||||
| } | |||||
| function Test-Letter([int]$b) { return ($b -ge 0x41 -and $b -le 0x5A) -or ($b -ge 0xC1 -and $b -le 0xDA) } | |||||
| function Test-Digit([int]$b) { return $b -ge 0x30 -and $b -le 0x39 } | |||||
| function Test-IdentPart([int]$b) { return (Test-Letter $b) -or (Test-Digit $b) -or $b -eq 0x5F } | |||||
| # Returns the token lines for one PETSCII source. | |||||
| function Get-ReferenceTokens([byte[]]$Source) { | |||||
| $out = New-Object System.Collections.Generic.List[string] | |||||
| $lines = New-Object System.Collections.Generic.List[byte[]] | |||||
| $start = 0 | |||||
| for ($k = 0; $k -lt $Source.Length; $k++) { | |||||
| if ($Source[$k] -eq 13) { | |||||
| $lines.Add([byte[]]$(if ($k -gt $start) { $Source[$start..($k - 1)] } else { @() })) | |||||
| $start = $k + 1 | |||||
| } | |||||
| } | |||||
| if ($start -lt $Source.Length) { $lines.Add([byte[]]$Source[$start..($Source.Length - 1)]) } | |||||
| for ($ln = 0; $ln -lt $lines.Count; $ln++) { | |||||
| $line = $lines[$ln]; $n = $line.Length; $lineNo = $ln + 1 | |||||
| $at = { param($p) if ($p -lt $n) { [int]$line[$p] } else { 13 } } | |||||
| $p = 0 | |||||
| while ($true) { | |||||
| while ((& $at $p) -eq 0x20 -or (& $at $p) -eq 0x09) { $p++ } | |||||
| $c = & $at $p | |||||
| if ($c -eq 13) { break } | |||||
| $col = $p + 1; $s = $p | |||||
| $emit = { param($kind, $detail) $out.Add(("{0}:{1} {2} {3}{4}" -f $lineNo, $col, $kind, ($p - $s), $detail)) } | |||||
| $err = $null; $errCol = $col | |||||
| if (Test-Letter $c) { | |||||
| $name = New-Object System.Collections.Generic.List[byte] | |||||
| while (Test-IdentPart (& $at $p)) { | |||||
| $b = & $at $p | |||||
| if (Test-Letter $b) { $b = $b -band 0x7F } | |||||
| $name.Add([byte]$b); $p++ | |||||
| } | |||||
| if ($name.Count -gt 16) { $err = "IDENT_TOO_LONG" } | |||||
| else { | |||||
| $text = Format-Name $name.ToArray() | |||||
| if ($script:Keywords.ContainsKey($text)) { & $emit $script:Keywords[$text] "" } | |||||
| else { & $emit "IDENT" (" " + $text) } | |||||
| } | |||||
| } elseif ((Test-Digit $c) -or $c -eq 0x24 -or $c -eq 0x25) { | |||||
| $value = [long]0; $count = 0 | |||||
| if ($c -eq 0x24) { | |||||
| $p++ | |||||
| while ($true) { | |||||
| $b = & $at $p; $d = -1 | |||||
| if (Test-Digit $b) { $d = $b - 0x30 } | |||||
| elseif ((Test-Letter $b) -and ($b -band 0x7F) -ge 0x41 -and ($b -band 0x7F) -le 0x46) { | |||||
| $d = ($b -band 0x7F) - 0x41 + 10 | |||||
| } | |||||
| if ($d -lt 0) { break } | |||||
| $value = $value * 16 + $d; $count++; $p++ | |||||
| } | |||||
| } elseif ($c -eq 0x25) { | |||||
| $p++ | |||||
| while ((& $at $p) -eq 0x30 -or (& $at $p) -eq 0x31) { | |||||
| $value = $value * 2 + ((& $at $p) - 0x30); $count++; $p++ | |||||
| } | |||||
| } else { | |||||
| while (Test-Digit (& $at $p)) { $value = $value * 10 + ((& $at $p) - 0x30); $count++; $p++ } | |||||
| } | |||||
| if ($count -eq 0 -or (Test-IdentPart (& $at $p))) { $err = "BAD_NUMBER" } | |||||
| elseif ($value -gt 65535) { $err = "NUMBER_RANGE" } | |||||
| else { & $emit "NUMBER" (" " + $value) } | |||||
| } elseif ($c -eq 0x22) { | |||||
| $p++; $text = New-Object System.Collections.Generic.List[byte] | |||||
| while ($null -eq $err) { | |||||
| $b = & $at $p | |||||
| if ($b -eq 13) { $err = "UNTERMINATED_STRING"; break } | |||||
| if ($b -eq 0x22) { $p++; break } | |||||
| if ($b -eq 0x5C) { | |||||
| $e = & $at ($p + 1) | |||||
| if ($e -eq 0x5C -or $e -eq 0x22) { $b = $e } | |||||
| elseif (($e -band 0x7F) -eq 0x4E -or ($e -band 0x7F) -eq 0x52) { $b = 0x0D } | |||||
| else { $err = "BAD_ESCAPE"; $errCol = $p + 1; break } | |||||
| $p++ | |||||
| } | |||||
| $text.Add([byte]$b); $p++ | |||||
| } | |||||
| if ($null -eq $err) { & $emit "STRING" (" " + (Format-PetsciiText $text.ToArray())) } | |||||
| } elseif ($c -eq 0x27) { | |||||
| $p++; $b = & $at $p | |||||
| if ($b -eq 13 -or $b -eq 0x27) { $err = "BAD_CHAR_LITERAL" } | |||||
| else { | |||||
| if ($b -eq 0x5C) { | |||||
| $e = & $at ($p + 1) | |||||
| if ($e -eq 0x5C -or $e -eq 0x27) { $b = $e } | |||||
| elseif (($e -band 0x7F) -eq 0x4E -or ($e -band 0x7F) -eq 0x52) { $b = 0x0D } | |||||
| else { $err = "BAD_ESCAPE"; $errCol = $p + 1 } | |||||
| $p++ | |||||
| } | |||||
| if ($null -eq $err) { | |||||
| $p++ | |||||
| if ((& $at $p) -ne 0x27) { $err = "BAD_CHAR_LITERAL" } | |||||
| else { $p++; & $emit "CHAR" (" `${0:X2}" -f $b) } | |||||
| } | |||||
| } | |||||
| } elseif ($script:Punct.ContainsKey($c)) { | |||||
| $next = & $at ($p + 1) | |||||
| if ($c -eq 0x2F -and $next -eq 0x2F) { break } # comment | |||||
| $p++ | |||||
| if ($c -eq 0x3C -and $next -eq 0x3E) { $p++; & $emit "NE" "" } | |||||
| elseif ($c -eq 0x3C -and $next -eq 0x3D) { $p++; & $emit "LE" "" } | |||||
| elseif ($c -eq 0x3E -and $next -eq 0x3D) { $p++; & $emit "GE" "" } | |||||
| else { & $emit $script:Punct[$c] "" } | |||||
| } else { | |||||
| $err = "INVALID_CHAR" | |||||
| } | |||||
| if ($null -ne $err) { | |||||
| $out.Add(("{0}:{1} ERROR {2}" -f $lineNo, $errCol, $err)) | |||||
| break # skip the line | |||||
| } | |||||
| } | |||||
| $out.Add(("{0}:{1} EOL" -f $lineNo, ($n + 1))) | |||||
| } | |||||
| $out.Add(("{0}:0 EOF" -f $lines.Count)) | |||||
| return ,$out | |||||
| } | |||||
| # Decode the C128 dump (tests/lexer/lexer-test.s) into per-fixture token lines. | |||||
| # Returns a list (in dump order) of @{ Index; Load; Tokens }. | |||||
| function ConvertFrom-TokenDump([byte[]]$Dump) { | |||||
| $result = New-Object System.Collections.Generic.List[object] | |||||
| $p = 0; $current = $null | |||||
| while ($p -lt $Dump.Length) { | |||||
| $kind = $Dump[$p] | |||||
| if ($kind -eq 0xFE) { break } | |||||
| if ($kind -eq 0xFF) { | |||||
| $current = [pscustomobject]@{ Index = [int]$Dump[$p + 1]; Load = $Dump[$p + 2] | |||||
| Tokens = New-Object System.Collections.Generic.List[string] } | |||||
| $result.Add($current) | |||||
| $p += 3; continue | |||||
| } | |||||
| $line = $Dump[$p + 1] + 256 * $Dump[$p + 2]; $col = $Dump[$p + 3]; $p += 4 | |||||
| $name = $script:KindNames[[int]$kind] | |||||
| if ($null -eq $name) { throw ("unknown token kind `${0:X2} in dump" -f $kind) } | |||||
| switch ($name) { | |||||
| "EOF" { $current.Tokens.Add("${line}:$col EOF") } | |||||
| "EOL" { $current.Tokens.Add("${line}:$col EOL") } | |||||
| "ERROR" { $current.Tokens.Add("${line}:$col ERROR " + $script:ErrorNames[[int]$Dump[$p]]); $p++ } | |||||
| { $_ -in "IDENT", "STRING" } { | |||||
| $len = $Dump[$p]; $tl = $Dump[$p + 1] | |||||
| $text = if ($tl -gt 0) { [byte[]]$Dump[($p + 2)..($p + 1 + $tl)] } else { [byte[]]@() } | |||||
| $detail = if ($name -eq "IDENT") { Format-Name $text } else { Format-PetsciiText $text } | |||||
| $current.Tokens.Add("${line}:$col $name $len $detail"); $p += 2 + $tl | |||||
| } | |||||
| "NUMBER" { $current.Tokens.Add("${line}:$col NUMBER $($Dump[$p]) $($Dump[$p + 1] + 256 * $Dump[$p + 2])"); $p += 3 } | |||||
| "CHAR" { $current.Tokens.Add(("{0}:{1} CHAR {2} `${3:X2}" -f $line, $col, $Dump[$p], $Dump[$p + 1])); $p += 2 } | |||||
| default { $current.Tokens.Add("${line}:$col $name $($Dump[$p])"); $p++ } | |||||
| } | |||||
| } | |||||
| return ,$result | |||||
| } | |||||
+ 16
- 0
newLang/tests/lexer/tokens.n8
Просмотреть файл
| @@ -0,0 +1,16 @@ | |||||
| // Lexical edge cases for E02-S02. Every line here must lex without error. | |||||
| program Tokens | |||||
| var a_b1 as word | |||||
| var ABCDEFGHIJKLMNOP as byte | |||||
| a_b1 = $ffFF + %1010 + 65535 + 0 + $0 + %0 | |||||
| END Program WHILE wHiLe | |||||
| x = ( ) [ ] , . : = + - * / < > <> <= >= | |||||
| y=a<>b<=c>=d<e>f | |||||
| s = "q\"\\\n\r" + "" + "x ' y" | |||||
| c = 'x' + '\'' + '\\' + '\n' + '\R' + '"' | |||||
| tabbed = 1 // tab separated | |||||
| z = 7 / 2 // division, then a comment | |||||
| // comment-only line | |||||
| self.data[i] = not true and false or null | |||||
| end program | |||||