Commit f741e6a
committed
fix: Handling of non-ASCII identifiers
Non-ASCII-identifiers where not handled safely:
```
>>> ö : Int
thread 'main' panicked at ~/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokay-0.6.12/src/compiler/ast.rs:1169:42:
byte index 1 is not a char boundary; it is inside 'ö' (bytes 0..2) of `ö`
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```1 parent 3b473ba commit f741e6a
3 files changed
Lines changed: 41 additions & 15 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
251 | 251 | | |
252 | 252 | | |
253 | 253 | | |
254 | | - | |
| 254 | + | |
255 | 255 | | |
256 | 256 | | |
257 | 257 | | |
| |||
570 | 570 | | |
571 | 571 | | |
572 | 572 | | |
573 | | - | |
| 573 | + | |
574 | 574 | | |
575 | 575 | | |
576 | 576 | | |
| |||
591 | 591 | | |
592 | 592 | | |
593 | 593 | | |
594 | | - | |
| 594 | + | |
595 | 595 | | |
596 | 596 | | |
597 | 597 | | |
| |||
1165 | 1165 | | |
1166 | 1166 | | |
1167 | 1167 | | |
1168 | | - | |
1169 | | - | |
| 1168 | + | |
| 1169 | + | |
1170 | 1170 | | |
1171 | 1171 | | |
1172 | 1172 | | |
| |||
1177 | 1177 | | |
1178 | 1178 | | |
1179 | 1179 | | |
1180 | | - | |
| 1180 | + | |
1181 | 1181 | | |
1182 | 1182 | | |
1183 | 1183 | | |
1184 | 1184 | | |
1185 | 1185 | | |
1186 | | - | |
1187 | | - | |
| 1186 | + | |
| 1187 | + | |
1188 | 1188 | | |
1189 | 1189 | | |
1190 | 1190 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | | - | |
| 16 | + | |
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| 27 | + | |
| 28 | + | |
26 | 29 | | |
27 | 30 | | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
0 commit comments