Expr/SyntaxTree parity: trivial conversions#60373
Expr/SyntaxTree parity: trivial conversions#60373topolarity merged 3 commits intoJuliaLang:masterfrom
Expr/SyntaxTree parity: trivial conversions#60373Conversation
Merged!
At what point do we replace |
PR 4 in the list replaces it completely from the perspective of macro authors. Internal to lowering, we can make individual changes to |
mlechu
force-pushed
the
est-2-expr-est
branch
from
1f327e5 to
64ec26d
Compare
I'd prefer to keep CST and RawGreenNode close together, and aim for CST to eventually be the new AST once we release new-style macros. In the meantime it's an implementation detail which we can keep refining as we see fit. My very strong preference is that we don't get stuck with the |
mlechu
force-pushed
the
est-2-expr-est
branch
from
8d6d49c to
88a9977
Compare
topolarity
left a comment
topolarity
left a comment
There was a problem hiding this comment.
Had a good chat with @c42f and @mlechu about this change today. Sounds like we're OK with this direction for now, although the question about how to do syntax evolution (and whether that's better done in a user-facing CST that diverges from Expr, or by evolving EST / Expr in lock-step) remains somewhat open.
For now, the idea is to shoot for parity since it avoids correct-ness issues around macros consuming / generating ~invalid Julia AST. "New macro" authors will write / process EST. Those authors are just us anyway until "new" macros are polished and released to external users, at which point we can revisit this (presumably we'll have a nice, battle-tested pipeline at that point)
Step 4 in the the plan at #60373. Main changes: - Delete old `expr_to_syntaxtree` (hooray) - Introduce `est_to_dst`, which does a similar conversion as the above, but can assume all macros have expanded. (I've chosen "dst" for to mean "desugarable syntax tree" instead of the "current syntax tree" naming I've been using thus far) - Update macro expansion to be a function EST->EST - Add the AST validator from JuliaLang/JuliaLowering.jl#93, which I've changed to work on Expr structure. It's included in this PR so that `est_to_dst` doesn't need to do all the same checking (since nothing can be assumed about the output of macro expansion). Most of the long write-up there still applies, and using the same assumptions in desugaring could make it much less messy in the future. - This also means we're now able to return multiple AST-related syntax errors per top-level thunk, which may be of interest to @aviatesk. I haven't implemented pretty-printing for this beyond a `for` loop, though. - We now go though the `Core.@doc`system instead of the `K"doc"`system, which should be a temporary solution before we (1) get JuliaLowering working, then (2) handle docstrings in lowering in both implementations. - Many tests have been updated with the changes in behaviour. I've deleted a few where the point of the test is no longer relevant, but otherwise tried to maintain the spirit of each test. - Changes in syntax_macros.jl to produce the new old AST. The largest change is to our version of `@ccall`. Desugaring has also been tweaked so that it can tolerate `@ccall` expanded by flisp. stdlib status: There shouldn't be any regressions, and TOML and LibGit2 now precompile. Atop #60710 --------- Co-authored-by: Cody Tapscott <topolarity@tapscott.me>
3 participants
As discussed on slack with @c42f and @topolarity, I gave expansion-time conversions an honest try, but couldn't see the end of the tunnel in terms of correctness. The equivalence between SyntaxNode and Expr is only defined on surface syntax or fully-macro-expanded syntax, and not the steps in between, or if errors occur. We have a plan for syntax evolution now, too, so there compelling reasons not to lump changes to AST layout into JuliaLowering's macro expansion.
Let
ESTbe the "new" Expr-like SyntaxTree I'm working on andCSTbe the current SyntaxNode-like version. PRs planned:SyntaxGraph/SyntaxTreefrom JuliaLowering to JuliaSyntax #60370EST<->Exprconversion, tested by parsing code in bulk and round-trippingRawGreenNode->EST(identical logic to existingRawGreenNode->Expr, and testable with this PR'sExpr->EST)EST->CST*: This would just be a simplification of our existingExpr->CST, and prevents us from needing to make huge changes in desugaring (which understandsCST*). Macro expansion will becomeEST->ESTrather thanCST->CST*.