More ergonomic morph alternative without clone

[therealbnut]

Hi,

Thanks for the library, I'm enjoying using it! I find that I often want to break up the lexing into different modes, but morph can often seem quite heavy due to the clone.

Since initially submitting this PR I got the Sublexer working ergonomically like I wanted to.

The PR now introduces two methods on Lexer:

• sublexer_with which creates a new Lexer for a different token type wrapped in a guard structure, it receives a closure which can be used to derive Extras from the parent lexer.
• sublexer works the same as sublexer_with except it uses Default for Extras rather than a closure.

When Sublexer is dropped it updates the span on the calling lexer.

I think this is much more ergonomic than the current morph based methods. What do you think?

NOTE: Besides the ergonomics of the guard, this also allows you to take the parent lexer's extras by reference, if that's useful.

Old PR description

This PR introduces two new methods: - `morph_next`, which constructs default extras for a child lexer, then calls next on it. - `morph_next_with_extras`, which allows you to borrow from the parent lexer's extras to construct the child's extras, then calls `next` on it.

As seen in the tests it lets you construct an umbrella Logos conforming type that allows the composition of different lexers in an ergonomic way. This example keeps the umbrella type responsible for maintaining the mode, however the morph_next_with_extras could allow more complex lexers to shared responsibility without knowledge of each other's token types.

I also considered other approaches:

• Methods to copy or synchronize scopes between lexers, but this required too much manual state synchronisation.
• A MorphMut guard type, which gets the parent's extras using BorrowMut then automatically synchronises the spans on Drop, however this needed a lot of types in its signature, and was ultimately less flexible and less ergonomic.
• A method morph_with, which takes a FnOnce(&mut Lexer<Child>)->T closure that lets you use the child while it's in scope. It had many of the same issues as MorphMut.

[therealbnut]

The last commit (71cc3e7) is quite large, but is what I believe is needed to make extras in a sublexer borrowable. It adds a lifetime to the Extras associated type on the Logos trait, and adjusts the derive macro to have an 'e type to refer to it. An example of this can be seen on the Alpha type in tests/tests/sublexer.rs.

I imagine this will require some code changes for some users of the library, if they manually implement the Logos trait. It may be as simple as adding <'e> to the Extras type in almost every case.

[jeertmans]

Hi, sorry for the delay in reviewing, my time for this project is very limited recently. Could you try to sync your branch with the new master branch? Also, do you think your PR is compatible with #542?

[therealbnut]

@jeertmans No worries! It’ll take me a little bit to get across these changes again, so I suggest merging the other PR first then ping me and then I should be able to quickly update this one to make sure it still works.

[jeertmans]

Hi @therealbnut, the other PR has been merged :-)

[therealbnut]

@jeertmans pushed those changes, I tried to keep them minimal but as I needed to add a lifetime parameter to the lexer there was only so much I could do. Although possibly the lexer's lifetime is already the same as extras and not source?

This probably needs some book changes too but I'm not very familiar with it. Effectively:

• Lexer gained an extras lifetime, it can often be set to a new lifetime, '_, or 'static depending on the context.
• Logos::Extras gained an extras lifetime (it can be set to something like 'ext and ignored if you don't need it.
• #[logos] gained an #[logos(extras_lifetime = 'extras)] parameter

[therealbnut]

Some of these changes aren't necessary, like this, they're a leftover from when I was adding multiple lifetime traits to Logos. I can revert them if you want, but I think it's a little more consistent with other parts of the codebase.