Code generation and delegation to nested routes

[parsonsmatt]

Nested Route Separation

We have almost 2,000 total routes in our codebase now, with 1,100 of them on the top-level.
We have been leveraging nested routes to organize the routes and make certain things easier - namely, reducing the overall size of the Route WebApp which makes pattern matching more efficient in compilation.

However, compiling mkYesodData for our application is a large bottleneck.
Compilation takes a significant amount of time and much work must be redone, even when most of the routing information could have been saved.
One thing that would significantly help is the ability to separate out the sub-route datatypes, generate the instance for these separately, and refer to them elsewhere.

Another thing that would help is having a finer grained YesodDispatch facility - allowing us to, in tests, specify more precisely which parts of the route structure we actually need to deal with.
Currently, we need almost 7k modules in order to compile a test that refers to YesodDispatch, since this brings in the transitive dependencies of every web handler.
With a finer grained YesodDispatch, able to refer to route fragments, we'd be able to avoid depending on more of the website than we really need to.
We expect this to yield significant benefits when we move more testing infrastructure to buck2 - right now, changing anything used by any part of the web app would require running all tests that exercise anything in the webapp!
With more granular YesodDispatch, we gain the ability to only run tests on parts of the app that actually use the route structure in question.

---

This PR is a breaking change, and so will be yesod-core-0.2.0.0. When I tried this branch on our package, it required 0 modifications to upgrade - most of the breaking changes here are breaking "advanced power user" type functions, like the ability to customize the MkDispatchSettings to have highly custom route behaviors. We don't use any of those, and I have no idea how widespread their use is. I will admit to finding them very difficult to use and adapt, which is part of why I ended up scrapping their use in the ParseRoute generation (the class isn't even used).

For the most part, though, all prior behavior remains unchanged. The performance of routing may be slightly worse - YesodDispatchNested returns a Maybe Application (but with the Request pulled out - only the "send response" side is in the Maybe) to support fallthrough. And if you enable fallthrough, then the app will do more checking (proportional to the number of routes and nested routes) rather than early-404ing. But IMO for perf, we're better off coming up with a trie-based router than trying to optimize this one much more.

For an example of the testing facilities this provides, see parsonsmatt/hspec-yesod#7 - particularly, this file has an introduction to how the tests are separated and defined.

---

Fix for #1880

Before submitting your PR, check that you've:

• Bumped the version number
• Documented new APIs with Haddock markup
• Added @since declarations to the Haddocks for new, public APIs

After submitting your PR:

• Update the Changelog.md file with a link to your PR
• Check that CI passes (or if it fails, for reasons unrelated to your change, like CI timeouts)

[L0neGamer]

Hi Matt! @jezen asked me to test this out on our codebase, and from what I can tell there is some kind of error when we use mkYesodSubData.

Here's an example excerpt:

mkYesodSubData "(SubsiteClass subsite master) => SubsiteData subsite"
  $(makeRelativeToProject "config/routes.yesodroutes" >>= parseRoutesFile)

results in many errors of the format:

src/Subsite/Data.hs:158:1: error: [GHC-53634]
    Out of scope type variable ‘subsite’ in the RHS of a family instance.
      All such variables must be bound on the LHS.
    |
158 | mkYesodSubData "(SubsiteClass subsite master) => SubsiteData subsite"
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...

I can't immediately spot why in your PR.

[parsonsmatt]

Great, thank you for the report! I don't think we have a test case for that in the repository yet. I'll make one and see if I can get it fixed.

[parsonsmatt]

@L0neGamer I added a test case and fixed, mind trying again?

[L0neGamer]

Getting another compile time issue, this time with mkYesodSubDispatch.

Here are some more snippets:

-- routes.yesodroutes
!/#{AssociatedType subsite}/route RouteR':
  / RouteR GET POST DELETE

-- Data.hs
mkYesodSubData "(SubsiteClass subsite master) => Subsite subsite"
  $(makeRelativeToProject "routes.yesodroutes" >>= parseRoutesFile)

-- Subsite.hs
instance
  ( SubsiteClass subsite master
  , OtherSubsiteClass master
  ) => YesodSubDispatch (Subsite subsite) master where
  yesodSubDispatch = $(mkYesodSubDispatch resourcesSubsite)

results in the error:

src/Subsite.hs:127:23: error: [GHC-83865]
    • Expecting one more argument to ‘RouteR'’
      Expected a type, but ‘RouteR'’ has kind ‘k0 -> *’
    • In the first argument of ‘YesodDispatchNested’, namely
        ‘RouteR'’
      In the type ‘YesodDispatchNested RouteR'’
      In the argument of reifyInstances: Yesod.Core.Class.Dispatch.YesodDispatchNested Data.RouteR'
    |
127 |   yesodSubDispatch = $(mkYesodSubDispatch resourcesSubsite)

It seems that you're not passing the subsite variable when doing the dispatch?

[parsonsmatt]

Great! Thanks. The more of a reproducer you can give me the faster I can fix it up 😄

[L0neGamer]

I'll have a look shortly if this is insufficient.

[L0neGamer]

Please observe PR here for failing test example: #1910

[L0neGamer]

I believe Claude's suggested test does not demonstrate the error while my commit does; feel free to cherry pick and rejig mine as you see fit.

[parsonsmatt]

OK, I think we have incorporated your test case- mind giving it another go? Thanks for your patience 🙏🏻

[L0neGamer]

getting this incredibly unhelpful error when making an example app:

example/ExampleApp.hs:47:1: error: [GHC-22385]
    Not in scope: record field ‘pathInfo’
   |
47 | mkYesodDispatchOpts siteOpts "App" routes
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

the splices look like this

I intuit that there's something wrong with how the TH is building the the splice somehow. I can't do any more investigation right now (way past when I should be up), but might be able to help or explore in ~12 hours. Let me know if that's something you want me to try or if you're gonna get up to it.

EDIT: There are many cases of pathInfo being used as a record field.

[L0neGamer]

Good news and bad news.

Good news is I can get the project to a compiling state from this.

The bad news is that adding the type parameter is a breaking change. I encountered this when working on #1866; it seems that #1887 would want to permanently turn that option on if it needs this.

[parsonsmatt]

Can you clarify what you mean? ie that each use of mkSomeTh (ro :: RouteOpts) needs to enable the parameterized subsite flag? If you can show an example of some code that is breaking then I'd be happy to try and make the backwards compatibility work out for y'all.

[L0neGamer]

So, the changes you make add the type arguments to the type of the subroutes, but that is a breaking change if any of the subroutes are used in a type signature.

For example if we take the following example from before:

-- routes.yesodroutes
!/#{AssociatedType subsite}/route RouteR':
  / RouteR GET POST DELETE

Then we will get the following types and constructors:
Route (SomethingThatUses subsite), with constructor RouteR' (AssociatedType subsite) RouteR'
RouteR', with constructor RouteR
But after your changes (and my flag elsewhere), the result will be
Route (SomethingThatUses subsite), with constructor RouteR' (AssociatedType subsite) (RouteR' subsite)
RouteR' subsite, with constructor RouteR

I hope you can see the change; the type RouteR' goes from having no type arguments to having one.