Directory structure of the source code?

[BachiLi]

The number of files in src is growing steadily -- I wonder if it is a good idea to group the files in directories. The easiest grouping is to put the simplifying rules into one folder, and to put the codegen files into another folder. Maybe the lowering passes should be in one folder as well.

[alexreinking]

I have long been in favor of this. I think it would be useful to enforce some degree of isolation between components and public/private parts of our API. I am in the process of moving the autoschedulers to src/plugins/autoschedulers in #5228 which is one step in this direction.

For instance, we have this ASLog thing that duplicates our other Error handling for the autoschedulers. If that were factored out as an internal logging library, we wouldn't have to duplicate it or work around exposing parts of the interface to our end users. Everything could just link to it.

Another benefit would be to isolate LLVM to only those parts that require it. That will make it easier to produce builds of Halide that do not use LLVM, which is something that has come up in discussions, and will also make it easier to maintain the file lists for Halide.h (right now there's a comment saying, essentially, "please don't put anything here that includes LLVM")

[jrk]

I have also long been in favor of this. To quickly surface past points of discussion:

• I think this has often been understood as somewhat of a tradeoff between reducing overhead for the core full-time maintainers, who like a simple list of files they know well, and making the code accessible to newer developers (at this point, probably anyone outside a core set of <= 5 people)
• There are actually at least 2-3, partially entangled issues at play:
  • How are the files organized into directories?
  • How is the code factored into modules, with more restricted interfaces between modules?
  • Perhaps also, how is the code built: monolithically or as modules?

[abadams]

This probably only hasn't happened because the core devs know all the filenames by heart, so never actually pay a cost for having tons of files in one folder, but I support it to better communicate the existing structure.

The conceptual modules are back-ends, front-end (Func.h, Generator.h, etc), lowering passes (AllocationBoundsInference.h), autoschedulers, the IR and analysis on it (e.g. simplify, is_monotonic, expr_uses_vars), the runtime, and then probably misc utils like filesystem helpers, the threadpool, etc. The dependencies are something like:

backends are the only things that may depend on LLVM
front-end only depends on lowering, backends
autoschedulers depend on front-end
lowering depends on backends (some lowering passes need to compile and embed code on the way (hexagon))
everything depends on IR
the runtime depends on nothing

tests and apps may depend on whatever they like for whatever weird thing they're trying to do, but in the normal course of things they only need the front-end.

We should have src/lower src/codegen src/frontend, src/ir, src/runtime, src/util

If util is really just Util.h Util.cpp, it doesn't need to be in a subdirectory.

You could probably break it up more than that (e.g. separating llvm-using backends from source-based backends might be useful, distinguishing between the simplifier and the rest of the IR could have value), or less than that (it might be that there's no clean separation between analysis passes on the IR and lowering passes on it) but that would be a good coarse cut that I think the core devs would support.

At first we wouldn't mechanically prevent any cross-directory dependencies, but they would be considered a code-smell if they violate the relationships above. Let's tackle the placement of files in directories as its own task at first. Later we could enforce it, but we'd have to distinguish between headers internal to each module and headers that the module exports.

[abadams]

Also I don't think there's actually much of a trade-off here. CodeGen_OpenCL.cpp is no simpler than CodeGen/OpenCL.cpp

[alexreinking]

CodeGen_OpenCL.cpp is no simpler than CodeGen/OpenCL.cpp

Right -- many tools treat directories as special entities. They don't necessarily grok the underscore convention without ad-hoc globbing.

[steven-johnson]

tests and apps may depend on whatever they like for whatever weird thing they're trying to do, but in the normal course of things they only need the front-end.

(threadjack: agreed for tests, but disagreed (strongly) for apps. I realize we are using apps/ to develop new things (autoschedulers, simplifiers, etc) that need more-internal access, but IMHO that's a bad precedent; apps/ should be examples of Halide usage using the public API. Maybe we should consider adding something like experiments/ as a place to put work of that sort?)

[abadams]

To clarify: I think our example apps should be strict and only use the front-end, but I want third parties to be able to easily do any sorts of wild metaprogramming and misuse of the compiler they feel like, so we shouldn't take any mechanical measures to enforce things in a way that would also apply to third parties.

[steven-johnson]

In general I support this effort. As a practical matter, let me suggest that we wait until we (finally) do another binary release and start proper semver usage; this sort of rearrangement makes a lot more sense to do once a version-barrier is enacted, IMHO.

[abadams]

I did a first pass over the files and my proposed breakdown seems to work. But yeah, happy to wait.

[BachiLi]

What's the status of this?

[alexreinking]

Let's wait to release v11.0 before we start restructuring code

[abadams]

We all agree roughly how it should be done, but nobody is actively working on it.