When to return a "simpler" type than a function was invoked with?

[gwhitney]

The discussion in #3374 outlining how mathjs will uniformize when mathjs returns different types with config.predictable = false and how it will behave in out-of-domain cases with config.predictable = true has explicitly opened the door to functions returning a "simpler" type when it happens to be exact, e.g. floor of a BigNumber could be a bigint, if that's useful. It turns out that a couple of functions that typically return a complex number already return a plain number when the imaginary part of the result happens to be zero, and certainly those examples should be regularized and that principle, assuming it's deemed valuable, should be extended to other functions. This discussion is dedicated to determining other cases, if any, when such "type simplification" could and should occur.

[gwhitney]

As a first attempt at principles, I think the discussions in #3374 support the following:

1. "Type simplification" should only occur when config.predictable = false, and not at all when config.predictable = true.
2. Type simplification should not occur from approximate types (like number/BigNumber) to exact types (like bigint/Fraction) or vice-versa: only within each category.
3. Type simplification may reasonably occur from Fraction to bigint when an outcome turns out to be an integer, and from Complex to the corresponding real type (in mathjs14, that is only number) when an outcome turns out to be real.
4. It's unclear whether the simplifications in point 3 should be mandatory (i.e., when config.predictable = false, all operations where the outcome happens to be as stated will automatically perform the type conversion), or if not, what will decide whether a specific operation does type simplification.
5. It's unclear whether any other type simplifications could/should be made. I will say that under the very reasonable regime of point 2, no other type simplifications come to mind.

[gwhitney]

On point 4, I definitely think that some type simplifications should be made: floor on a Fraction should be a bigint, for example, when config.predictable is false. That one is easy because the result will always be an integer. And when taking the eigenvalues of a complex matrix, it's very useful to know which, if any, of the eigenvalues are real, so it would be nice to get those back as number entities rather than complex.

However, at the other end of the spectrum in which every operation does simplification checks, I worry about performance (the computational cost of all of the checks on every intermediate result) and ability to chain operations (knowing what version of an operation to execute on the result of previous computation). The latter issue is particularly acute in prototypes like nanomath, which preselect the implementations of dependencies based on the type that they will operate based on. If complex addition could at any point return a number rather than a Complex(number), then the overhead of re-checking types as that result is used in later steps, which exists in mathjs but where pocomath obtained a big performance advantage by avoiding, would be re-introduced.

For example, for any Complex type for z, the expression z + conj(z) will always produce a real result. If there's a desire for that result to be captured in a number type rather than Complex, then even the most basic operation of + would have to check for 0 imaginary part and convert the result. A complicated function that does a lot of additions could suffer a fair amount of overhead for this.

So what's clear to me from these prototypes is that at least there needs to be a way to obtain implementations of basic operations that will not do type simplification, even when a global predictable setting is false, so that they can be used in the guts of other methods. That's not yet a fully fleshed-out system for determining how and when to do type simplification, but it seems to me to be one minimum requirement for a successful and performant system.