[WIP]ggml-hexagon: improve leftover element calc at vec_dot_f16_f32

[chraac]

Performance

Device: 8Gen2

Baseline: ed7597771
Optimization: 2058f28b3

Operation	Params	Baseline (GFLOPS)	Optimization (GFLOPS)	Speedup
MUL_MAT (f16, f32)	k=128, n=1	3.68	5.74	1.56x
MUL_MAT (f16, f32)	k=14336, n=1	3.43	7.26	2.12x
MUL_MAT (f16, f32)	k=14336, n=2	3.46	7.29	2.11x
MUL_MAT (f16, f32)	k=14336, n=3	3.46	7.29	2.11x
MUL_MAT (f16, f32)	k=14336, n=4	3.46	7.34	2.12x
MUL_MAT (f16, f32)	k=14336, n=5	3.46	7.31	2.11x
MUL_MAT (f16, f32)	k=14336, n=8	3.48	7.35	2.11x

[chraac]

Based on my observations, using volatile here seems to have several drawbacks:

• Prevents inlining: With volatile, the binary retains a separate vec_dot_f16_f32 function instead of inlining it into matmul_f16_f32.
• Generates extra store instructions: noticed that the compiler generates extra vmem instructions to write the result registers to the stack, as seen in the highlight below. This will increase the mem bandwidth pressure, which impacts the processing speed.
  

[chraac]

Here we add a new field to track the available bytes per session.

[max-krasnyansky]

@chraac
Apologies for the delayed review. I looked at it earlier and the host side changes were breaking all kinds of things.
We don't need to enforce 2GB limit, the actual limit is 3.8GB so it's OK to spill over.

Re: f16/f32 matmuls. I've been playing with Flash Attention implementation and ended up overhauling how f16/f32 stuff is handled in many places. We should do as much as possible in f16 and only accumulate into f32. Leftover handling is quite simple with f16 using the mask predicate trick.
Take a look at #18611

[chraac]

Apologies for the delayed review. I looked at it earlier and the host side changes were breaking all kinds of things.
We don't need to enforce 2GB limit, the actual limit is 3.8GB so it's OK to spill over.

@max-krasnyansky
No worries. Regarding the limitation, I assume the limitation comes from the 32-bit address space of the Hexagon RTOS. Given newer versions support a 64-bit address space, will we be able to increase the memory allocation per session?

[max-krasnyansky]

Apologies for the delayed review. I looked at it earlier and the host side changes were breaking all kinds of things.
We don't need to enforce 2GB limit, the actual limit is 3.8GB so it's OK to spill over.

@max-krasnyansky No worries. Regarding the limitation, I assume the limitation comes from the 32-bit address space of the Hexagon RTOS. Given newer versions support a 64-bit address space, will we be able to increase the memory allocation per session?

Only the DMA engine has 64-bit address space, the rest of the HW is 32-bit address space.
If we were to use the DDR <-> DMA <-> VTCM for every single part of the processing then yes we can increase the memory allocation per session. Otherwise we need to keep it under 4GB.