Disable task sync in CUDA

[luraess]

This PR implements the changes needed to support opting out of syncing on task switching for the CUDA backend JuliaGPU/CUDA.jl#2662.

It thus avoids serialisation of async operations and brings back async execution

I am unsure whether this is the best way of addressing things tho...

Requires CUDA#vc/unsafe_stream_switching for testing

[luraess]

According to our latest discussion, we may rework the communication / computation overlap in Chmy. By serialising all the compute (Outer -> BCs -> Inner) we could then spawn a new task only for doing IO, i.e., MPI comm preceded by copying from the array view into buffers and back into the array upon termination.

As I understand, this would only require opting out from implicit sync using e.g. unsafe_disable_task_sync! on the buffer arrays without needing further engineering.

It would still be nice to update CUDA.jl's proposed unsafe_disable_task_sync!(arr) to return the array instead of that status as to e.g. use the following construct:

a = CUDA.rand(2, 1) |> CUDA.unsafe_disable_task_sync!

[luraess]

@utkinis what if we use the above strategy to circumvent the implicit sync issue with CUDA.jl? This would actually make the launch_with_bc function "unsafe" wrt CUDA.jl, but would confine the unsafeness ti this function and not interfere with other parts of the code. Testing on ALPS, the approach seems to work and give good perf and ... overlap

Happy to get your feedback and if positive to going this way, potential hints on improving my prototype attempt

[utkinis]

Thanks @luraess for digging further into this. Yes, checking the arguments at runtime could be a good way to solve the issue. One needs to recursively check all Julia composite types that could be passed to kernels, namely, structs and tuples (Not sure about arrays and refs, I guess only the bits types could be passed to GPU kernels, but might be wrong).

Also, the implicit sync happens at the array level, so the recursion must descend to the level of arrays and not just Fields, as the user might just pass a regular GPU array. I left the specific comments in the PR. Otherwise, I'm positive to make it work this way.

[luraess]

Needs to be removed before merge

[luraess]

And the ALPS check that things overlap as they should

[luraess]

EDITED

From my side this could be ready to go. Before merging, one needs to:

• revert testing from a specific branch
• bump minor (or patch?) version
• wait Support disabling implicit synchronization JuliaGPU/CUDA.jl#2662 gets merged and release tagged
• update CUDA compat bounds

[luraess]

Upon final rework of JuliaGPU/CUDA.jl#2662, the approach still works and successfully overlaps

[luraess]

This should be ready to go from my side.