ROCm Multi-GPU troubleshooting help

[bhuv-z]

GPUs:

1. 7900xtx (gfx1100)
2. 9700 AI PRO (gfx1201)

I'm currently using ROCm on WSL2 with Ubuntu 22.04. ROCm detects both gpus, and i was able to validate this with pytorch as well

Issue: Llama-server runs fine on my 7900xtx which is my primary gpu, but fails to run on the 9700 AI Pro when the model is being loaded to vram with the following error:

pid:1704203 tid:0x7ac2cf3df6c0 [CreateContext] fail c000000d
test-backend-ops: ./sources/wsl/libhsakmt/src/wddm/queue.cpp:1078: wsl::thunk::SDMAQueue::SDMAQueue(wsl::thunk::WDDMDevice*, void*, uint64_t, uint32_t, bool): Assertion `ret' failed.
Aborted (core dumped)

Example Scenario:

moyashi@DESKTOP-JIIHNLK:~/llama.cpp ((HEAD detached at b8248))$ ./build/bin/test-backend-ops -o "GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=4,n_seqs=2,v_repeat=1,permuted=1,kda=1)"
ggml_cuda_init: found 2 ROCm devices (Total VRAM: 57005 MiB):
  Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24517 MiB (21976 MiB free)
  Device 1: AMD Radeon AI PRO R9700, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 32488 MiB (32439 MiB free)
Testing 3 devices

Backend 1/3: ROCm0
  Device description: AMD Radeon RX 7900 XTX
  Device memory: 24517 MB (21976 MB free)

  GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=4,n_seqs=2,v_repeat=1,permuted=1,kda=1): OK
  1/1 tests passed
  Backend ROCm0: OK
Backend 2/3: ROCm1
  Device description: AMD Radeon AI PRO R9700
  Device memory: 32488 MB (32439 MB free)

pid:1704203 tid:0x7ac2cf3df6c0 [CreateContext] fail c000000d
test-backend-ops: ./sources/wsl/libhsakmt/src/wddm/queue.cpp:1078: wsl::thunk::SDMAQueue::SDMAQueue(wsl::thunk::WDDMDevice*, void*, uint64_t, uint32_t, bool): Assertion `ret' failed.
Aborted (core dumped)

Here's what i've tried so far but havent had much luck resolving the issue:

1. llama-server (and other llama.cpp tools) are able to run using single 7900xtx gpu, but fail on multi-gpu, and single 9700 AI Pro gpu.
2. Reinstalling rocm 7.2 which is supported on wsl2
3. Reinstalling Windows AMD GPU drivers
4. Setting HIP_VISIBLE_DEVICES=0,1
5. Setting HSA_OVERRIDE_GFX_VERSION=11.0.0
6. Running VRAM memtest on my 9700 AI Pro to make sure system can use the VRAM and it passed.

Any help in resolving this would be appreciated.

[bhuv-z]

Bump