[BUG] Kimi-K2.5 fails on 4 node cluster

[phil-brass]

Describe the bug

A clear and concise description of what the bug is. When running Kimi-K2.5 on a 4 node cluster it will crash after a few requests. Typically three nodes go to 0% CPU and one goes to 100%.

To Reproduce

Steps to reproduce the behavior:

1. Fire up a 4-node Kimi-K2.5 cluster
2. Build up some context
3. Watch it crash

Expected behavior

Two-node K2.5 clusters are super stable, so I expected a 4-node cluster to be stable and faster.

Actual behavior

The cluster fails, typically with one node at 100% CPU and the rest at 0%

Environment

• macOS Version: macOS 26.3 (25D125) Darwin 25.3.0
• EXO Version: 1.0.68
• Hardware:
  • Device 1: Mac Studio M3 Ultra 512GB RAM
  • Device 2: Mac Studio M3 Ultra 512GB RAM
  • Device 3: Mac Studio M3 Ultra 512GB RAM
  • Device 4: Mac Studio M3 Ultra 512GB RAM
• Interconnection:
  • Thunderbolt 5 fully cross-wired
  • 10GbE Ethernet between all devices

Additional context

N/A

[vskiwi]

We've been investigating the exact same issue on a 4× Mac Studio M3 Ultra cluster (TB5 fully cross-wired) and found two bugs in the rltakashige MLX fork that most TB5 JACCL users run:

1. Missing read-side memory barrier in fence.cpp — causes GPU fence deadlocks (the "one node at 100% CPU, rest at 0%" pattern you're describing)
2. Bounded fence wait loop in fence.metal (for(30) instead of while(1)) — causes silent data corruption when the GPU fence takes longer than 30 iterations

Our PR #1 fixes both. On our cluster it went from constant deadlocks to 350+ stable inference requests.

If you'd like to test on your cluster:

# 1. Reboot all nodes first (clears stale RDMA state)

# 2. Clone the fork and apply the PR
git clone -b address-rdma-gpu-locks https://github.com/rltakashige/mlx-jaccl-fix-small-recv.git mlx
cd mlx
git fetch origin pull/1/head:pr1 && git merge pr1 --no-edit

# 3. Build wheel and install on all nodes (into your exo venv)
pip wheel . --no-deps -w /tmp/wheels/
pip install /tmp/wheels/mlx-*.whl --force-reinstall --no-deps

# 4. Run exo as usual

Two-node clusters are more stable likely because there's only 1 RDMA pair vs 6 in a 4-node mesh.

Would be great to know if you see improved stability with this build.