Strix Halo Dual — Optimized vLLM Inference (MiniMax-M2, AWQ INT4, TP=2 over USB4)

A complete, reproducible recipe for running a large MoE LLM across two AMD Strix Halo (Ryzen AI MAX+ 395 / Radeon 8060S iGPU, gfx1151 / RDNA3.5) mini-PCs using vLLM with tensor parallelism (TP=2) over a USB4 / Thunderbolt interconnect.

This setup pairs a custom-tuned fused-MoE Triton kernel config, a device-name shim that makes the tuned config actually load on this silicon, USB4 low-latency tuning, and a serialized (single-in-flight) serving policy. Every non-obvious flag is explained so the stack can be rebuilt from scratch.

Scope / honesty note: These are single-node-class consumer APUs bonded over USB4, not a datacenter fabric. The wins here come from (a) tuning the MoE GEMM kernel to gfx1151, (b) cutting interconnect latency ~8x, and (c) matching the serving policy to a latency-sensitive single-user workload. Numbers below are single-stream unless noted.

Repo layout

Path	What it is
moe/sitecustomize.py	The device-name shim (mandatory — §6). Put its dir on PYTHONPATH.
moe/E=256,N=768,device_name=Radeon_8060S_Graphics,dtype=int4_w4a16.json	The tuned MoE kernel config (§7).
launch/vllm-inner.sh	The TP=2 launcher (§8). Fill in placeholders, front with llama-swap.
sysctl/99-usb4net-lowlatency.conf	USB4 low-latency sysctls (§5). → /etc/sysctl.d/
systemd/usb4net-lowlatency.service	Keeps Thunderbolt awake + CPU EPP=performance (§5). → /etc/systemd/system/

Every file is generic — fill in the <PLACEHOLDER> tokens (see the table in §2) for your own environment. No real IPs, hostnames, or usernames are committed.

---

1. Model

• Model: MiniMax-M2 (mixture-of-experts).
• Quant used here (first-party): MiniMax-M2.7-AWQ-G32-STRIX-2H — AWQ 4-bit weights, group size 32, packaged for Strix Halo. (This quant was produced by the author of this repo.)
• Effective dtype at the MoE GEMM: int4_w4a16 (4-bit weights, fp16 activations).
• MoE shape that matters for tuning: E=256 experts, N=768 intermediate size. This is why the tuned kernel file is named E=256,N=768,device_name=Radeon_8060S_Graphics,dtype=int4_w4a16.json.

If you use a different model/quant, the MoE E/N/dtype change and you must re-tune (see §7); the tuned JSON in this repo is specific to the shape above.

---

2. Hardware & software topology

            USB4 / Thunderbolt (thunderbolt-net, IP over TB)
   ┌──────────────────────┐   ~8 us latency    ┌──────────────────────┐
   │   HEAD NODE (rank 0) │◄──────────────────►│  WORKER NODE (rank 1)│
   │  Strix Halo / gfx1151│                    │  Strix Halo / gfx1151│
   │  Ray head + vLLM API │                    │  Ray worker          │
   └──────────────────────┘                    └──────────────────────┘
        1 iGPU each  ──────────  TP=2 (2 GPUs total)  ──────────  1 iGPU each

Component	Value
GPUs	2 × AMD Radeon 8060S (gfx1151, RDNA3.5), one per node
APU	AMD Ryzen AI MAX+ 395 (Strix Halo)
Interconnect	USB4 / Thunderbolt, IP-over-Thunderbolt (thunderbolt0)
OS	Fedora 43 (traditional/dnf, not immutable)
Container	toolbox/podman container running the ROCm+vLLM userspace
Runtime	vLLM ROCm build for gfx1151 (kyuz0/vllm-therock-gfx1151, v0.19.2rc1 line)
Parallelism	Ray-backed tensor parallel, --tensor-parallel-size 2
Front door	llama-swap (on-demand model load/unload) → spawns the vLLM process

Placeholders used throughout (fill these in for your environment)

Placeholder	Meaning
<HEAD_IP> / <WORKER_IP>	Thunderbolt IPs of the two nodes (a private /24 you assign)
<HEAD_USER> / <WORKER_USER>	Login users on each node
<HEAD_MODEL_DIR> / <WORKER_MODEL_DIR>	Absolute path to the downloaded model on each node
<PROJECT_DIR>	A persistent dir on each node for the shim + tuned configs
<TB_IFACE>	Thunderbolt netdev name (usually thunderbolt0)
<TOOLBOX>	Name of your ROCm/vLLM toolbox container
<PORT>	Port vLLM listens on locally

---

3. Why this is non-trivial on gfx1151 (the core problem)

Two independent issues make a naive vLLM launch leave large performance on the table:

1. The tuned MoE config never loads. This vLLM build mocks amdsmi with a MagicMock, so RocmPlatform.get_device_name() returns a mock instead of a string. vLLM builds the tuned-config filename from that device name, so it computes a garbage, per-process-random filename and silently falls back to the generic default MoE config. The entire VLLM_TUNED_CONFIG_FOLDER mechanism is dead on arrival until the device name is forced to a real string. → Fixed by the shim (§6, moe/sitecustomize.py).

2. The default MoE kernel schedule is not tuned for gfx1151. vLLM ships a generic fused-MoE tiling. On RDNA3.5 the right block sizes / warps differ. → Fixed by the tuned config (§7), which you generate with vLLM's benchmark_moe.py --tune.

Plus the interconnect: stock IP-over-Thunderbolt sits at ~65 µs RTT, which taxes every TP=2 collective. → Fixed by USB4 low-latency tuning (§5).

---

4. Results (single-stream, concurrency = 1)

Clean A/B on identical hardware, generic default MoE config vs the tuned+shimmed config, both freshly served, 3 measured passes per tier, uncapped power. PP = prefill throughput (TTFT-derived tok/s). TG = decode throughput (per-sequence, TPOT-derived tok/s).

Prefill (PP)

Context (tokens)	Default	Optimized	Δ
1,024	498.8	499.1	+0.1%
2,048	596.0	541.7	−9.1%
20,480	486.1	507.8	+4.5%
40,960	292.0	303.9	+4.1%
61,440	209.5	215.4	+2.8%

Decode (TG)

Context (tokens)	Default	Optimized	Δ
1,024	13.39	14.84	+10.8%
2,048	13.24	14.57	+10.1%
20,480	11.29	12.19	+7.9%
40,960	9.83	10.48	+6.6%
61,440	8.76	9.28	+5.9%

Reading it: consistent win on both prefill and decode at long context, with one honest cost — 2k prefill −9.1%. Under 2-way concurrency the same optimization showed much larger decode gains because concurrency exposes more MoE-GEMM headroom for the tuned kernel; single-stream (above) is the conservative floor.

4b. Concurrency = 2 (throughput-oriented)

Same optimization, benchmarked at --max-num-seqs 2 (two in-flight sequences). TG is reported per-sequence here, so absolute values are lower than the c=1 table above (the two sequences share the GPUs), but the relative gain from the tuned kernel is much larger.

Comparison basis note: these c=2 numbers were taken against an earlier recorded baseline. Its 1,024-token PP value was an anomalous capture (~2× too high), so the 1,024 PP −52% is a measurement artifact, not a real regression — a clean re-run puts 1k PP at ~flat (see the c=1 table and Appendix B). Every other row is a like-for-like delta.

Prefill (PP)

Context (tokens)	Baseline	Optimized	Δ
1,024	1,043.0	500.7	−52.0% ⚠️ artifact
2,048	596.1	538.3	−9.7%
20,480	452.3	521.7	+15.3%
40,960	280.2	312.2	+11.4%
61,440	205.9	221.3	+7.5%

Decode (TG, per-sequence)

Context (tokens)	Baseline	Optimized	Δ
1,024	6.24	11.49	+84.2%
2,048	6.12	10.82	+76.8%
20,480	2.92	4.19	+43.4%
40,960	1.79	2.13	+19.4%
61,440	0.94	1.05	+11.4%

Reading it: at concurrency 2 the tuned MoE kernel delivers large decode gains (+11% to +84%) and recovers long-context prefill (+7.5% to +15.3%), because two concurrent sequences push the fused-MoE GEMM into a regime where the tuned tiling matters most. Choose c=2 for aggregate throughput; choose c=1 (§10) for lowest per-request latency.

---

5. USB4 / Thunderbolt low-latency tuning (~65 µs → ~8 µs)

IP-over-Thunderbolt is the TP=2 fabric. Stock latency (~65 µs) throttles every collective. The following no-reboot runtime tuning drops it to ~8 µs and is installed on both nodes. (Measured with qperf.)

Direction	TCP lat before	TCP lat after
head → worker	65.3 µs	8.2 µs
worker → head	65.3 µs	8.4 µs

5.1 sysctls → sysctl/99-usb4net-lowlatency.conf

Copy to /etc/sysctl.d/99-usb4net-lowlatency.conf on both nodes, then sudo sysctl --system.

net.core.busy_read = 100
net.core.busy_poll = 100
net.ipv4.tcp_low_latency = 1
net.ipv4.tcp_fastopen = 3

• busy_read / busy_poll are the important knobs (the NIC stack briefly spins waiting for packets instead of sleeping → lower latency, higher CPU/power).
• tcp_low_latency is a legacy no-op on modern kernels; included to mirror the tested recipe.
• tcp_fastopen=3 is part of the measured profile; unlikely to matter for established flows.

5.2 systemd unit → systemd/usb4net-lowlatency.service

Keeps Thunderbolt devices awake (no runtime power-down) and pins CPU energy-performance preference to performance. Copy to /etc/systemd/system/, then:

sudo systemctl enable --now usb4net-lowlatency.service

5.3 Verify

sysctl net.core.busy_read net.core.busy_poll net.ipv4.tcp_low_latency net.ipv4.tcp_fastopen
systemctl is-enabled usb4net-lowlatency.service   # -> enabled
systemctl is-active  usb4net-lowlatency.service   # -> active
cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference  # -> performance

# qperf spot check (server on worker, client on head):
#   worker:  qperf -lp 19966
#   head:    qperf <WORKER_IP> -lp 19966 -ip 19967 -t 5 tcp_lat udp_lat tcp_bw

Caveats: sysctls are global (affect all networking); busy-polling and EPP=performance raise idle power/heat. Reboot-only knobs (processor.max_cstate=2, pcie_aspm=off) were not needed to reach ~8 µs.

---

6. The MoE device-name shim (mandatory) → moe/sitecustomize.py

Without this, the tuned config in §7 will not load and you silently get default speeds. Install moe/sitecustomize.py as sitecustomize.py in <PROJECT_DIR>/moe-shim/ on both nodes and put that dir on PYTHONPATH for both the head vLLM process and the Ray worker.

Python auto-imports sitecustomize at interpreter startup from any sys.path entry. The shim installs a one-shot import hook that, the moment vllm.platforms.rocm finishes loading, overrides get_device_name() to return torch.cuda.get_device_name(0) (which resolves correctly to "Radeon 8060S Graphics" on this driver). vLLM then computes the stable, correct tuned-config filename and loads it.

Confirm it worked — after launch the serve log must show, on both ranks:

[moe-shim] patched RocmPlatform.get_device_name -> 'Radeon 8060S Graphics'
... Using configuration from .../E=256,N=768,device_name=Radeon_8060S_Graphics,dtype=int4_w4a16.json for MoE layer.

A few Using default MoE config lines for the model's separate BF16 (dtype=None) expert path are expected and fine; the int4_w4a16 line above is the one that must appear.

---

7. The tuned MoE config → moe/E=256,N=768,...int4_w4a16.json

Place the JSON at <PROJECT_DIR>/moe-configs/E=256,N=768,device_name=Radeon_8060S_Graphics,dtype=int4_w4a16.json on both nodes and point VLLM_TUNED_CONFIG_FOLDER at that dir.

The keys are per-batch-size (M) kernel schedules. Small-M keys (1–32) optimize decode; the large-M key (4096) optimizes long-context prefill.

triton_version must match the Triton in your runtime (here 3.7.0). The lookup is keyed on device name (from the shim) + dtype, and the M key nearest the current batch.

7.1 Regenerating for another model/shape

Use vLLM's kernel autotuner (from benchmarks/kernels/benchmark_moe.py, matching your vLLM tag). Run it inside the toolbox with the shim on PYTHONPATH so the output filename is correct:

PYTHONPATH=<PROJECT_DIR>/moe-shim \
python3 benchmark_moe.py \
  --model <HEAD_MODEL_DIR> \
  --dtype int4_w4a16 --tp-size 2 --trust-remote-code \
  --tune --batch-size 4096 \
  --save-dir <OUT_DIR>

gfx1151 pruning (big speedup of the tune itself): the full grid is ~7,000+ configs and takes many hours. On RDNA3.5 you can safely restrict the int4 search space in get_rocm_tuning_space() to BLOCK_M/N ∈ {64,128,256}, BLOCK_K ∈ {64,128,256}, num_warps ∈ {4,8} (num_stages is already [2] on ROCm — RDNA lacks the deep async pipelining that makes stages>2 useful, and BLOCK_K=32 / small blocks / low warps are architecturally weak). This cut ~7,280 → ~880 configs (~6 h vs ~60 h) with the same winner.

You can tune each M-key in a separate run and merge the JSONs (last-writer wins for triton_version, so write the run matching your runtime's Triton last).

---

8. The launch script → launch/vllm-inner.sh

launch/vllm-inner.sh is the vLLM inner launcher spawned by the front-end (llama-swap). It starts a Ray head on the head node, joins a Ray worker on the second node over <TB_IFACE>, then execs vllm serve with TP=2. Fill in every <PLACEHOLDER>, save on the head node; the worker is driven over SSH.

It runs inside the ROCm/vLLM toolbox container. Front it with llama-swap (on-demand load/unload) or run it directly with a <PORT> argument.

---

9. Flag & env reference (why each matters)

Distributed / interconnect

Flag / env	Purpose
--tensor-parallel-size 2	Split the model across the two iGPUs.
--distributed-executor-backend ray	Use Ray to span two physical hosts (multiprocessing can't).
GLOO_SOCKET_IFNAME / NCCL_SOCKET_IFNAME = <TB_IFACE>	Force all collectives onto the USB4 link, not the LAN.
NCCL_IB_DISABLE=1	No InfiniBand here; disable to avoid probing.
NCCL_NET_GDR_LEVEL=0	No GPUDirect RDMA on this fabric.
VLLM_HOST_IP (per node)	Pin each rank's advertised IP to its Thunderbolt address.
RAY_CGRAPH_get_timeout=1800	Generous timeout for cross-node compiled-graph ops.

gfx1151 / ROCm stability & perf

Env	Purpose
--enforce-eager	Disable CUDA/HIP graph capture (unstable path on this build).
VLLM_ROCM_USE_AITER=0	Disable AITER kernels (not a win / unstable here).
VLLM_USE_DEEP_GEMM=0, VLLM_USE_FLASHINFER_MOE_FP16=0, VLLM_USE_FLASHINFER_SAMPLER=0	Turn off code paths not supported/beneficial on gfx1151.
HSA_NO_SCRATCH_RECLAIM=1	Avoid scratch reclaim churn (long-context stability/perf).
AMDGCN_USE_BUFFER_OPS=0	Safer codegen on RDNA3.5.
PYTORCH_HIP_ALLOC_CONF='expandable_segments:True'	Reduce allocator fragmentation at long context.
HIP_FORCE_DEV_KERNARG=1, SAFETENSORS_FAST_GPU=1	Minor launch/load speedups.
TORCHDYNAMO_DISABLE=1, OMP_NUM_THREADS=1, TOKENIZERS_PARALLELISM=false	Remove CPU-side variance/oversubscription.

The HSA_NO_SCRATCH_RECLAIM + AMDGCN_USE_BUFFER_OPS + PYTORCH_HIP_ALLOC_CONF + HIP_FORCE_DEV_KERNARG + SAFETENSORS_FAST_GPU block was A/B-validated as a long-context win (PP up to +12% at 60k, TG up to +7% at 60k) with a small short-context decode regression — keep it for long-context targets.

MoE tuning

Env	Purpose
VLLM_TUNED_CONFIG_FOLDER (per node)	Where vLLM looks up the tuned MoE JSON.
PYTHONPATH=<shim dir> (per node)	Loads the device-name shim so the lookup filename is correct.

Capacity / batching

Flag	Value	Purpose
--gpu-memory-utilization	0.92	Leave headroom for KV cache at 196k context.
--max-model-len	196608	Max context.
--max-num-batched-tokens	20480	Prefill chunk/scheduler budget. A sweep (8k/16k/32k/64k) found 20k the best all-round default; 64k can't fit KV at 196k length. Not a concurrency knob.
--max-num-seqs	1	Serialize requests (see §10).

Model-specific

--tool-call-parser minimax_m2, --reasoning-parser minimax_m2, --enable-auto-tool-choice, and --override-generation-config are MiniMax-M2 specifics (reasoning + tool-calling). Adjust for other models.

---

10. Serialized serving (--max-num-seqs 1)

This deployment targets a single interactive user where two in-flight requests make both feel slow. --max-num-seqs 1 caps the running batch to one sequence: vLLM admits one request, runs it to completion (prefill + all decode), and holds any concurrent requests in its FIFO waiting queue, then admits the next. No requests are dropped — they're serialized, so each runs at full single-stream speed.

• --max-num-batched-tokens does not re-introduce concurrency; it only sizes prefill chunks within a step. --max-num-seqs is the sole concurrency gate.

10b. Optional: concurrency = 2 config (throughput-oriented)

If you'd rather have aggregate throughput than lowest per-request latency, run two in-flight sequences. This is the config that produced the §4b numbers (large per-sequence decode gains, +11% to +84%). The only change from the launch/vllm-inner.sh exec vllm serve block is:

--max-num-seqs 1     ->    --max-num-seqs 2

Trade-offs vs --max-num-seqs 1:

	c=1 (default, §10)	c=2 (optional, §10b)
Best for	Lowest per-request latency; single interactive user	Aggregate throughput; 2 concurrent users/agents
A 2nd request while 1 is running	Queued (FIFO), runs full-speed after	Runs concurrently, both share the GPUs
Per-request speed	Full single-stream	Lower (two sequences share compute)
Results table	§4	§4b

• Everything else (shim, tuned config, USB4 tuning, env block, KV memory) is identical — no re-tune needed; the tuned MoE JSON serves both modes.
• Higher --max-num-seqs (3+) is possible but re-check KV-cache headroom at 196k context; the memory budget was sized for a small running set.

---

11. End-to-end setup checklist (for an automation agent)

On both nodes unless stated:

1. Base: Fedora (traditional), a ROCm+vLLM toolbox container for gfx1151 (kyuz0/vllm-therock-gfx1151 line). Confirm torch.cuda.get_device_name(0) returns Radeon 8060S Graphics inside the container.
2. USB4 link up: assign a private /24 to <TB_IFACE> on each node (<HEAD_IP>, <WORKER_IP>); confirm bidirectional ping and ethtool <TB_IFACE> shows the link up.
3. USB4 low-latency tuning (§5): install sysctl/99-usb4net-lowlatency.conf
   • systemd/usb4net-lowlatency.service on both nodes; verify ~8 µs with qperf.
4. Passwordless SSH head → worker for the login users.
5. Model: download MiniMax-M2.7-AWQ-G32-STRIX-2H to <HEAD_MODEL_DIR> and <WORKER_MODEL_DIR>.
6. Shim (§6): copy moe/sitecustomize.py to <PROJECT_DIR>/moe-shim/sitecustomize.py on both nodes.
7. Tuned config (§7): copy the moe/E=256,N=768,...int4_w4a16.json to <PROJECT_DIR>/moe-configs/ on both nodes (or regenerate for your model). Ensure triton_version matches your runtime.
8. Launcher (§8): install launch/vllm-inner.sh on the head node with placeholders filled; front it with llama-swap (or run it directly with a <PORT>).
9. Launch & verify (§12).

---

12. Verification

# Health
curl -fsS http://127.0.0.1:<PORT>/v1/models

# In the serve log, BOTH ranks must show the shim patch:
grep "\[moe-shim\] patched" serve.log      # expect 2+ hits (Radeon 8060S Graphics)

# The int4 tuned config must load (NOT default) for the quantized experts:
grep "Using configuration from .*int4_w4a16.json" serve.log   # expect >=1
# (a few "Using default MoE config" lines for the BF16 dtype=None path are OK)

# Ray sees 2 GPUs:
/opt/venv/bin/ray status | grep GPU        # 0.0/2.0 GPU when idle

Coherence smoke test:

curl -fsS http://127.0.0.1:<PORT>/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"minimax-m2","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":2000,"temperature":0.1}'

---

Appendix A — Autotuner power-safety (tuning phase only, NOT runtime)

Running benchmark_moe.py --tune hammers the iGPU with back-to-back GEMMs and can drive sub-second package-power transients well above the sustained TDP (observed ~220 W on a ~120 W-class board), tripping an instant VRM/EDC hard-lock — distinct from a slow thermal soak. This only affects the autotuning phase, not normal inference (which never hits that burst).

If you re-tune and hit hard-locks, cap the fast/FPPT limit for the duration of the tune using ryzenadj (build FlyGoat/RyzenAdj + an SMU kernel module such as a maintained ryzen_smu fork; on a locked-down kernel this requires Secure Boot disabled). Example transient cap:

sudo ryzenadj --fast-limit=90000 --stapm-limit=80000 --slow-limit=70000 \
              --apu-slow-limit=60000 --tctl-temp=85

Do NOT make this persistent — it can reduce normal inference performance. It is a tuning-time safety harness only; let it reset on reboot and run inference uncapped.

---

Appendix B — Notes & caveats

• Kernel pinning: if your ROCm/gfx1151 stack is validated against a specific kernel, keep it pinned; build any out-of-tree modules against the running kernel's headers.
• /tmp is volatile: keep the shim, tuned configs, and model dirs on persistent paths; the launcher only uses /tmp for a disposable model-alias symlink.
• A measured 1,024-token PP "regression" vs an older baseline was a baseline capture artifact (an anomalously high recorded value); a clean re-run shows ~flat at 1k. Always A/B default-vs-tuned on the same freshly-served process, warmups excluded.
• The tuned JSON is shape-specific (E=256,N=768,int4_w4a16). Re-tune for any other model/quant/dtype.

---

License

MIT — see LICENSE. The tuned config and shim are provided as-is; re-tune for your own model/hardware.