QAT x SWA Ablation: SWA sabotages QAT (-3.64 mBPB, 3-seed validated)

[alexanderaperry-arch]

Leaderboard-relevant ablation: SWA and QAT are antagonistic

Systematic 2×2 factorial (QAT on/off × SWA on/off) on the PR #180 stack. 3-seed validated on 8xH100, all runs under 10 min and 16MB.

Result

Config	QAT	SWA	Mean BPB (3 seeds)	Delta vs Control
no_swa_qat	Yes	No	1.14018	-3.64 mBPB
control	No	Yes	1.14382	baseline
qat_snap70	Yes	Yes	1.14468	+0.86 mBPB
no_swa	No	No	1.14486	+1.04 mBPB

Why this matters

Every QAT submission in this competition (#117, #139, smeargate_ortho) also used SWA — and every one underperformed non-QAT entries. Our ablation shows why: SWA's checkpoint averaging dilutes the quantization-boundary alignment that QAT works to achieve. Combining them is worse than either alone.

The fix is simple: remove SWA when using QAT. This alone yields ~3.6 mBPB.

Actionable for competitors

• If you're using SWA + QAT together, drop SWA
• QAT alone is 3.5x more effective than SWA alone for quantization quality
• Training val_bpb is misleading for QAT — post-quantization BPB is the metric that matters
• QAT weights need ~10% magnitude pruning (vs 3%) to fit under 16MB — they compress worse

Open question

Top entries now use EMA instead of SWA. Nagel et al. (2022) proposes EMA to stabilize QAT, but our short-horizon results suggest averaging mechanisms in general may conflict with QAT under tight wallclock constraints. EMA × QAT interaction is untested.

Validation

• Seeds: 42, 1337, 2024 (std ±0.00044)
• Artifact: 15,787,003 bytes (158KB under limit)
• Hardware: 8xH100 SXM, 600s wallclock cap
• Based on PR Record: 10L Int5-MLP + BigramHash(10240) + SWA(0.4) + WD=0.04 (val_bpb=1.1428, mean 3 seeds) #180 (10L/512d/MLP3x, int5-MLP/int6-attn, zstd-22)