ClownCar: Frugendorff compression baseline + canonical DeltaNet integration

[newjordan]

The Frugendorff Architecture — Origin and Signal Analysis

The Frugendorff (F-Wing) crawler is a weight-shared recurrent architecture designed around one observation: parameter golf rewards compression, and the most efficient way to compress a transformer is to share weights across depth while running them repeatedly.

Thesis: The Frugendorff (F-Wing) crawler explores a simple idea;

Increase effective depth per stored byte by reusing weights across loops, then reallocate saved parameters into width.

Structure

Input → [2 flat encoder layers] → [1 shared crawler block × 4 loops] → [2 flat decoder layers] → Output
         unique weights             shared weights, looped                unique weights

The crawler block is one set of weights executed 4 times. The flat layers are unique. Total params: ~13.4M at our standard config (dim=512, 4 flat + 1 crawler × 4 loops). For comparison: a standard 11-layer transformer at the same dim runs 27M params. We get 8 effective depth passes at 13.4M params vs 11 unique passes at 27M.

Flow Instructions (FX-Wing innovation): instead of static per-loop positional offsets, each loop projects the current hidden state x through a shared bottleneck to produce a loop-specific perturbation:

inst_k = loop_inst_up[loop](loop_inst_proj(x))  # recomputed from current x each loop
x_loop = x + inst_k

This makes each loop's instruction respond to what the previous loop produced — genuine iterative refinement rather than a fixed plan.

---

What We Actually Found: Crawler Signal Analysis

We ran a comprehensive signal analysis across 8 micro-crawler TSVs, 175 Frugendorff sweep configs, production cadence ablations, and pod logs. Seven statistical tests applied.

Finding 1: Width is the primary lever (~0.033 BPB)

Config	dim	BPB	Delta vs flat
3f+1cx2	608	2.157	−0.033
4f+1cx2	544	2.191	+0.001
6flat ctrl	496	2.190	baseline

The 0.033 BPB advantage tracks dim, not sharing. Fewer unique layers → fixed params spread across more channels per layer → wider dim → better quality. When dim is equalized, the advantage vanishes.

Finding 2: Recursion provides zero per-step benefit

C-steps (crawler fires) vs N-steps (no crawler) produce identical training loss at every phase. No momentum, no gradient interference difference, no variance difference. The second-through-fourth passes through the shared block don't "refine" in any measurable per-step sense.

Finding 3: More looping = early boost that decays

Config	s50 delta	s500 delta
3f+1cx2	−0.086	−0.033 (stable)
3f+1cx3	−0.092	−0.017 (decayed 81%)

Additional loops give early gains that fade over training.

Finding 4: ~0.01 BPB from implicit regularization

The 4f+1cx2 (no trigram, dim=544, 16.8M params) beats 6flat (trigram, dim=496, 17.9M params) by 0.009 BPB. With 6% fewer params and no trigram assist, this gap can't be entirely explained by width — approximately 0.01 BPB is from weight sharing as implicit regularization.

Finding 5: Post-processing is hostile to shared weights

SWA + quantization widen the gap with more recursion: quant gap 0.136 (4x2 cad1) vs 0.059 (4x2 cad4). More looping = worse quantization resilience. The crawler's int8 quantization mode (CRAWLER_QUANT_INT8=1) was specifically added to mitigate this.

---

The Compression Story

We ran a control experiment: a misconfigured ClownCar (missing USE_CRAWLER=1) ran a plain 27M-param GPT against our 13.4M crawler head-to-head.

Model	Params	int6+zstd size	BPB	Submittable
Plain GPT (control)	27M	16.6MB	1.1230	❌ over limit
A-Wing SOTA	~26M	~15.5MB	1.1129	✅
ClownCar (crawler)	13.4M	~9.1MB	1.1813 ± 0.0002	✅
ClownCar_II (+ canonical DeltaNet [#875])	~13.5M	~9.5MB	pending	✅

The 27M GPT cannot submit — 16.6MB exceeds the 16MB cap. Our crawler at 9.1MB has 6.9MB of headroom. At 1.1813 BPB vs the 27M GPT's 1.1230, we are within 0.058 BPB of a model twice our size that can't even enter.

Half the size. 94% of SOTA quality. Confirmed across 3 seeds.

---

ClownCar — Confirmed Results (3 Seeds)

experiments/ClownCar/ — FX_Wing_Delta stripped to its legal core. No ngram eval (ruled illegal). Sliding window only.

Seed	BPB (sliding window)	Size (int6+zstd)	Post-EMA BPB	Steps
1337	1.1812	9.05MB	1.1999	7,321
42	1.1815	9.15MB	1.1999	7,320
123	1.1813	8.98MB	1.2003	7,317
Mean	1.18133	9.06MB
Std dev	0.00015

Variance: 0.00015 BPB across 3 seeds. The number is 1.1813.

Config: 4 flat + 1 crawler × 4 loops, INST_DIM=32, CRAWLER_QUANT_INT8=1, DELTA_NET_HEADS=0, WARMDOWN_ITERS=2000.

---

Where We're Headed: Ludicrous Speed

The ClownCar_II kernel injection is the first step in a broader direction: replace every Python-level overhead loop in the module with native compiled kernels.

The overhead map

Component	Current	Bottleneck	Fix
DeltaNet (#875, arxiv 2406.06484)	Python loop over T	@torch.compiler.disable, 1000× slower	✅ chunk_delta_rule (FLA Triton)
GDN (Cambrian)	Python loop over 32 chunks × 64 tokens	19× slower (3.87s/step vs 0.2s)	chunk_delta_rule or vectorized form
Short convolutions	Pure PyTorch Conv1d	JIT-able but not fused	Fuse with projection in custom kernel
Crawler loop itself	Python for loop in range(4)	Unrollable but PyTorch graph overhead	CUDA graph capture across loops

The thesis

Every for t in range(T): in a neural network is a promise that someone will eventually replace it with a Triton kernel. The FLA library by Songlin Yang & Yu Zhang (#875) is the reference implementation of that promise for linear attention. chunk_delta_rule parallelizes the delta rule over sequence chunks exactly the way flash_attn parallelized softmax attention.

The next frontier: capture the entire [encoder → crawler×4 → decoder] computation graph as a single CUDA graph, eliminating Python dispatch overhead between loops entirely.

ClownCar_III

Reserved for the next injection experiment — CUDA graph capture across crawler loops and/or vectorized GDN chunk form. Target TBD based on ClownCar_II results.

---

Files

• experiments/ClownCar/run.sh — sliding window baseline, DELTA_NET_HEADS=0
• experiments/ClownCar/train_gpt.py — FX_Wing_Delta engine (3283 lines)
• experiments/ClownCar_II/run.sh — canonical DeltaNet (#875), DELTA_NET_HEADS=4
• experiments/ClownCar_II/train_gpt.py — adds CanonicalDeltaNet + chunk_delta_rule import

---

Other parts of this will probably go towards this PR thiniking - #875 (comment)

[shalyhinpavel]

Man, the sheer velocity and rigor of your R&D is terrifying (in a good way). Brilliant breakdown on the Frugendorff crawler.

I completely agree with your Finding 1: under the 16MB constraint, reallocating saved parameters from depth into width (Dmodel) is the most dominant lever. It fundamentally changes the capacity bottleneck.

However, regarding Finding 2 (recursion provides zero per-step benefit) and Finding 3 (early boost that decays) — this is exactly the "Catastrophic Interference" wall I’ve been hitting in my own models. When you force the exact same shared weights to process sequential loops without a mechanism to 'anchor' the state or break the symmetry, the latent representation just turns into mush (noise) after the 2nd or 3rd pass. The network literally overwrites its own thoughts. You might need something at the very end of the loop to 'translate' that abstract recursive state back into distinct logits, rather than forcing the shared block to do both grammar and vocabulary mapping.
Your roadmap towards Ludicrous Speed (Triton / chunk_delta_rule / CUDA graphs) is 100% the future of this leaderboard. The Python dispatch overhead is killing us all. If you get that FLA kernel stable in ClownCar_III, it’s going to change the meta.

Keep pushing these limits. I'll be watching your CUDA graph experiments very closely while I finalize my next purely architectural swing!

[newjordan]

I got it stable dude... I am hitting a .97 and a 10mb file size. But it's fucjing nuts, I'm getting to .37 9' THE MODEL BUILD!!!! But I can't stabilize the signal, spent all night working it down but couldn't get it stabilized to get a submission it. It screams. I had this built before I saw your delta net and it was the key to about 4 things I was working on. This is the very tip.

It the crawler needs reverse gradients and a system to re-combine. Like a polarized gradient funnel, with easing

Rust scaffold built and smoke testing all day today