Record: LOGOS-44 Z=0 - Toroidal CDMA Field Decoder (44-pass iterative routing)#980
Open
slowomir33-arch wants to merge 7 commits intoopenai:mainfrom
Open
Record: LOGOS-44 Z=0 - Toroidal CDMA Field Decoder (44-pass iterative routing)#980slowomir33-arch wants to merge 7 commits intoopenai:mainfrom
slowomir33-arch wants to merge 7 commits intoopenai:mainfrom
Conversation
Introduced the LOGOS-44 micro-LLM architecture with a focus on parameter efficiency and compute scaling. Added key features, training results, and future plans for experimentation.
… quantum-coded CDMA field
Added experimental micro-LLM architecture LOGOS-44 focused on extreme parameter efficiency
and compute-over-parameters scaling.
Key characteristics:
- ~300k parameters total
- iterative routing depth: 44 passes
- shared-weight recurrent transformer core
- low-rank orthogonal projections
- CDMA-style semantic field decoder
- optional quantum-coded basis initialization (Qiskit)
Architecture highlights:
- embedding dim: 512
- vocab: 4096 (archetypal tokenizer)
- routing depth: 44
- weight tying enabled
- toroidal bottleneck dynamics
- coherence-gated residual mixing
Training results:
- final cross-entropy ("impedance"): 0.4388
- stable convergence across 50k iterations
- no gradient explosion despite deep routing
- emergent attractor dynamics observed
Motivation:
This experiment explores the hypothesis that iterative compute depth can partially
substitute parameter count in ultra-small language models.
Notes:
- quantum layer currently acts as structured stochastic initialization
- further experiments planned: depth scaling (8/16/44), ALiBi positional encoding,
and expanded dataset evaluation
Files added:
- logos44_micro.py
- quantum_codes.py
- training_logs.txt
- experimental_notes.md
… README - logos44/train.py: full LOGOS-44 Quantum training pipeline (from Colab) - logos44/logs/training_300k.txt: quantum ignition training log (300k params) - logos44/README.md: updated with Quantum Edition architecture docs - .gitignore: allow logos44/logs/ subdirectory
- submission.json: proper competition format (author, github_id, val_bpb, bytes_total) - README.md: accurate architecture docs, repo structure, honest pending status - logos44_quantum.pth: trained Quantum Edition model (1.2 MB, ~300k params) - Logos44_Quantum.ipynb: Colab training notebook
New record submission:
- records/track_10min_16mb/2026-03-28_LOGOS44_Quantum_ToroidalCDMA_Z0/
- train_gpt.py: LOGOS-44 architecture integrated with FineWeb eval pipeline
(SentencePiece tokenizer, BPB metric, DDP, Muon, int8+zlib quantization)
- submission.json: proper competition format
- README.md: architecture docs
Removed:
- logos44_extreme.pth (18.6 MB > 16 MB limit)
slowomir33-arch
changed the title
slowomir33-arch
changed the title
slowomir33-arch
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LOGOS-44 Z=0: Toroidal CDMA Field Decoder
Architecture: Non-transformer iterative routing micro-LLM
val_bpb: TBD (pending 8xH100 training run)
Parameters: ~4.5M (with tied 1024-vocab embeddings)
What is this?
An experimental non-transformer architecture that routes the signal 44 times through a single shared block instead of stacking N separate layers. Each pass consists of:
Key innovation
Trades parameters for compute depth: 44 shared passes = 44-layer effective depth with ~4.5M params. The CDMA field provides non-local memory without attention quadratic cost.
Files
records/track_10min_16mb/2026-03-28_LOGOS44_Quantum_ToroidalCDMA_Z0/train_gpt.py- Self-contained, competition-compatible (754 lines)records/track_10min_16mb/2026-03-28_LOGOS44_Quantum_ToroidalCDMA_Z0/submission.jsonrecords/track_10min_16mb/2026-03-28_LOGOS44_Quantum_ToroidalCDMA_Z0/README.mdCompatible with existing evaluation pipeline (SentencePiece tokenizer, FineWeb BPB, DDP, Muon optimizer, int8+zlib quantization).