A Dual-Rate Architecture for Context-Aware Language Modeling
A PyTorch implementation bridging Digital Signal Processing (DSP) multirate theory with discrete text generation. This repository introduces a decoupled LLM architecture that separates high-level semantic reasoning (the "Planner") from low-level grammatical synthesis (the "Vocoder").
This approach natively solves the
This repository is the direct evolution of the foundational experiments in eladwf/adaptive-multirate-transformers. While the previous work explored exploratory LFO wrappers and adaptive hyperparameters on character-level models, this repository scales those DSP principles into a highly structured, sentence-level semantic vocoder for BPE tokenization.
Standard dense LLMs (like GPT and LLaMA) treat text generation as a flat, sequence-to-sequence math problem. The model spends the exact same amount of compute predicting the letter "e" at the end of "the" as it does calculating the logical crux of a mathematical proof.
Because self-attention scales quadratically (
In Text-to-Speech (TTS), it is standard practice to decouple the pipeline: a model generates a low-sample-rate continuous signal (a mel-spectrogram), which is then fed into a high-sample-rate Vocoder (like WaveNet) to synthesize discrete audio samples.
This repository applies that exact paradigm to discrete text (BPE/characters) using a dual-rate architecture:
- The Semantic Planner (Slow Rate): A multirate sentence-level autoregressive semantic planner. It operates on highly compressed semantic embeddings and predicts the next sentence's embedding exactly at the 1/C token rate.
-
The Lexical Vocoder (Fast Rate): An autoregressive GPT that learns high-frequency local grammar and spells out the tokens using an
$O(N \cdot W)$ sliding-window local attention mechanism, strictly modulated by the top-down Planner.
My dual-track mechanism processes text at two independently optimized clock rates. A slow, sequence-level top track provides dense conceptual conditioning, while the fast autoregressive bottom track emits discrete sequence tokens. The semantic vector "vocodes" the generation at key adapter bottlenecks.
[Top Track: Slow Rate 1/C] [Bottom Track: Fast Rate]
Sentence Embeddings BPE Tokens
│ │
▼ ▼
[Autoregressive Planner] [GPT Blocks (Sliding Window)]
│ │
▼ ▼
(Future Semantic Vector) ══════╗ (Hidden States)
║ │
▼ ▼
[ _TopDownAdapterBlock (Conditioning Phase) ]
│
▼
(Next Token Emission)
Rather than concatenating semantic conditions into the prompt or embedding layer—which often collapses the base model's language capabilities—this architecture uses a late-stage adapter fusion.
The base GPT computes the local grammar unimpeded via its fast sliding-window blocks. The TopDownAdapterBlock then calculates a "delta" probability distribution based on the global semantic phase. The final output is a feed-forward correction:
To prevent the control signal from becoming a static DC offset, the architecture implements dynamic upsampling (upsample_add or cross_attn). The slow-rate semantic timeline is interpolated to match the fast-rate lexical sequence, ensuring the vocoder knows exactly which conceptual "frame" it is currently inside during step-by-step autoregressive generation.
By running the heavy reasoning over a compressed semantic sequence, this architecture acts as a massive context-window multiplier.
Let
Standard GPT Attention Complexity:
Multirate System Complexity:
The attention cost is split between the slow-rate Planner and the fast-rate Vocoder:
Total Logical Complexity:
Example: For a 100,000-token document at
Note
Current Implementation: This repository validates the logical complexity using banded boolean masks within standard PyTorch nn.MultiheadAttention. Because standard PyTorch still allocates the full attention matrix prior to masking, wall-clock speedups and VRAM reductions are not yet fully realized in this reference codebase. True sub-quadratic hardware scaling requires swapping the base blocks with a block-sparse sparse attention kernel (e.g., FlashAttention-2 or xFormers BlockDiagonalMask).
Warning
Conditioning Over-Reliance & Exposure Bias: The _TopDownAdapterBlock transmits the semantic signal so efficiently that the base GPT heavily prioritizes latent-decoding over local syntactic learning. Even with 15% Semantic Dropout applied from step zero, the model achieves an unusually high Top-1 accuracy (~85%), indicating it uses the 384D vector as a hash-key for the sentence. Future work will explore aggressive Classifier-Free Guidance (e.g., 50%+ dropout) or continuous noise injection to force the base model to build a more robust, independent grammar engine and prevent repetitive loops during nucleus sampling.
The following sample demonstrate the multirate vocoder generating greedy continuations based on the top-down semantic vectors. This is a raw, unedited outputs from the validation set evaluated by the Ollama LLM judge.
Example: Successful Semantic Alignment
The vocoder successfully internalizes the narrative direction from the semantic vector and generates a fluent, grammatically correct continuation that aligns with the reference plot.
Reference Concept: "...near the pit. One day, Tom lost his red ball. He was very sad. Tom asked his friend, Sam, to help him search for the ball. They look"
Vocoder Output: "...near the pit. One day, Tom lost his red ball. Tom was very sad. Tom went to the west field and asked his friend, Sam, to help him..."
Judge Score: 4/5 (High Coherence & Fluency)
When evaluated on standard narrative datasets (e.g., TinyStories via BPE tokenization), the TopDownSemanticVocoder outperforms the predictive power of a standard autoregressive GPT baseline of equivalent size, achieving better results from the LLM judge.
Below is the training loss convergence curve, demonstrating stable optimization despite the complex dual-rate adapter configuration.
The core pipeline is stripped down for reproducibility and integration:
llm_dsp_lab/models/topdown_semantic_vocoder.py: The core PyTorch architecture containingTopDownSemanticVocoderand_TopDownAdapterBlock.llm_dsp_lab/models/semantic_planner.py: The multirate sentence-level autoregressive semantic planner.scripts/prepare_topdown_annotations.py: The data pipeline script. UsesspaCyandSentenceTransformersto extract sentence-level semantics and BPE-aligned POS tags to build the multirate.ptcache. Includes memory-safe sharding for massive corpora.scripts/run_experiment.py: A lightweight, standalone training loop demonstrating how to load the multirate cache and calculate the joint loss (logits_loss + planner_loss).
(1) Annotate your dataset:
python scripts/prepare_topdown_annotations.py --dataset tinystories(2) Train the Vocoder:
python scripts/run_experiment.py configs/topdown_semantic_vocoder_tinystories_bpe.yamlMIT License. Feel free to integrate the TopDownAdapterBlock into your own multirate workflows.
If you have questions, ideas, or want to collaborate, feel free to reach out:
- Email: eladwf@gmail.com