NeuroCAP-QARN

Neural-Cognitive Audio Perception with Quality-Aware Retrieval Neuro-Fusion.

NeuroCAP-QARN is a reproducible research package for robust audio target detection under out-of-distribution acoustic shifts. The system trains a compact acoustic detector, generates virtual EEG-like cognitive responses from audio, and fuses the generated neuro evidence through a gated residual pathway.

Paper PDF | Supplementary PDF | LaTeX source | Results | Model files

Highlights

• Three-stage training pipeline: acoustic student, virtual EEG generator, and QARN fusion.
• Public reproduction entry points for training, evaluation, figure generation, and manifest verification.
• Full result tables for OOD evaluation, residual utility, ablations, confusion matrices, and feature-space analysis.
• Packaged checkpoints for the main NeuroCAP-QARN model and lightweight acoustic baselines.
• Manuscript-ready figures, supplementary figures, LaTeX source, and source-data CSV files.

Hero Demo

The release includes generated figures and source data for the main diagnostic views.

OOD difficulty landscape	QARN decision utility

Feature Matrix

Area	Included	Entry points / artifacts
Acoustic backbone training	Yes	scripts/train_stage1_backbone.py, scripts/train_baselines.py
Virtual EEG generation	Yes	scripts/train_stage2_neuro_generator.py
QARN fusion training	Yes	scripts/train_final_qarn.py, scripts/train.py --stage all
OOD evaluation	Yes	scripts/evaluate.py, scripts/evaluate_qarn.py, results/main_model/
Baseline comparison	Yes	results/baselines/, checkpoints/baseline_light_models/
Feature invariance analysis	Yes	scripts/analyze_feature_invariance.py, results/feature_invariance/
Figure and table regeneration	Yes	scripts/make_figures.py, scripts/make_supplement_tables.py
Manuscript source	Yes	manuscript/source/main.tex, manuscript/source/supplementary.tex
Integrity verification	Yes	MANIFEST.csv, scripts/verify_manifest.py

Architecture

flowchart LR
    X["Audio clip / log-Mel spectrogram"] --> S["Stage 1<br/>Sound encoder"]
    S --> ZS["Sound logits"]
    S --> A["Temporal acoustic features"]

    E["Paired EEG during training"] --> VAE["Stage 2a<br/>EEG VAE latent space"]
    A --> FLOW["Stage 2b<br/>Conditional latent generator"]
    VAE --> DEC["VAE decoder"]
    FLOW --> DEC
    DEC --> EEG["Virtual EEG response"]

    EEG --> IN["EEG interpreter"]
    IN --> N["Neuro sequence + global feature"]
    A --> QARN["Stage 3<br/>Quality-Aware Retrieval Neuro-Fusion"]
    N --> QARN
    QARN --> DZ["Gated neuro residual"]
    ZS --> FUSED["Final fused logits"]
    DZ --> FUSED
    FUSED --> Y["Target / non-target / background prediction"]

Loading

Quick Start

1. Create an environment

python -m venv .venv
. .venv/Scripts/activate  # Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

2. Configure data paths

Copy and edit the public config template:

cp configs/public_train_eval.example.json configs/local_train_eval.json

Minimal fields:

{
  "dataset_root": "data/public_dataset_root",
  "metadata_csv": "data/metadata/split_metadata.csv",
  "checkpoint": "checkpoints/neurocap_main/model.pt",
  "results_dir": "results/main_model",
  "output_dir": "outputs/public_run",
  "device": "cuda"
}

The metadata CSV should contain:

Column	Meaning
spec_path	Path to a .npy or .npz log-Mel spectrogram tensor
label or class_id	Integer class label in [0, 4]
eeg_path	Path to a .npy or .npz EEG tensor, required for Stage 2

Relative paths are resolved under dataset_root.

3. Run the three-stage training pipeline

python scripts/train_stage1_backbone.py --config configs/local_train_eval.json --backbone deformable
python scripts/train_stage2_neuro_generator.py --config configs/local_train_eval.json --backbone deformable
python scripts/train_final_qarn.py --config configs/local_train_eval.json --backbone deformable

Equivalent all-in-one command:

python scripts/train.py --stage all --config configs/local_train_eval.json --backbone deformable

4. Evaluate

python scripts/evaluate.py --config configs/local_train_eval.json --checkpoint checkpoints/neurocap_main/model.pt
python scripts/evaluate_qarn.py --config configs/local_train_eval.json --checkpoint checkpoints/neurocap_main/model.pt
python scripts/evaluate_baselines.py --config configs/local_train_eval.json --dry-run

5. Regenerate figures and tables

python scripts/make_figures.py --dry-run
python scripts/make_supplement_tables.py --dry-run
python scripts/analyze_feature_invariance.py --dry-run

Model Files

The repository includes compact model checkpoints and training logs for the main model and acoustic baselines.

File	Approx. size	Description
checkpoints/neurocap_main/model.pt	20.0 MB	Main NeuroCAP-QARN checkpoint
checkpoints/neurocap_main/train_log.csv	<1 MB	Main-model training log
checkpoints/baseline_light_models/best_resnet10.pt	19.8 MB	ResNet10 acoustic baseline
checkpoints/baseline_light_models/best_paulnet.pt	8.6 MB	PaulNet acoustic baseline
checkpoints/baseline_light_models/best_deformable.pt	7.5 MB	Deformable-convolution acoustic baseline
checkpoints/baseline_light_models/best_dilated.pt	7.1 MB	Dilated-attention acoustic baseline
checkpoints/baseline_light_models/best_dualbranch.pt	0.7 MB	Dual-branch acoustic baseline
checkpoints/baseline_light_models/best_gtcnn.pt	0.5 MB	Gammatone-CNN acoustic baseline

All model files are below GitHub's 100 MB single-file limit.

Edge AI & Inference

NeuroCAP-QARN is designed around a compact acoustic student rather than a large frozen audio backbone at inference time. The deployed path uses:

1. Log-Mel audio input.
2. Compact sound encoder.
3. Conditional virtual-EEG generation.
4. EEG interpreter.
5. QARN residual fusion.

Typical inference call:

python scripts/evaluate_qarn.py \
  --config configs/local_train_eval.json \
  --checkpoint checkpoints/neurocap_main/model.pt

For edge-oriented experiments:

• Use the deformable, dilated, or resnet10 backbones from the public entry points.
• Compare against results/baselines/classic_light_models/new_light_baselines_ood_summary.csv.
• Profile memory and latency with your deployment batch size, because virtual EEG generation adds targeted compute beyond the sound-only branch.

Repository Layout

.
├── checkpoints/                  # Main and baseline model checkpoints
├── configs/                      # Public config templates and experiment configs
├── docs/assets/                  # README preview images
├── manuscript/                   # Paper PDFs and LaTeX source
│   └── source/
│       ├── figures/              # fig1.pdf ... fig8.pdf
│       ├── supplementary_figures/# sfig1.pdf ... sfig8.pdf
│       └── supplementary_source_data/
├── outputs/                      # Runtime outputs
├── results/                      # Main, baseline, and feature-invariance results
├── scripts/                      # Training, evaluation, plotting, and checks
├── src/neurocap_repro/           # Reusable public Python helpers
├── MANIFEST.csv                  # File integrity manifest
├── README.md
└── requirements.txt

Docs Index

Document	Purpose
manuscript/main.pdf	Main paper PDF
manuscript/supplementary.pdf	Supplementary material PDF
manuscript/source/main.tex	Main LaTeX source
manuscript/source/supplementary.tex	Supplementary LaTeX source
configs/README.md	Configuration notes
configs/public_train_eval.example.json	Public training/evaluation template
results/main_model/summary_abcd_mean.csv	ABCD mean OOD summary
results/main_model/overall_by_dataset.csv	Per-OOD-set main-model metrics
results/main_model/residual_utility.csv	Residual decision-utility source data
results/feature_invariance/feature_shift_metrics.csv	Feature discrepancy metrics
MANIFEST.csv	Release file manifest

Reproducibility Check

Verify package integrity:

python scripts/verify_manifest.py --manifest MANIFEST.csv

Run a lightweight smoke check:

python scripts/public_entrypoint.py --dry-run --config configs/public_train_eval.example.json

Git Policy

Recommended contribution workflow:

• Keep main stable and reproducible.
• Use focused branches such as feature/eval-script, fix/config-paths, or docs/readme-refresh.
• Commit source, configs, figures, and result CSVs intentionally; avoid committing local datasets.
• Keep generated experiment outputs under outputs/ unless they are promoted into results/.
• Run python scripts/verify_manifest.py --manifest MANIFEST.csv before tagging a release.
• For large future checkpoints, use GitHub Releases or Git LFS rather than normal git blobs.

Badge Syntax

The badges at the top use Shields.io static badge URLs. The pattern is:

[![Label](https://img.shields.io/badge/<left_text>-<right_text>-<color>?logo=<logo>&logoColor=white)](<target>)

Examples used here:

[![Python](https://img.shields.io/badge/Python-3.10%2B-3776AB?logo=python&logoColor=white)](#quick-start)
[![PyTorch](https://img.shields.io/badge/PyTorch-2.0%2B-EE4C2C?logo=pytorch&logoColor=white)](requirements.txt)
[![Reproducibility](https://img.shields.io/badge/reproducibility-manifest_checked-2ea44f)](#reproducibility-check)

Tips:

• Escape spaces as _ or %20.
• Escape + as %2B.
• Use stable internal anchors for README sections, such as #quick-start.
• Prefer factual badges over decorative badges.