Nhoodie/omni-dna-20m-hgt-stacked-lora

Omni-DNA-20M HGT Stacked LoRA — Harmonics Architecture

⚠️ Research Artifact — NOT Recommended for Production Use

This model is an unintended but valuable research artifact. It was produced across multiple interrupted training runs with known bugs. Its behavior is unpredictable and it should only be used for research into hierarchical LoRA architectures.

Architecture: Hierarchical Stacked LoRA (HSL) — "Harmonics"

3 sequential LoRA levels (rank=16 each) stacked on Omni-DNA-20M:

h = W_base(x)                    # frozen base model
h = h + B₀A₀(x)                  # Level 0: weight-space (fundamental)
h = h + B₁A₁(h)                  # Level 1: activation-space (1st harmonic)
h = h + B₂A₂(h)                  # Level 2: activation-space (2nd harmonic)

• Effective rank: 16³ = 4,096 via sequential composition (not additive)
• Level 0 finds the fundamental representation (converges first)
• Level 1 refines on top of Level 0 output (mid-frequency)
• Level 2 captures residual signal (highest frequency / rarest patterns)
• Each level receives the accumulated output of all previous levels

Key Finding: Per-Layer Gradient Handling

This is a heterogeneous architecture — each level operates in a different representational space. Global gradient clipping destroys learning by compressing all layers through a single bottleneck. Per-layer gradient norms must be tracked independently.

Training History (The Messy Truth)

Run	Steps	Data Batches	Why It Died
1	0→4356	17,424	OOM (eval memory leak + long sequence attention)
2	4356→5742	5,544	Killed to deploy log timestamps
3	5742→7950	8,832	OOM (same eval leak pattern)
4	7920→9999	8,316	Killed at completion

Total: ~39,516 batches ≈ 5.0 effective epochs (but across 4 interrupted runs with reshuffled data due to broken epoch tracking — each restart reshuffles from epoch 0)

Known Bugs in Training Code

1. Epoch tracking broken on resume: always starts epoch loop at 0. Fix: start_epoch = resume_step // steps_per_epoch
2. Memory leak in eval: ~6MB leaked per eval from CUDA allocator fragmentation. gc.collect + empty_cache slows but does not stop it.
3. VRAM log shows PyTorch-allocated only (72MB), not real nvidia-smi total (2.3GB). Gap is CUDA allocator fragmentation.
4. expandable_segments:True must be set via tmux, not export VAR=val && nohup (env var is lost)

Performance at Step 9999

Token	AUC	F1	Samples (pos)
HGT_prok_host	0.7855	0.0000	11
HGT_arch_euk	0.6417	0.0000	13
HGT_detection	0.6290	0.0000	97
HGT_fungi_euk	0.5118	0.0000	81
HGT_bac_euk	0.4557	0.0149	133
HGT_prot_euk	0.4393	0.0625	62
Overall	0.5140	—	—

Best overall AUC: 0.5745 at step 1287 (early training before snap-collapse oscillations)

What Worked

• Ranking (AUC) learned well — prok_host at 0.79 with only 11 positive samples
• Hierarchical emergence observed: detection → prok_host → fungi_euk → arch_euk (rarest last)
• Level 0 converges first, then resumes learning when upper levels create new signal
• Per-layer gradient handling kept training alive where global clipping failed
• 101 successful evals before OOM

What Did Not Work

• F1 never stabilized — logit rescaling phase never completed across 5 effective epochs
• Decision thresholds never calibrated — model ranks well but cannot classify
• Memory leak capped training length at ~8000 steps per uninterrupted run

Task Tokens

Token ID	Token	Type
4096	0	Label
4097	1	Label
4098	2	Label
4117	HGT_detection	Binary HGT detection
4119	HGT_bac_euk	Bacteria→Eukaryote transfer
4120	HGT_arch_euk	Archaea→Eukaryote transfer
4121	HGT_fungi_euk	Fungi→Eukaryote transfer
4122	HGT_prot_euk	Protist→Eukaryote transfer
4123	HGT_prok_host	Prokaryotic host transfer

Configuration

BASE_MODEL = "zehui127/Omni-DNA-20M"  # expanded to omni-dna-20m-hgt
NUM_LEVELS = 3
LORA_R = 16
LORA_ALPHA = 32
TARGET_MODULES = ["att_proj", "attn_out", "ff_proj", "ff_out"]
LEARNING_RATE = 2e-4
GRAD_ACCUM = 4
MAX_LENGTH = 4096
MAX_TOKENS_PER_BATCH = 4096  # 1 sequence per batch
TRAINABLE_PARAMS = 2,693,312  # 20.41% of model

Datasets

• IslandViewer4: Genomic island detection (binary)
• euHGT: Cross-domain transfer type classification
• DeepHGT: Large-scale HGT dataset

Total: 95,344 train / 23,837 eval

Code

Training script and full eval history available at: https://github.com/NamelessHoodie/dnabert-mut-exp/tree/dev/sad-omni-hyena/harmonics/

Citation

If you use this architecture or findings, please reference:

• Architecture: "Harmonics" — Hierarchical Stacked LoRA with sequential composition
• Base model: Omni-DNA by Zehui et al.
• Datasets: IslandViewer4, euHGT, DeepHGT