mimo-7b-gdn-hybrid-stage2a-kd

Stage-2a endpoint — forward-KL KD @4k, 600M tokens (job 42852481). Exit: val KL 1.512->0.2812, top-1 agree 0.777, wikitext PPL 24.2 (0.88x teacher = calibration drift, not health). LR 5e-5 probe-picked on val KL.

Part of the MiMo second-model arm of a study converting full-attention models into GDN (gated DeltaNet) hybrids (uniform 1:4 retention) and recovering capability via staged distillation. Teacher/base: MiMo-7B-RL-0530.

NOTE: these checkpoints use the custom mimo_gdn architecture — register it before loading (import src.models.mimo_gdn from the project repo, or use the load_causal_lm helper in src/eval/sample_generations.py). Bare AutoModelForCausalLM will not work. vLLM serving needs src.models.vllm_mimo_gdn.register().

Pipeline: surgery/init (uniform 1:4 plain inheritance) -> stage-1 hidden-state alignment -> stage-2a forward-KL KD @4k -> stage-2b long-context KD @32k. Within-arm reference: teacher wikitext-2 PPL 27.51 (RL'd teacher; never cross-arm-compare absolute PPL).