mimo-7b-gdn-hybrid-stage2b-kd

Stage-2b endpoint — long-context forward-KL KD @32k, 294M tokens, offloaded-optimizer 4-rank recipe (LR 2.5e-5). See stage2b_kd_summary.json for exit metrics.

Part of the MiMo second-model arm of a study converting full-attention models into GDN (gated DeltaNet) hybrids (uniform 1:4 retention) and recovering capability via staged distillation. Teacher/base: MiMo-7B-RL-0530.

NOTE: these checkpoints use the custom mimo_gdn architecture — register it before loading (import src.models.mimo_gdn from the project repo, or use the load_causal_lm helper in src/eval/sample_generations.py). Bare AutoModelForCausalLM will not work. vLLM serving needs src.models.vllm_mimo_gdn.register().

Pipeline: surgery/init (uniform 1:4 plain inheritance) -> stage-1 hidden-state alignment -> stage-2a forward-KL KD @4k -> stage-2b long-context KD @32k. Within-arm reference: teacher wikitext-2 PPL 27.51 (RL'd teacher; never cross-arm-compare absolute PPL).