---
license: apache-2.0
base_model: allenai/OLMoE-1B-7B-0125-Instruct
tags: [moe, pruning, olmoe, winnow]
---

# winnow-olmoe-math-keep25

Channel-level pruned + healed [OLMoE-1B-7B-0125-Instruct](https://huggingface.co/allenai/OLMoE-1B-7B-0125-Instruct).
Requires `trust_remote_code=True` (ragged variable-width experts).

| | |
|---|---|
| Base | allenai/OLMoE-1B-7B-0125-Instruct |
| Params | 2.09B |
| Experts fully deleted | 442/1024 |
| Keep fraction | 0.25 |
| Criterion | channel-level REAP, per-layer budgets, block 128, min width 128 |
| Calibration | Dolmino-math (`scores_0125inst_dolmino-math`) |
| Final forward top-128 KL | 0.088 |

## Healing

Off-policy forward-KL distillation against cached top-128 teacher targets
(`dolci_math_curated_opd_top128`), 150 steps, 120k loss tokens/step (18M total), AdamW8bit,
lr 3e-5, wd 0.1, 10 warmup steps, grad clip 1.0, chat frames, max seq len 2048, seed 1223.

All six models in this release share this recipe and are matched on optimizer
steps and loss tokens. Note they are **not** matched on data exposure: the math
cache is 6.48M unique tokens (~2.8 epochs at 150 steps) while the general cache
is 22.3M (~0.81 epochs).